==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-b776b7d88-thrqf Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-default-pool-f05840a3-i7d7/10.142.0.25 Start Time: Tue, 09 Apr 2024 22:06:23 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=b776b7d88 skaffold.dev/run-id=4b7964b4-e9a0-4b57-b0e4-191137bec797 Annotations: Status: Running IP: 10.106.40.159 IPs: IP: 10.106.40.159 Controlled By: ReplicaSet/lodemon-b776b7d88 Containers: lodemon: Container ID: containerd://9e4ec388ffbc8dffd0c34738770df99eeb00e25e5e7a565f0da6a210ff1b9362 Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:e4be5d425b9198ab976022ff44dd785b61fe8218e43dfc240e77fc7ef03ecad1 Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Tue, 09 Apr 2024 22:06:24 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /lodestar/config/deployments_to_monitor.yaml from deployments-config (rw,path="deployments_to_monitor.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hswzh (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false deployments-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-deployments Optional: false kube-api-access-hswzh: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== 05:11:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722285" 05:11:25 INFO [http_cmd]: http status code OK 05:11:25 DEBUG --- status code --- 05:11:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:25 DEBUG --- http response --- 05:11:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:11:25 DEBUG ----- output ----- 05:11:25 DEBUG { 05:11:25 DEBUG "status": "success", 05:11:25 DEBUG "data": { 05:11:25 DEBUG "resultType": "vector", 05:11:25 DEBUG "result": [ 05:11:25 DEBUG { 05:11:25 DEBUG "metric": { 05:11:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:11:25 DEBUG }, 05:11:25 DEBUG "value": [ 05:11:25 DEBUG 05:11:25 DEBUG 05:11:25 DEBUG 05:11:25 DEBUG [...] 05:11:25 DEBUG 05:11:25 DEBUG 05:11:25 DEBUG 05:11:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:11:25 DEBUG }, 05:11:25 DEBUG "value": [ 05:11:25 DEBUG 1712722285, 05:11:25 DEBUG "1257.9666666666667" 05:11:25 DEBUG ] 05:11:25 DEBUG } 05:11:25 DEBUG ] 05:11:25 DEBUG } 05:11:25 DEBUG } 05:11:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712722285" 05:11:25 INFO [http_cmd]: http status code OK 05:11:25 DEBUG --- status code --- 05:11:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:25 DEBUG --- http response --- 05:11:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:11:25 DEBUG ----- output ----- 05:11:25 DEBUG { 05:11:25 DEBUG "status": "success", 05:11:25 DEBUG "data": { 05:11:25 DEBUG "resultType": "vector", 05:11:25 DEBUG "result": [ 05:11:25 DEBUG { 05:11:25 DEBUG "metric": { 05:11:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:11:25 DEBUG }, 05:11:25 DEBUG "value": [ 05:11:25 DEBUG 05:11:25 DEBUG 05:11:25 DEBUG 05:11:25 DEBUG [...] 05:11:25 DEBUG 05:11:25 DEBUG 05:11:25 DEBUG 05:11:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:11:25 DEBUG }, 05:11:25 DEBUG "value": [ 05:11:25 DEBUG 1712722285, 05:11:25 DEBUG "0.04" 05:11:25 DEBUG ] 05:11:25 DEBUG } 05:11:25 DEBUG ] 05:11:25 DEBUG } 05:11:25 DEBUG } 05:11:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722285" 05:11:25 INFO [http_cmd]: http status code OK 05:11:25 DEBUG --- status code --- 05:11:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:25 DEBUG --- http response --- 05:11:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:11:25 DEBUG ----- output ----- 05:11:25 DEBUG { 05:11:25 DEBUG "status": "success", 05:11:25 DEBUG "data": { 05:11:25 DEBUG "resultType": "vector", 05:11:25 DEBUG "result": [ 05:11:25 DEBUG { 05:11:25 DEBUG "metric": { 05:11:25 DEBUG "pod": "ds-idrepo-0" 05:11:25 DEBUG }, 05:11:25 DEBUG "value": [ 05:11:25 DEBUG 05:11:25 DEBUG 05:11:25 DEBUG 05:11:25 DEBUG [...] 05:11:25 DEBUG 05:11:25 DEBUG 05:11:25 DEBUG 05:11:25 DEBUG "pod": "ds-idrepo-1" 05:11:25 DEBUG }, 05:11:25 DEBUG "value": [ 05:11:25 DEBUG 1712722285, 05:11:25 DEBUG "0" 05:11:25 DEBUG ] 05:11:25 DEBUG } 05:11:25 DEBUG ] 05:11:25 DEBUG } 05:11:25 DEBUG } 05:11:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722286" 05:11:26 INFO [http_cmd]: http status code OK 05:11:26 DEBUG --- status code --- 05:11:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:26 DEBUG --- http response --- 05:11:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:11:26 DEBUG ----- output ----- 05:11:26 DEBUG { 05:11:26 DEBUG "status": "success", 05:11:26 DEBUG "data": { 05:11:26 DEBUG "resultType": "vector", 05:11:26 DEBUG "result": [ 05:11:26 DEBUG { 05:11:26 DEBUG "metric": { 05:11:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:11:26 DEBUG }, 05:11:26 DEBUG "value": [ 05:11:26 DEBUG 05:11:26 DEBUG 05:11:26 DEBUG 05:11:26 DEBUG [...] 05:11:26 DEBUG 05:11:26 DEBUG 05:11:26 DEBUG 05:11:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:11:26 DEBUG }, 05:11:26 DEBUG "value": [ 05:11:26 DEBUG 1712722286, 05:11:26 DEBUG "1253.3" 05:11:26 DEBUG ] 05:11:26 DEBUG } 05:11:26 DEBUG ] 05:11:26 DEBUG } 05:11:26 DEBUG } 05:11:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722286" 05:11:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722286" 05:11:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712722286" 05:11:26 INFO [http_cmd]: http status code OK 05:11:26 DEBUG --- status code --- 05:11:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:26 DEBUG --- http response --- 05:11:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:11:26 DEBUG ----- output ----- 05:11:26 DEBUG { 05:11:26 DEBUG "status": "success", 05:11:26 DEBUG "data": { 05:11:26 DEBUG "resultType": "vector", 05:11:26 DEBUG "result": [ 05:11:26 DEBUG { 05:11:26 DEBUG "metric": { 05:11:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:11:26 DEBUG }, 05:11:26 DEBUG "value": [ 05:11:26 DEBUG 05:11:26 DEBUG 05:11:26 DEBUG 05:11:26 DEBUG [...] 05:11:26 DEBUG 05:11:26 DEBUG 05:11:26 DEBUG 05:11:26 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:11:26 DEBUG }, 05:11:26 DEBUG "value": [ 05:11:26 DEBUG 1712722286, 05:11:26 DEBUG "0" 05:11:26 DEBUG ] 05:11:26 DEBUG } 05:11:26 DEBUG ] 05:11:26 DEBUG } 05:11:26 DEBUG } 05:11:26 INFO [http_cmd]: http status code OK 05:11:26 DEBUG --- status code --- 05:11:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:26 DEBUG --- http response --- 05:11:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:11:26 DEBUG ----- output ----- 05:11:26 DEBUG { 05:11:26 DEBUG "status": "success", 05:11:26 DEBUG "data": { 05:11:26 DEBUG "resultType": "vector", 05:11:26 DEBUG "result": [ 05:11:26 DEBUG { 05:11:26 DEBUG "metric": { 05:11:26 DEBUG "pod": "ds-idrepo-0" 05:11:26 DEBUG }, 05:11:26 DEBUG "value": [ 05:11:26 DEBUG 05:11:26 DEBUG 05:11:26 DEBUG 05:11:26 DEBUG [...] 05:11:26 DEBUG 05:11:26 DEBUG 05:11:26 DEBUG 05:11:26 DEBUG "pod": "ds-idrepo-1" 05:11:26 DEBUG }, 05:11:26 DEBUG "value": [ 05:11:26 DEBUG 1712722286, 05:11:26 DEBUG "0" 05:11:26 DEBUG ] 05:11:26 DEBUG } 05:11:26 DEBUG ] 05:11:26 DEBUG } 05:11:26 DEBUG } 05:11:26 INFO [http_cmd]: http status code OK 05:11:26 DEBUG --- status code --- 05:11:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:26 DEBUG --- http response --- 05:11:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:11:26 DEBUG ----- output ----- 05:11:26 DEBUG { 05:11:26 DEBUG "status": "success", 05:11:26 DEBUG "data": { 05:11:26 DEBUG "resultType": "vector", 05:11:26 DEBUG "result": [ 05:11:26 DEBUG { 05:11:26 DEBUG "metric": { 05:11:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:11:26 DEBUG }, 05:11:26 DEBUG "value": [ 05:11:26 DEBUG 05:11:26 DEBUG 05:11:26 DEBUG 05:11:26 DEBUG [...] 05:11:26 DEBUG 05:11:26 DEBUG 05:11:26 DEBUG 05:11:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:11:26 DEBUG }, 05:11:26 DEBUG "value": [ 05:11:26 DEBUG 1712722286, 05:11:26 DEBUG "2485600256" 05:11:26 DEBUG ] 05:11:26 DEBUG } 05:11:26 DEBUG ] 05:11:26 DEBUG } 05:11:26 DEBUG } 05:11:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722286" 05:11:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722286" 05:11:26 INFO [http_cmd]: http status code OK 05:11:26 DEBUG --- status code --- 05:11:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:26 DEBUG --- http response --- 05:11:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:11:26 INFO [http_cmd]: http status code OK 05:11:26 DEBUG ----- output ----- 05:11:26 DEBUG --- status code --- 05:11:26 DEBUG { 05:11:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:11:26 DEBUG "status": "success", 05:11:26 DEBUG --- http response --- 05:11:26 DEBUG "data": { 05:11:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:11:26 DEBUG "resultType": "vector", 05:11:26 DEBUG ----- output ----- 05:11:26 DEBUG "result": [ 05:11:26 DEBUG { 05:11:26 DEBUG { 05:11:26 DEBUG "status": "success", 05:11:26 DEBUG "metric": { 05:11:26 DEBUG "data": { 05:11:26 DEBUG "instance": "10.142.0.25:9100" 05:11:26 DEBUG "resultType": "vector", 05:11:26 DEBUG }, 05:11:26 DEBUG "result": [ 05:11:26 DEBUG "value": [ 05:11:26 DEBUG { 05:11:26 DEBUG 05:11:26 DEBUG "metric": { 05:11:26 DEBUG 05:11:26 DEBUG "instance": "10.142.0.25:9100" 05:11:26 DEBUG 05:11:26 DEBUG }, 05:11:26 DEBUG [...] 05:11:26 DEBUG "value": [ 05:11:26 DEBUG 05:11:26 DEBUG 05:11:26 DEBUG 05:11:26 DEBUG 05:11:26 DEBUG 05:11:26 DEBUG 05:11:26 DEBUG "instance": "10.142.0.77:9100" 05:11:26 DEBUG [...] 05:11:26 DEBUG }, 05:11:26 DEBUG 05:11:26 DEBUG "value": [ 05:11:26 DEBUG 05:11:26 DEBUG 1712722286, 05:11:26 DEBUG 05:11:26 DEBUG "37468146.733333334" 05:11:26 DEBUG "instance": "10.142.0.77:9100" 05:11:26 DEBUG ] 05:11:26 DEBUG }, 05:11:26 DEBUG } 05:11:26 DEBUG "value": [ 05:11:26 DEBUG ] 05:11:26 DEBUG 1712722286, 05:11:26 DEBUG } 05:11:26 DEBUG "0.016999999999999283" 05:11:26 DEBUG } 05:11:26 DEBUG ] 05:11:26 DEBUG } 05:11:26 DEBUG ] 05:11:26 DEBUG } 05:11:26 DEBUG } 05:11:39 INFO 05:11:39 INFO [loop_until]: kubectl --namespace=xlou top pods 05:11:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:11:39 INFO [loop_until]: OK (rc = 0) 05:11:39 DEBUG --- stdout --- 05:11:39 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 8788m 8805Mi am-654c4fcbcf-9scrj 8743m 8414Mi am-654c4fcbcf-v5qb5 7384m 8401Mi ds-cts-0 7272m 12459Mi ds-cts-1 7415m 12535Mi ds-cts-2 6989m 13246Mi ds-idrepo-0 9789m 5621Mi ds-idrepo-1 308m 5668Mi ds-idrepo-2 373m 5697Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 5m 2213Mi lodemon-b776b7d88-thrqf 8m 70Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3566m 1648Mi 05:11:39 DEBUG --- stderr --- 05:11:39 DEBUG 05:11:45 INFO 05:11:45 INFO [loop_until]: kubectl --namespace=xlou top node 05:11:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:11:45 INFO [loop_until]: OK (rc = 0) 05:11:45 DEBUG --- stdout --- 05:11:45 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 98m 0% 2304Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8130m 51% 9275Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 83m 0% 3253Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10781m 67% 6506Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9213m 57% 9306Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9780m 61% 9649Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 78m 0% 852Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9695m 61% 6275Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 7184m 45% 13185Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 6862m 43% 13898Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 423m 2% 6357Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7548m 47% 13135Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 362m 2% 6321Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3672m 23% 5342Mi 9% 05:11:45 DEBUG --- stderr --- 05:11:45 DEBUG 05:12:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712722342" 05:12:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722342" 05:12:22 ERROR [http_cmd]: ERROR 05:12:22 INFO [http_cmd]: http status code OK 05:12:22 ERROR --- status code --- 05:12:22 DEBUG --- status code --- 05:12:22 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:12:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:22 ERROR --- http response --- 05:12:22 DEBUG --- http response --- 05:12:22 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:12:22 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:12:22 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:12:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722342" 05:12:22 INFO [http_cmd]: http status code OK 05:12:22 DEBUG --- status code --- 05:12:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:22 DEBUG --- http response --- 05:12:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:12:22 DEBUG ----- output ----- 05:12:22 DEBUG { 05:12:22 DEBUG "status": "success", 05:12:22 DEBUG "data": { 05:12:22 DEBUG "resultType": "vector", 05:12:22 DEBUG "result": [ 05:12:22 DEBUG { 05:12:22 DEBUG "metric": { 05:12:22 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:12:22 DEBUG }, 05:12:22 DEBUG "value": [ 05:12:22 DEBUG 05:12:22 DEBUG 05:12:22 DEBUG 05:12:22 DEBUG [...] 05:12:22 DEBUG 05:12:22 DEBUG 05:12:22 DEBUG 05:12:22 DEBUG "pod": "am-654c4fcbcf-487cv" 05:12:22 DEBUG }, 05:12:22 DEBUG "value": [ 05:12:22 DEBUG 1712722342, 05:12:22 DEBUG "0" 05:12:22 DEBUG ] 05:12:22 DEBUG } 05:12:22 DEBUG ] 05:12:22 DEBUG } 05:12:22 DEBUG } 05:12:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722342" 05:12:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722342" 05:12:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722342" 05:12:22 INFO [http_cmd]: http status code OK 05:12:22 DEBUG --- status code --- 05:12:22 INFO [http_cmd]: http status code OK 05:12:22 INFO [http_cmd]: http status code OK 05:12:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:22 DEBUG --- status code --- 05:12:22 DEBUG --- status code --- 05:12:22 DEBUG --- http response --- 05:12:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:12:22 DEBUG --- http response --- 05:12:22 DEBUG --- http response --- 05:12:22 DEBUG ----- output ----- 05:12:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:12:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:12:22 DEBUG { 05:12:22 DEBUG ----- output ----- 05:12:22 DEBUG ----- output ----- 05:12:22 DEBUG "status": "success", 05:12:22 DEBUG "data": { 05:12:22 DEBUG { 05:12:22 DEBUG "resultType": "vector", 05:12:22 DEBUG { 05:12:22 DEBUG "status": "success", 05:12:22 DEBUG "result": [ 05:12:22 DEBUG "status": "success", 05:12:22 DEBUG "data": { 05:12:22 DEBUG { 05:12:22 DEBUG "data": { 05:12:22 DEBUG "resultType": "vector", 05:12:22 DEBUG "metric": { 05:12:22 DEBUG "resultType": "vector", 05:12:22 DEBUG "result": [ 05:12:22 DEBUG "pod": "ds-idrepo-0" 05:12:22 DEBUG "result": [ 05:12:22 DEBUG { 05:12:22 DEBUG }, 05:12:22 DEBUG { 05:12:22 DEBUG "metric": { 05:12:22 DEBUG "value": [ 05:12:22 DEBUG "metric": { 05:12:22 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:12:22 DEBUG 05:12:22 DEBUG "instance": "10.142.0.25:9100" 05:12:22 DEBUG }, 05:12:22 DEBUG 05:12:22 DEBUG }, 05:12:22 DEBUG "value": [ 05:12:22 DEBUG 05:12:22 DEBUG "value": [ 05:12:22 DEBUG 05:12:22 DEBUG [...] 05:12:22 DEBUG 05:12:22 DEBUG 05:12:22 DEBUG 05:12:22 DEBUG 05:12:22 DEBUG 05:12:22 DEBUG 05:12:22 DEBUG 05:12:22 DEBUG [...] 05:12:22 DEBUG 05:12:22 DEBUG "pod": "ds-cts-2" 05:12:22 DEBUG 05:12:22 DEBUG [...] 05:12:22 DEBUG }, 05:12:22 DEBUG 05:12:22 DEBUG 05:12:22 DEBUG "value": [ 05:12:22 DEBUG 05:12:22 DEBUG 05:12:22 DEBUG 1712722342, 05:12:22 DEBUG "pod": "am-654c4fcbcf-487cv" 05:12:22 DEBUG 05:12:22 DEBUG "0" 05:12:22 DEBUG }, 05:12:22 DEBUG "instance": "10.142.0.77:9100" 05:12:22 DEBUG ] 05:12:22 DEBUG "value": [ 05:12:22 DEBUG }, 05:12:22 DEBUG } 05:12:22 DEBUG 1712722342, 05:12:22 DEBUG "value": [ 05:12:22 DEBUG ] 05:12:22 DEBUG "1247.8333333333333" 05:12:22 DEBUG 1712722342, 05:12:22 DEBUG } 05:12:22 DEBUG ] 05:12:22 DEBUG "255742972928" 05:12:22 DEBUG } 05:12:22 DEBUG } 05:12:22 DEBUG ] 05:12:22 DEBUG ] 05:12:22 DEBUG } 05:12:22 DEBUG } 05:12:22 DEBUG } 05:12:22 DEBUG ] 05:12:22 DEBUG } 05:12:22 DEBUG } 05:12:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722342" 05:12:22 INFO [http_cmd]: http status code OK 05:12:22 DEBUG --- status code --- 05:12:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:22 DEBUG --- http response --- 05:12:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:12:22 DEBUG ----- output ----- 05:12:22 DEBUG { 05:12:22 DEBUG "status": "success", 05:12:22 DEBUG "data": { 05:12:22 DEBUG "resultType": "vector", 05:12:22 DEBUG "result": [ 05:12:22 DEBUG { 05:12:22 DEBUG "metric": { 05:12:22 DEBUG "pod": "ds-idrepo-0" 05:12:22 DEBUG }, 05:12:22 DEBUG "value": [ 05:12:22 DEBUG 05:12:22 DEBUG 05:12:22 DEBUG 05:12:22 DEBUG [...] 05:12:22 DEBUG 05:12:22 DEBUG 05:12:22 DEBUG 05:12:22 DEBUG "pod": "ds-cts-2" 05:12:22 DEBUG }, 05:12:22 DEBUG "value": [ 05:12:22 DEBUG 1712722342, 05:12:22 DEBUG "29.866666666666667" 05:12:22 DEBUG ] 05:12:22 DEBUG } 05:12:22 DEBUG ] 05:12:22 DEBUG } 05:12:22 DEBUG } 05:12:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722342" 05:12:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722342" 05:12:22 INFO [http_cmd]: http status code OK 05:12:22 DEBUG --- status code --- 05:12:22 INFO [http_cmd]: http status code OK 05:12:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:22 DEBUG --- status code --- 05:12:22 DEBUG --- http response --- 05:12:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:12:22 DEBUG --- http response --- 05:12:22 DEBUG ----- output ----- 05:12:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:12:22 DEBUG { 05:12:22 DEBUG ----- output ----- 05:12:22 DEBUG "status": "success", 05:12:22 DEBUG { 05:12:22 DEBUG "data": { 05:12:22 DEBUG "status": "success", 05:12:22 DEBUG "resultType": "vector", 05:12:22 DEBUG "data": { 05:12:22 DEBUG "result": [ 05:12:22 DEBUG "resultType": "vector", 05:12:22 DEBUG { 05:12:22 DEBUG "result": [ 05:12:22 DEBUG "metric": { 05:12:22 DEBUG { 05:12:22 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:12:22 DEBUG "metric": { 05:12:22 DEBUG }, 05:12:22 DEBUG "instance": "10.142.0.25:9100" 05:12:22 DEBUG "value": [ 05:12:22 DEBUG }, 05:12:22 DEBUG "value": [ 05:12:22 DEBUG 05:12:22 DEBUG 05:12:22 DEBUG 05:12:22 DEBUG 05:12:22 DEBUG 05:12:22 DEBUG 05:12:22 DEBUG [...] 05:12:22 DEBUG [...] 05:12:22 DEBUG 05:12:22 DEBUG 05:12:22 DEBUG 05:12:22 DEBUG 05:12:22 DEBUG 05:12:22 DEBUG "instance": "10.142.0.77:9100" 05:12:22 DEBUG 05:12:22 DEBUG }, 05:12:22 DEBUG "pod": "ds-idrepo-1" 05:12:22 DEBUG "value": [ 05:12:22 DEBUG }, 05:12:22 DEBUG 1712722342, 05:12:22 DEBUG "value": [ 05:12:22 DEBUG "37312222.96666667" 05:12:22 DEBUG 1712722342, 05:12:22 DEBUG ] 05:12:22 DEBUG } 05:12:22 DEBUG ] 05:12:22 DEBUG } 05:12:22 DEBUG } 05:12:22 DEBUG "0" 05:12:22 DEBUG ] 05:12:22 DEBUG } 05:12:22 DEBUG ] 05:12:22 DEBUG } 05:12:22 DEBUG } 05:12:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722343" 05:12:23 INFO [http_cmd]: http status code OK 05:12:23 DEBUG --- status code --- 05:12:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:23 DEBUG --- http response --- 05:12:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:12:23 DEBUG ----- output ----- 05:12:23 DEBUG { 05:12:23 DEBUG "status": "success", 05:12:23 DEBUG "data": { 05:12:23 DEBUG "resultType": "vector", 05:12:23 DEBUG "result": [ 05:12:23 DEBUG { 05:12:23 DEBUG "metric": { 05:12:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:12:23 DEBUG }, 05:12:23 DEBUG "value": [ 05:12:23 DEBUG 05:12:23 DEBUG 05:12:23 DEBUG 05:12:23 DEBUG [...] 05:12:23 DEBUG 05:12:23 DEBUG 05:12:23 DEBUG 05:12:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:12:23 DEBUG }, 05:12:23 DEBUG "value": [ 05:12:23 DEBUG 1712722343, 05:12:23 DEBUG "226410.5" 05:12:23 DEBUG ] 05:12:23 DEBUG } 05:12:23 DEBUG ] 05:12:23 DEBUG } 05:12:23 DEBUG } 05:12:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722343" 05:12:23 INFO [http_cmd]: http status code OK 05:12:23 DEBUG --- status code --- 05:12:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:23 DEBUG --- http response --- 05:12:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:12:23 DEBUG ----- output ----- 05:12:23 DEBUG { 05:12:23 DEBUG "status": "success", 05:12:23 DEBUG "data": { 05:12:23 DEBUG "resultType": "vector", 05:12:23 DEBUG "result": [ 05:12:23 DEBUG { 05:12:23 DEBUG "metric": { 05:12:23 DEBUG "pod": "ds-idrepo-0" 05:12:23 DEBUG }, 05:12:23 DEBUG "value": [ 05:12:23 DEBUG 05:12:23 DEBUG 05:12:23 DEBUG 05:12:23 DEBUG [...] 05:12:23 DEBUG 05:12:23 DEBUG 05:12:23 DEBUG 05:12:23 DEBUG "pod": "ds-idrepo-1" 05:12:23 DEBUG }, 05:12:23 DEBUG "value": [ 05:12:23 DEBUG 1712722343, 05:12:23 DEBUG "0" 05:12:23 DEBUG ] 05:12:23 DEBUG } 05:12:23 DEBUG ] 05:12:23 DEBUG } 05:12:23 DEBUG } 05:12:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722343" 05:12:23 INFO [http_cmd]: http status code OK 05:12:23 DEBUG --- status code --- 05:12:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:23 DEBUG --- http response --- 05:12:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:12:23 DEBUG ----- output ----- 05:12:23 DEBUG { 05:12:23 DEBUG "status": "success", 05:12:23 DEBUG "data": { 05:12:23 DEBUG "resultType": "vector", 05:12:23 DEBUG "result": [ 05:12:23 DEBUG { 05:12:23 DEBUG "metric": { 05:12:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:12:23 DEBUG }, 05:12:23 DEBUG "value": [ 05:12:23 DEBUG 05:12:23 DEBUG 05:12:23 DEBUG 05:12:23 DEBUG [...] 05:12:23 DEBUG 05:12:23 DEBUG 05:12:23 DEBUG 05:12:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:12:23 DEBUG }, 05:12:23 DEBUG "value": [ 05:12:23 DEBUG 1712722343, 05:12:23 DEBUG "1247.6" 05:12:23 DEBUG ] 05:12:23 DEBUG } 05:12:23 DEBUG ] 05:12:23 DEBUG } 05:12:23 DEBUG } 05:12:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722343" 05:12:23 INFO [http_cmd]: http status code OK 05:12:23 DEBUG --- status code --- 05:12:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:23 DEBUG --- http response --- 05:12:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:12:23 DEBUG ----- output ----- 05:12:23 DEBUG { 05:12:23 DEBUG "status": "success", 05:12:23 DEBUG "data": { 05:12:23 DEBUG "resultType": "vector", 05:12:23 DEBUG "result": [ 05:12:23 DEBUG { 05:12:23 DEBUG "metric": { 05:12:23 DEBUG "instance": "10.142.0.25:9100" 05:12:23 DEBUG }, 05:12:23 DEBUG "value": [ 05:12:23 DEBUG 05:12:23 DEBUG 05:12:23 DEBUG 05:12:23 DEBUG [...] 05:12:23 DEBUG 05:12:23 DEBUG 05:12:23 DEBUG 05:12:23 DEBUG "instance": "10.142.0.77:9100" 05:12:23 DEBUG }, 05:12:23 DEBUG "value": [ 05:12:23 DEBUG 1712722343, 05:12:23 DEBUG "0.03859999999999921" 05:12:23 DEBUG ] 05:12:23 DEBUG } 05:12:23 DEBUG ] 05:12:23 DEBUG } 05:12:23 DEBUG } 05:12:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722344" 05:12:24 INFO [http_cmd]: http status code OK 05:12:24 DEBUG --- status code --- 05:12:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:24 DEBUG --- http response --- 05:12:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:12:24 DEBUG ----- output ----- 05:12:24 DEBUG { 05:12:24 DEBUG "status": "success", 05:12:24 DEBUG "data": { 05:12:24 DEBUG "resultType": "vector", 05:12:24 DEBUG "result": [ 05:12:24 DEBUG { 05:12:24 DEBUG "metric": { 05:12:24 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:12:24 DEBUG }, 05:12:24 DEBUG "value": [ 05:12:24 DEBUG 05:12:24 DEBUG 05:12:24 DEBUG 05:12:24 DEBUG [...] 05:12:24 DEBUG 05:12:24 DEBUG 05:12:24 DEBUG 05:12:24 DEBUG "pod": "ds-idrepo-1" 05:12:24 DEBUG }, 05:12:24 DEBUG "value": [ 05:12:24 DEBUG 1712722344, 05:12:24 DEBUG "166672192.7912727" 05:12:24 DEBUG ] 05:12:24 DEBUG } 05:12:24 DEBUG ] 05:12:24 DEBUG } 05:12:24 DEBUG } 05:12:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712722344" 05:12:24 INFO [http_cmd]: http status code OK 05:12:24 DEBUG --- status code --- 05:12:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:24 DEBUG --- http response --- 05:12:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:12:24 DEBUG ----- output ----- 05:12:24 DEBUG { 05:12:24 DEBUG "status": "success", 05:12:24 DEBUG "data": { 05:12:24 DEBUG "resultType": "vector", 05:12:24 DEBUG "result": [ 05:12:24 DEBUG { 05:12:24 DEBUG "metric": { 05:12:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:12:24 DEBUG }, 05:12:24 DEBUG "value": [ 05:12:24 DEBUG 05:12:24 DEBUG 05:12:24 DEBUG 05:12:24 DEBUG [...] 05:12:24 DEBUG 05:12:24 DEBUG 05:12:24 DEBUG 05:12:24 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:12:24 DEBUG }, 05:12:24 DEBUG "value": [ 05:12:24 DEBUG 1712722344, 05:12:24 DEBUG "3.62104685256681" 05:12:24 DEBUG ] 05:12:24 DEBUG } 05:12:24 DEBUG ] 05:12:24 DEBUG } 05:12:24 DEBUG } 05:12:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712722345" 05:12:25 INFO [http_cmd]: http status code OK 05:12:25 DEBUG --- status code --- 05:12:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:25 DEBUG --- http response --- 05:12:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:12:25 DEBUG ----- output ----- 05:12:25 DEBUG { 05:12:25 DEBUG "status": "success", 05:12:25 DEBUG "data": { 05:12:25 DEBUG "resultType": "vector", 05:12:25 DEBUG "result": [ 05:12:25 DEBUG { 05:12:25 DEBUG "metric": { 05:12:25 DEBUG "pod": "ds-idrepo-0" 05:12:25 DEBUG }, 05:12:25 DEBUG "value": [ 05:12:25 DEBUG 05:12:25 DEBUG 05:12:25 DEBUG 05:12:25 DEBUG [...] 05:12:25 DEBUG 05:12:25 DEBUG 05:12:25 DEBUG 05:12:25 DEBUG "pod": "ds-cts-2" 05:12:25 DEBUG }, 05:12:25 DEBUG "value": [ 05:12:25 DEBUG 1712722345, 05:12:25 DEBUG "0" 05:12:25 DEBUG ] 05:12:25 DEBUG } 05:12:25 DEBUG ] 05:12:25 DEBUG } 05:12:25 DEBUG } 05:12:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722345" 05:12:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722345" 05:12:25 INFO [http_cmd]: http status code OK 05:12:25 DEBUG --- status code --- 05:12:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:25 DEBUG --- http response --- 05:12:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:12:25 DEBUG ----- output ----- 05:12:25 DEBUG { 05:12:25 DEBUG "status": "success", 05:12:25 DEBUG "data": { 05:12:25 DEBUG "resultType": "vector", 05:12:25 DEBUG "result": [ 05:12:25 DEBUG { 05:12:25 DEBUG "metric": { 05:12:25 DEBUG "pod": "ds-idrepo-2" 05:12:25 DEBUG }, 05:12:25 DEBUG "value": [ 05:12:25 DEBUG 05:12:25 DEBUG 05:12:25 DEBUG 05:12:25 DEBUG [...] 05:12:25 DEBUG 05:12:25 DEBUG 05:12:25 DEBUG 05:12:25 DEBUG "pod": "ds-cts-2" 05:12:25 DEBUG }, 05:12:25 DEBUG "value": [ 05:12:25 DEBUG 1712722345, 05:12:25 DEBUG "0" 05:12:25 DEBUG ] 05:12:25 DEBUG } 05:12:25 DEBUG ] 05:12:25 DEBUG } 05:12:25 DEBUG } 05:12:25 INFO [http_cmd]: http status code OK 05:12:25 DEBUG --- status code --- 05:12:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:25 DEBUG --- http response --- 05:12:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:12:25 DEBUG ----- output ----- 05:12:25 DEBUG { 05:12:25 DEBUG "status": "success", 05:12:25 DEBUG "data": { 05:12:25 DEBUG "resultType": "vector", 05:12:25 DEBUG "result": [ 05:12:25 DEBUG { 05:12:25 DEBUG "metric": { 05:12:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:12:25 DEBUG }, 05:12:25 DEBUG "value": [ 05:12:25 DEBUG 05:12:25 DEBUG 05:12:25 DEBUG 05:12:25 DEBUG [...] 05:12:25 DEBUG 05:12:25 DEBUG 05:12:25 DEBUG 05:12:25 DEBUG "pod": "ds-idrepo-1" 05:12:25 DEBUG }, 05:12:25 DEBUG "value": [ 05:12:25 DEBUG 1712722345, 05:12:25 DEBUG "0.24471224277209708" 05:12:25 DEBUG ] 05:12:25 DEBUG } 05:12:25 DEBUG ] 05:12:25 DEBUG } 05:12:25 DEBUG } 05:12:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722345" 05:12:25 INFO [http_cmd]: http status code OK 05:12:25 DEBUG --- status code --- 05:12:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:25 DEBUG --- http response --- 05:12:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:12:25 DEBUG ----- output ----- 05:12:25 DEBUG { 05:12:25 DEBUG "status": "success", 05:12:25 DEBUG "data": { 05:12:25 DEBUG "resultType": "vector", 05:12:25 DEBUG "result": [ 05:12:25 DEBUG { 05:12:25 DEBUG "metric": { 05:12:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:12:25 DEBUG }, 05:12:25 DEBUG "value": [ 05:12:25 DEBUG 05:12:25 DEBUG 05:12:25 DEBUG 05:12:25 DEBUG [...] 05:12:25 DEBUG 05:12:25 DEBUG 05:12:25 DEBUG 05:12:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:12:25 DEBUG }, 05:12:25 DEBUG "value": [ 05:12:25 DEBUG 1712722345, 05:12:25 DEBUG "1247.9666666666667" 05:12:25 DEBUG ] 05:12:25 DEBUG } 05:12:25 DEBUG ] 05:12:25 DEBUG } 05:12:25 DEBUG } 05:12:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712722345" 05:12:25 INFO [http_cmd]: http status code OK 05:12:25 DEBUG --- status code --- 05:12:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:25 DEBUG --- http response --- 05:12:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:12:25 DEBUG ----- output ----- 05:12:25 DEBUG { 05:12:25 DEBUG "status": "success", 05:12:25 DEBUG "data": { 05:12:25 DEBUG "resultType": "vector", 05:12:25 DEBUG "result": [ 05:12:25 DEBUG { 05:12:25 DEBUG "metric": { 05:12:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:12:25 DEBUG }, 05:12:25 DEBUG "value": [ 05:12:25 DEBUG 05:12:25 DEBUG 05:12:25 DEBUG 05:12:25 DEBUG [...] 05:12:25 DEBUG 05:12:25 DEBUG 05:12:25 DEBUG 05:12:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:12:25 DEBUG }, 05:12:25 DEBUG "value": [ 05:12:25 DEBUG 1712722345, 05:12:25 DEBUG "0.04" 05:12:25 DEBUG ] 05:12:25 DEBUG } 05:12:25 DEBUG ] 05:12:25 DEBUG } 05:12:25 DEBUG } 05:12:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722345" 05:12:25 INFO [http_cmd]: http status code OK 05:12:25 DEBUG --- status code --- 05:12:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:25 DEBUG --- http response --- 05:12:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:12:25 DEBUG ----- output ----- 05:12:25 DEBUG { 05:12:25 DEBUG "status": "success", 05:12:25 DEBUG "data": { 05:12:25 DEBUG "resultType": "vector", 05:12:25 DEBUG "result": [ 05:12:25 DEBUG { 05:12:25 DEBUG "metric": { 05:12:25 DEBUG "pod": "ds-idrepo-0" 05:12:25 DEBUG }, 05:12:25 DEBUG "value": [ 05:12:25 DEBUG 05:12:25 DEBUG 05:12:25 DEBUG 05:12:25 DEBUG [...] 05:12:25 DEBUG 05:12:25 DEBUG 05:12:25 DEBUG 05:12:25 DEBUG "pod": "ds-idrepo-1" 05:12:25 DEBUG }, 05:12:25 DEBUG "value": [ 05:12:25 DEBUG 1712722345, 05:12:25 DEBUG "0" 05:12:25 DEBUG ] 05:12:25 DEBUG } 05:12:25 DEBUG ] 05:12:25 DEBUG } 05:12:25 DEBUG } 05:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722346" 05:12:26 INFO [http_cmd]: http status code OK 05:12:26 DEBUG --- status code --- 05:12:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:26 DEBUG --- http response --- 05:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:12:26 DEBUG ----- output ----- 05:12:26 DEBUG { 05:12:26 DEBUG "status": "success", 05:12:26 DEBUG "data": { 05:12:26 DEBUG "resultType": "vector", 05:12:26 DEBUG "result": [ 05:12:26 DEBUG { 05:12:26 DEBUG "metric": { 05:12:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:12:26 DEBUG }, 05:12:26 DEBUG "value": [ 05:12:26 DEBUG 05:12:26 DEBUG 05:12:26 DEBUG 05:12:26 DEBUG [...] 05:12:26 DEBUG 05:12:26 DEBUG 05:12:26 DEBUG 05:12:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:12:26 DEBUG }, 05:12:26 DEBUG "value": [ 05:12:26 DEBUG 1712722346, 05:12:26 DEBUG "1254.3" 05:12:26 DEBUG ] 05:12:26 DEBUG } 05:12:26 DEBUG ] 05:12:26 DEBUG } 05:12:26 DEBUG } 05:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722346" 05:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722346" 05:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712722346" 05:12:26 INFO [http_cmd]: http status code OK 05:12:26 INFO [http_cmd]: http status code OK 05:12:26 DEBUG --- status code --- 05:12:26 DEBUG --- status code --- 05:12:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:26 DEBUG --- http response --- 05:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:12:26 DEBUG ----- output ----- 05:12:26 DEBUG { 05:12:26 DEBUG "status": "success", 05:12:26 DEBUG "data": { 05:12:26 DEBUG "resultType": "vector", 05:12:26 DEBUG "result": [ 05:12:26 DEBUG { 05:12:26 DEBUG "metric": { 05:12:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:12:26 DEBUG }, 05:12:26 DEBUG "value": [ 05:12:26 DEBUG 05:12:26 DEBUG 05:12:26 DEBUG 05:12:26 DEBUG [...] 05:12:26 DEBUG 05:12:26 DEBUG 05:12:26 DEBUG 05:12:26 DEBUG "pod": "ds-idrepo-1" 05:12:26 DEBUG }, 05:12:26 DEBUG "value": [ 05:12:26 DEBUG 1712722346, 05:12:26 DEBUG "2287.1195151515153" 05:12:26 DEBUG ] 05:12:26 DEBUG } 05:12:26 DEBUG ] 05:12:26 DEBUG } 05:12:26 DEBUG } 05:12:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:26 INFO [http_cmd]: http status code OK 05:12:26 DEBUG --- http response --- 05:12:26 DEBUG --- status code --- 05:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:12:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:26 DEBUG ----- output ----- 05:12:26 DEBUG --- http response --- 05:12:26 DEBUG { 05:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:12:26 DEBUG "status": "success", 05:12:26 DEBUG ----- output ----- 05:12:26 DEBUG "data": { 05:12:26 DEBUG { 05:12:26 DEBUG "resultType": "vector", 05:12:26 DEBUG "status": "success", 05:12:26 DEBUG "result": [ 05:12:26 DEBUG "data": { 05:12:26 DEBUG { 05:12:26 DEBUG "resultType": "vector", 05:12:26 DEBUG "result": [ 05:12:26 DEBUG "metric": { 05:12:26 DEBUG { 05:12:26 DEBUG "pod": "ds-idrepo-0" 05:12:26 DEBUG "metric": { 05:12:26 DEBUG }, 05:12:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:12:26 DEBUG "value": [ 05:12:26 DEBUG }, 05:12:26 DEBUG 05:12:26 DEBUG "value": [ 05:12:26 DEBUG 05:12:26 DEBUG 05:12:26 DEBUG 05:12:26 DEBUG 05:12:26 DEBUG [...] 05:12:26 DEBUG 05:12:26 DEBUG 05:12:26 DEBUG [...] 05:12:26 DEBUG 05:12:26 DEBUG 05:12:26 DEBUG 05:12:26 DEBUG 05:12:26 DEBUG "pod": "ds-idrepo-1" 05:12:26 DEBUG 05:12:26 DEBUG }, 05:12:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:12:26 DEBUG "value": [ 05:12:26 DEBUG }, 05:12:26 DEBUG 1712722346, 05:12:26 DEBUG "value": [ 05:12:26 DEBUG "0" 05:12:26 DEBUG 1712722346, 05:12:26 DEBUG ] 05:12:26 DEBUG "2485944320" 05:12:26 DEBUG } 05:12:26 DEBUG ] 05:12:26 DEBUG ] 05:12:26 DEBUG } 05:12:26 DEBUG } 05:12:26 DEBUG ] 05:12:26 DEBUG } 05:12:26 DEBUG } 05:12:26 DEBUG } 05:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722346" 05:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722346" 05:12:26 INFO [http_cmd]: http status code OK 05:12:26 INFO [http_cmd]: http status code OK 05:12:26 DEBUG --- status code --- 05:12:26 DEBUG --- status code --- 05:12:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:12:26 DEBUG --- http response --- 05:12:26 DEBUG --- http response --- 05:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:12:26 DEBUG ----- output ----- 05:12:26 DEBUG ----- output ----- 05:12:26 DEBUG { 05:12:26 DEBUG { 05:12:26 DEBUG "status": "success", 05:12:26 DEBUG "status": "success", 05:12:26 DEBUG "data": { 05:12:26 DEBUG "data": { 05:12:26 DEBUG "resultType": "vector", 05:12:26 DEBUG "resultType": "vector", 05:12:26 DEBUG "result": [ 05:12:26 DEBUG "result": [ 05:12:26 DEBUG { 05:12:26 DEBUG { 05:12:26 DEBUG "metric": { 05:12:26 DEBUG "metric": { 05:12:26 DEBUG "instance": "10.142.0.25:9100" 05:12:26 DEBUG "instance": "10.142.0.25:9100" 05:12:26 DEBUG }, 05:12:26 DEBUG }, 05:12:26 DEBUG "value": [ 05:12:26 DEBUG "value": [ 05:12:26 DEBUG 05:12:26 DEBUG 05:12:26 DEBUG 05:12:26 DEBUG 05:12:26 DEBUG 05:12:26 DEBUG 05:12:26 DEBUG [...] 05:12:26 DEBUG 05:12:26 DEBUG [...] 05:12:26 DEBUG 05:12:26 DEBUG 05:12:26 DEBUG 05:12:26 DEBUG 05:12:26 DEBUG "instance": "10.142.0.77:9100" 05:12:26 DEBUG 05:12:26 DEBUG }, 05:12:26 DEBUG "instance": "10.142.0.77:9100" 05:12:26 DEBUG "value": [ 05:12:26 DEBUG }, 05:12:26 DEBUG 1712722346, 05:12:26 DEBUG "value": [ 05:12:26 DEBUG "37313293.5" 05:12:26 DEBUG 1712722346, 05:12:26 DEBUG ] 05:12:26 DEBUG "0.018666666666666443" 05:12:26 DEBUG } 05:12:26 DEBUG ] 05:12:26 DEBUG ] 05:12:26 DEBUG } 05:12:26 DEBUG } 05:12:26 DEBUG ] 05:12:26 DEBUG } 05:12:26 DEBUG } 05:12:26 DEBUG } 05:12:39 INFO 05:12:39 INFO [loop_until]: kubectl --namespace=xlou top pods 05:12:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:12:39 INFO [loop_until]: OK (rc = 0) 05:12:39 DEBUG --- stdout --- 05:12:39 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 8813m 8806Mi am-654c4fcbcf-9scrj 8642m 8419Mi am-654c4fcbcf-v5qb5 7587m 8398Mi ds-cts-0 7426m 13078Mi ds-cts-1 7300m 13158Mi ds-cts-2 6538m 13334Mi ds-idrepo-0 9697m 5621Mi ds-idrepo-1 314m 5668Mi ds-idrepo-2 339m 5697Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 4m 2212Mi lodemon-b776b7d88-thrqf 8m 70Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3550m 1648Mi 05:12:39 DEBUG --- stderr --- 05:12:39 DEBUG 05:12:45 INFO 05:12:45 INFO [loop_until]: kubectl --namespace=xlou top node 05:12:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:12:46 INFO [loop_until]: OK (rc = 0) 05:12:46 DEBUG --- stdout --- 05:12:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 91m 0% 2303Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8212m 51% 9275Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 84m 0% 3256Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10772m 67% 6501Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9404m 59% 9307Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9851m 61% 9648Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 76m 0% 854Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9731m 61% 6275Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 7429m 46% 13814Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 6857m 43% 13914Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 400m 2% 6356Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7578m 47% 13755Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 369m 2% 6323Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3689m 23% 5344Mi 9% 05:12:46 DEBUG --- stderr --- 05:12:46 DEBUG 05:13:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712722402" 05:13:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722402" 05:13:22 INFO [http_cmd]: http status code OK 05:13:22 DEBUG --- status code --- 05:13:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:22 DEBUG --- http response --- 05:13:22 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:13:22 ERROR [http_cmd]: ERROR 05:13:22 ERROR --- status code --- 05:13:22 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:13:22 ERROR --- http response --- 05:13:22 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:13:22 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:13:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722402" 05:13:22 INFO [http_cmd]: http status code OK 05:13:22 DEBUG --- status code --- 05:13:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:22 DEBUG --- http response --- 05:13:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:13:22 DEBUG ----- output ----- 05:13:22 DEBUG { 05:13:22 DEBUG "status": "success", 05:13:22 DEBUG "data": { 05:13:22 DEBUG "resultType": "vector", 05:13:22 DEBUG "result": [ 05:13:22 DEBUG { 05:13:22 DEBUG "metric": { 05:13:22 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:13:22 DEBUG }, 05:13:22 DEBUG "value": [ 05:13:22 DEBUG 05:13:22 DEBUG 05:13:22 DEBUG 05:13:22 DEBUG [...] 05:13:22 DEBUG 05:13:22 DEBUG 05:13:22 DEBUG 05:13:22 DEBUG "pod": "am-654c4fcbcf-487cv" 05:13:22 DEBUG }, 05:13:22 DEBUG "value": [ 05:13:22 DEBUG 1712722402, 05:13:22 DEBUG "0" 05:13:22 DEBUG ] 05:13:22 DEBUG } 05:13:22 DEBUG ] 05:13:22 DEBUG } 05:13:22 DEBUG } 05:13:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722402" 05:13:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722402" 05:13:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722402" 05:13:22 INFO [http_cmd]: http status code OK 05:13:22 DEBUG --- status code --- 05:13:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:22 DEBUG --- http response --- 05:13:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:13:22 DEBUG ----- output ----- 05:13:22 DEBUG { 05:13:22 DEBUG "status": "success", 05:13:22 DEBUG "data": { 05:13:22 DEBUG "resultType": "vector", 05:13:22 DEBUG "result": [ 05:13:22 DEBUG { 05:13:22 DEBUG "metric": { 05:13:22 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:13:22 DEBUG }, 05:13:22 DEBUG "value": [ 05:13:22 DEBUG 05:13:22 DEBUG 05:13:22 DEBUG 05:13:22 DEBUG [...] 05:13:22 DEBUG 05:13:22 DEBUG 05:13:22 DEBUG 05:13:22 DEBUG "pod": "am-654c4fcbcf-487cv" 05:13:22 DEBUG }, 05:13:22 DEBUG "value": [ 05:13:22 DEBUG 1712722402, 05:13:22 DEBUG "1266.3666666666666" 05:13:22 DEBUG ] 05:13:22 DEBUG } 05:13:22 DEBUG ] 05:13:22 DEBUG } 05:13:22 DEBUG } 05:13:22 INFO [http_cmd]: http status code OK 05:13:22 INFO [http_cmd]: http status code OK 05:13:22 DEBUG --- status code --- 05:13:22 DEBUG --- status code --- 05:13:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:22 DEBUG --- http response --- 05:13:22 DEBUG --- http response --- 05:13:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:13:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:13:22 DEBUG ----- output ----- 05:13:22 DEBUG ----- output ----- 05:13:22 DEBUG { 05:13:22 DEBUG { 05:13:22 DEBUG "status": "success", 05:13:22 DEBUG "status": "success", 05:13:22 DEBUG "data": { 05:13:22 DEBUG "data": { 05:13:22 DEBUG "resultType": "vector", 05:13:22 DEBUG "resultType": "vector", 05:13:22 DEBUG "result": [ 05:13:22 DEBUG "result": [ 05:13:22 DEBUG { 05:13:22 DEBUG { 05:13:22 DEBUG "metric": { 05:13:22 DEBUG "metric": { 05:13:22 DEBUG "pod": "ds-idrepo-0" 05:13:22 DEBUG "instance": "10.142.0.25:9100" 05:13:22 DEBUG }, 05:13:22 DEBUG }, 05:13:22 DEBUG "value": [ 05:13:22 DEBUG "value": [ 05:13:22 DEBUG 05:13:22 DEBUG 05:13:22 DEBUG 05:13:22 DEBUG 05:13:22 DEBUG 05:13:22 DEBUG 05:13:22 DEBUG [...] 05:13:22 DEBUG [...] 05:13:22 DEBUG 05:13:22 DEBUG 05:13:22 DEBUG 05:13:22 DEBUG 05:13:22 DEBUG 05:13:22 DEBUG 05:13:22 DEBUG "pod": "ds-cts-2" 05:13:22 DEBUG "instance": "10.142.0.77:9100" 05:13:22 DEBUG }, 05:13:22 DEBUG }, 05:13:22 DEBUG "value": [ 05:13:22 DEBUG "value": [ 05:13:22 DEBUG 1712722402, 05:13:22 DEBUG 1712722402, 05:13:22 DEBUG "0" 05:13:22 DEBUG "257672242176" 05:13:22 DEBUG ] 05:13:22 DEBUG ] 05:13:22 DEBUG } 05:13:22 DEBUG } 05:13:22 DEBUG ] 05:13:22 DEBUG ] 05:13:22 DEBUG } 05:13:22 DEBUG } 05:13:22 DEBUG } 05:13:22 DEBUG } 05:13:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722402" 05:13:22 INFO [http_cmd]: http status code OK 05:13:22 DEBUG --- status code --- 05:13:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:22 DEBUG --- http response --- 05:13:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:13:22 DEBUG ----- output ----- 05:13:22 DEBUG { 05:13:22 DEBUG "status": "success", 05:13:22 DEBUG "data": { 05:13:22 DEBUG "resultType": "vector", 05:13:22 DEBUG "result": [ 05:13:22 DEBUG { 05:13:22 DEBUG "metric": { 05:13:22 DEBUG "pod": "ds-idrepo-0" 05:13:22 DEBUG }, 05:13:22 DEBUG "value": [ 05:13:22 DEBUG 05:13:22 DEBUG 05:13:22 DEBUG 05:13:22 DEBUG [...] 05:13:22 DEBUG 05:13:22 DEBUG 05:13:22 DEBUG 05:13:22 DEBUG "pod": "ds-cts-2" 05:13:22 DEBUG }, 05:13:22 DEBUG "value": [ 05:13:22 DEBUG 1712722402, 05:13:22 DEBUG "29.833333333333332" 05:13:22 DEBUG ] 05:13:22 DEBUG } 05:13:22 DEBUG ] 05:13:22 DEBUG } 05:13:22 DEBUG } 05:13:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722402" 05:13:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722402" 05:13:22 INFO [http_cmd]: http status code OK 05:13:22 DEBUG --- status code --- 05:13:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:22 DEBUG --- http response --- 05:13:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:13:22 DEBUG ----- output ----- 05:13:22 DEBUG { 05:13:22 DEBUG "status": "success", 05:13:22 DEBUG "data": { 05:13:22 DEBUG "resultType": "vector", 05:13:22 DEBUG "result": [ 05:13:22 DEBUG { 05:13:22 DEBUG "metric": { 05:13:22 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:13:22 DEBUG }, 05:13:22 DEBUG "value": [ 05:13:22 DEBUG 05:13:22 DEBUG 05:13:22 DEBUG 05:13:22 DEBUG [...] 05:13:22 DEBUG 05:13:22 DEBUG 05:13:22 DEBUG 05:13:22 DEBUG "pod": "am-654c4fcbcf-9scrj" 05:13:22 DEBUG }, 05:13:22 DEBUG "value": [ 05:13:22 DEBUG 1712722402, 05:13:22 DEBUG "0" 05:13:22 DEBUG ] 05:13:22 DEBUG } 05:13:22 DEBUG ] 05:13:22 DEBUG } 05:13:22 DEBUG } 05:13:22 INFO [http_cmd]: http status code OK 05:13:22 DEBUG --- status code --- 05:13:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:22 DEBUG --- http response --- 05:13:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:13:22 DEBUG ----- output ----- 05:13:22 DEBUG { 05:13:22 DEBUG "status": "success", 05:13:22 DEBUG "data": { 05:13:22 DEBUG "resultType": "vector", 05:13:22 DEBUG "result": [ 05:13:22 DEBUG { 05:13:22 DEBUG "metric": { 05:13:22 DEBUG "instance": "10.142.0.25:9100" 05:13:22 DEBUG }, 05:13:22 DEBUG "value": [ 05:13:22 DEBUG 05:13:22 DEBUG 05:13:22 DEBUG 05:13:22 DEBUG [...] 05:13:22 DEBUG 05:13:22 DEBUG 05:13:22 DEBUG 05:13:22 DEBUG "instance": "10.142.0.77:9100" 05:13:22 DEBUG }, 05:13:22 DEBUG "value": [ 05:13:22 DEBUG 1712722402, 05:13:22 DEBUG "37602615.06666666" 05:13:22 DEBUG ] 05:13:22 DEBUG } 05:13:22 DEBUG ] 05:13:22 DEBUG } 05:13:22 DEBUG } 05:13:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722403" 05:13:23 INFO [http_cmd]: http status code OK 05:13:23 DEBUG --- status code --- 05:13:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:23 DEBUG --- http response --- 05:13:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:13:23 DEBUG ----- output ----- 05:13:23 DEBUG { 05:13:23 DEBUG "status": "success", 05:13:23 DEBUG "data": { 05:13:23 DEBUG "resultType": "vector", 05:13:23 DEBUG "result": [ 05:13:23 DEBUG { 05:13:23 DEBUG "metric": { 05:13:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:13:23 DEBUG }, 05:13:23 DEBUG "value": [ 05:13:23 DEBUG 05:13:23 DEBUG 05:13:23 DEBUG 05:13:23 DEBUG [...] 05:13:23 DEBUG 05:13:23 DEBUG 05:13:23 DEBUG 05:13:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:13:23 DEBUG }, 05:13:23 DEBUG "value": [ 05:13:23 DEBUG 1712722403, 05:13:23 DEBUG "226563" 05:13:23 DEBUG ] 05:13:23 DEBUG } 05:13:23 DEBUG ] 05:13:23 DEBUG } 05:13:23 DEBUG } 05:13:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722403" 05:13:23 INFO [http_cmd]: http status code OK 05:13:23 DEBUG --- status code --- 05:13:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:23 DEBUG --- http response --- 05:13:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:13:23 DEBUG ----- output ----- 05:13:23 DEBUG { 05:13:23 DEBUG "status": "success", 05:13:23 DEBUG "data": { 05:13:23 DEBUG "resultType": "vector", 05:13:23 DEBUG "result": [ 05:13:23 DEBUG { 05:13:23 DEBUG "metric": { 05:13:23 DEBUG "pod": "ds-idrepo-0" 05:13:23 DEBUG }, 05:13:23 DEBUG "value": [ 05:13:23 DEBUG 05:13:23 DEBUG 05:13:23 DEBUG 05:13:23 DEBUG [...] 05:13:23 DEBUG 05:13:23 DEBUG 05:13:23 DEBUG 05:13:23 DEBUG "pod": "ds-idrepo-1" 05:13:23 DEBUG }, 05:13:23 DEBUG "value": [ 05:13:23 DEBUG 1712722403, 05:13:23 DEBUG "0" 05:13:23 DEBUG ] 05:13:23 DEBUG } 05:13:23 DEBUG ] 05:13:23 DEBUG } 05:13:23 DEBUG } 05:13:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722403" 05:13:23 INFO [http_cmd]: http status code OK 05:13:23 DEBUG --- status code --- 05:13:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:23 DEBUG --- http response --- 05:13:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:13:23 DEBUG ----- output ----- 05:13:23 DEBUG { 05:13:23 DEBUG "status": "success", 05:13:23 DEBUG "data": { 05:13:23 DEBUG "resultType": "vector", 05:13:23 DEBUG "result": [ 05:13:23 DEBUG { 05:13:23 DEBUG "metric": { 05:13:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:13:23 DEBUG }, 05:13:23 DEBUG "value": [ 05:13:23 DEBUG 05:13:23 DEBUG 05:13:23 DEBUG 05:13:23 DEBUG [...] 05:13:23 DEBUG 05:13:23 DEBUG 05:13:23 DEBUG 05:13:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:13:23 DEBUG }, 05:13:23 DEBUG "value": [ 05:13:23 DEBUG 1712722403, 05:13:23 DEBUG "1268.2" 05:13:23 DEBUG ] 05:13:23 DEBUG } 05:13:23 DEBUG ] 05:13:23 DEBUG } 05:13:23 DEBUG } 05:13:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722404" 05:13:24 INFO [http_cmd]: http status code OK 05:13:24 DEBUG --- status code --- 05:13:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:24 DEBUG --- http response --- 05:13:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:13:24 DEBUG ----- output ----- 05:13:24 DEBUG { 05:13:24 DEBUG "status": "success", 05:13:24 DEBUG "data": { 05:13:24 DEBUG "resultType": "vector", 05:13:24 DEBUG "result": [ 05:13:24 DEBUG { 05:13:24 DEBUG "metric": { 05:13:24 DEBUG "instance": "10.142.0.25:9100" 05:13:24 DEBUG }, 05:13:24 DEBUG "value": [ 05:13:24 DEBUG 05:13:24 DEBUG 05:13:24 DEBUG 05:13:24 DEBUG [...] 05:13:24 DEBUG 05:13:24 DEBUG 05:13:24 DEBUG 05:13:24 DEBUG "instance": "10.142.0.77:9100" 05:13:24 DEBUG }, 05:13:24 DEBUG "value": [ 05:13:24 DEBUG 1712722404, 05:13:24 DEBUG "0.15556666666666447" 05:13:24 DEBUG ] 05:13:24 DEBUG } 05:13:24 DEBUG ] 05:13:24 DEBUG } 05:13:24 DEBUG } 05:13:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722404" 05:13:24 INFO [http_cmd]: http status code OK 05:13:24 DEBUG --- status code --- 05:13:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:24 DEBUG --- http response --- 05:13:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:13:24 DEBUG ----- output ----- 05:13:24 DEBUG { 05:13:24 DEBUG "status": "success", 05:13:24 DEBUG "data": { 05:13:24 DEBUG "resultType": "vector", 05:13:24 DEBUG "result": [ 05:13:24 DEBUG { 05:13:24 DEBUG "metric": { 05:13:24 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:13:24 DEBUG }, 05:13:24 DEBUG "value": [ 05:13:24 DEBUG 05:13:24 DEBUG 05:13:24 DEBUG 05:13:24 DEBUG [...] 05:13:24 DEBUG 05:13:24 DEBUG 05:13:24 DEBUG 05:13:24 DEBUG "pod": "am-654c4fcbcf-9scrj" 05:13:24 DEBUG }, 05:13:24 DEBUG "value": [ 05:13:24 DEBUG 1712722404, 05:13:24 DEBUG "224345527.41829395" 05:13:24 DEBUG ] 05:13:24 DEBUG } 05:13:24 DEBUG ] 05:13:24 DEBUG } 05:13:24 DEBUG } 05:13:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712722404" 05:13:24 INFO [http_cmd]: http status code OK 05:13:24 DEBUG --- status code --- 05:13:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:24 DEBUG --- http response --- 05:13:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:13:24 DEBUG ----- output ----- 05:13:24 DEBUG { 05:13:24 DEBUG "status": "success", 05:13:24 DEBUG "data": { 05:13:24 DEBUG "resultType": "vector", 05:13:24 DEBUG "result": [ 05:13:24 DEBUG { 05:13:24 DEBUG "metric": { 05:13:24 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:13:24 DEBUG }, 05:13:24 DEBUG "value": [ 05:13:24 DEBUG 05:13:24 DEBUG 05:13:24 DEBUG 05:13:24 DEBUG [...] 05:13:24 DEBUG 05:13:24 DEBUG 05:13:24 DEBUG 05:13:24 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:13:24 DEBUG }, 05:13:24 DEBUG "value": [ 05:13:24 DEBUG 1712722404, 05:13:24 DEBUG "3.603449401547812" 05:13:24 DEBUG ] 05:13:24 DEBUG } 05:13:24 DEBUG ] 05:13:24 DEBUG } 05:13:24 DEBUG } 05:13:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712722405" 05:13:25 INFO [http_cmd]: http status code OK 05:13:25 DEBUG --- status code --- 05:13:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:25 DEBUG --- http response --- 05:13:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:13:25 DEBUG ----- output ----- 05:13:25 DEBUG { 05:13:25 DEBUG "status": "success", 05:13:25 DEBUG "data": { 05:13:25 DEBUG "resultType": "vector", 05:13:25 DEBUG "result": [ 05:13:25 DEBUG { 05:13:25 DEBUG "metric": { 05:13:25 DEBUG "pod": "ds-idrepo-0" 05:13:25 DEBUG }, 05:13:25 DEBUG "value": [ 05:13:25 DEBUG 05:13:25 DEBUG 05:13:25 DEBUG 05:13:25 DEBUG [...] 05:13:25 DEBUG 05:13:25 DEBUG 05:13:25 DEBUG 05:13:25 DEBUG "pod": "ds-cts-2" 05:13:25 DEBUG }, 05:13:25 DEBUG "value": [ 05:13:25 DEBUG 1712722405, 05:13:25 DEBUG "0" 05:13:25 DEBUG ] 05:13:25 DEBUG } 05:13:25 DEBUG ] 05:13:25 DEBUG } 05:13:25 DEBUG } 05:13:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722405" 05:13:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722405" 05:13:25 INFO [http_cmd]: http status code OK 05:13:25 DEBUG --- status code --- 05:13:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:25 DEBUG --- http response --- 05:13:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:13:25 DEBUG ----- output ----- 05:13:25 DEBUG { 05:13:25 DEBUG "status": "success", 05:13:25 DEBUG "data": { 05:13:25 DEBUG "resultType": "vector", 05:13:25 DEBUG "result": [ 05:13:25 DEBUG { 05:13:25 DEBUG "metric": { 05:13:25 DEBUG "pod": "ds-idrepo-2" 05:13:25 DEBUG }, 05:13:25 DEBUG "value": [ 05:13:25 DEBUG 05:13:25 DEBUG 05:13:25 DEBUG 05:13:25 DEBUG [...] 05:13:25 INFO [http_cmd]: http status code OK 05:13:25 DEBUG 05:13:25 DEBUG --- status code --- 05:13:25 DEBUG 05:13:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:25 DEBUG 05:13:25 DEBUG --- http response --- 05:13:25 DEBUG "pod": "ds-cts-2" 05:13:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:13:25 DEBUG ----- output ----- 05:13:25 DEBUG { 05:13:25 DEBUG }, 05:13:25 DEBUG "value": [ 05:13:25 DEBUG "status": "success", 05:13:25 DEBUG 1712722405, 05:13:25 DEBUG "data": { 05:13:25 DEBUG "0.00026376444444444445" 05:13:25 DEBUG "resultType": "vector", 05:13:25 DEBUG ] 05:13:25 DEBUG "result": [ 05:13:25 DEBUG } 05:13:25 DEBUG { 05:13:25 DEBUG ] 05:13:25 DEBUG "metric": { 05:13:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:13:25 DEBUG }, 05:13:25 DEBUG "value": [ 05:13:25 DEBUG 05:13:25 DEBUG 05:13:25 DEBUG 05:13:25 DEBUG [...] 05:13:25 DEBUG 05:13:25 DEBUG 05:13:25 DEBUG 05:13:25 DEBUG "pod": "ds-cts-1" 05:13:25 DEBUG }, 05:13:25 DEBUG "value": [ 05:13:25 DEBUG 1712722405, 05:13:25 DEBUG "4.942911664831864" 05:13:25 DEBUG ] 05:13:25 DEBUG } 05:13:25 DEBUG ] 05:13:25 DEBUG } 05:13:25 DEBUG } 05:13:25 DEBUG } 05:13:25 DEBUG } 05:13:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722405" 05:13:25 INFO [http_cmd]: http status code OK 05:13:25 DEBUG --- status code --- 05:13:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:25 DEBUG --- http response --- 05:13:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:13:25 DEBUG ----- output ----- 05:13:25 DEBUG { 05:13:25 DEBUG "status": "success", 05:13:25 DEBUG "data": { 05:13:25 DEBUG "resultType": "vector", 05:13:25 DEBUG "result": [ 05:13:25 DEBUG { 05:13:25 DEBUG "metric": { 05:13:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:13:25 DEBUG }, 05:13:25 DEBUG "value": [ 05:13:25 DEBUG 05:13:25 DEBUG 05:13:25 DEBUG 05:13:25 DEBUG [...] 05:13:25 DEBUG 05:13:25 DEBUG 05:13:25 DEBUG 05:13:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:13:25 DEBUG }, 05:13:25 DEBUG "value": [ 05:13:25 DEBUG 1712722405, 05:13:25 DEBUG "1268.3666666666666" 05:13:25 DEBUG ] 05:13:25 DEBUG } 05:13:25 DEBUG ] 05:13:25 DEBUG } 05:13:25 DEBUG } 05:13:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712722405" 05:13:25 INFO [http_cmd]: http status code OK 05:13:25 DEBUG --- status code --- 05:13:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:25 DEBUG --- http response --- 05:13:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:13:25 DEBUG ----- output ----- 05:13:25 DEBUG { 05:13:25 DEBUG "status": "success", 05:13:25 DEBUG "data": { 05:13:25 DEBUG "resultType": "vector", 05:13:25 DEBUG "result": [ 05:13:25 DEBUG { 05:13:25 DEBUG "metric": { 05:13:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:13:25 DEBUG }, 05:13:25 DEBUG "value": [ 05:13:25 DEBUG 05:13:25 DEBUG 05:13:25 DEBUG 05:13:25 DEBUG [...] 05:13:25 DEBUG 05:13:25 DEBUG 05:13:25 DEBUG 05:13:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:13:25 DEBUG }, 05:13:25 DEBUG "value": [ 05:13:25 DEBUG 1712722405, 05:13:25 DEBUG "0.04" 05:13:25 DEBUG ] 05:13:25 DEBUG } 05:13:25 DEBUG ] 05:13:25 DEBUG } 05:13:25 DEBUG } 05:13:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722405" 05:13:25 INFO [http_cmd]: http status code OK 05:13:25 DEBUG --- status code --- 05:13:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:25 DEBUG --- http response --- 05:13:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:13:25 DEBUG ----- output ----- 05:13:25 DEBUG { 05:13:25 DEBUG "status": "success", 05:13:25 DEBUG "data": { 05:13:25 DEBUG "resultType": "vector", 05:13:25 DEBUG "result": [ 05:13:25 DEBUG { 05:13:25 DEBUG "metric": { 05:13:25 DEBUG "pod": "ds-idrepo-0" 05:13:25 DEBUG }, 05:13:25 DEBUG "value": [ 05:13:25 DEBUG 05:13:25 DEBUG 05:13:25 DEBUG 05:13:25 DEBUG [...] 05:13:25 DEBUG 05:13:25 DEBUG 05:13:25 DEBUG 05:13:25 DEBUG "pod": "ds-idrepo-1" 05:13:25 DEBUG }, 05:13:25 DEBUG "value": [ 05:13:25 DEBUG 1712722405, 05:13:25 DEBUG "0" 05:13:25 DEBUG ] 05:13:25 DEBUG } 05:13:25 DEBUG ] 05:13:25 DEBUG } 05:13:25 DEBUG } 05:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722406" 05:13:26 INFO [http_cmd]: http status code OK 05:13:26 DEBUG --- status code --- 05:13:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:26 DEBUG --- http response --- 05:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:13:26 DEBUG ----- output ----- 05:13:26 DEBUG { 05:13:26 DEBUG "status": "success", 05:13:26 DEBUG "data": { 05:13:26 DEBUG "resultType": "vector", 05:13:26 DEBUG "result": [ 05:13:26 DEBUG { 05:13:26 DEBUG "metric": { 05:13:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:13:26 DEBUG }, 05:13:26 DEBUG "value": [ 05:13:26 DEBUG 05:13:26 DEBUG 05:13:26 DEBUG 05:13:26 DEBUG [...] 05:13:26 DEBUG 05:13:26 DEBUG 05:13:26 DEBUG 05:13:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:13:26 DEBUG }, 05:13:26 DEBUG "value": [ 05:13:26 DEBUG 1712722406, 05:13:26 DEBUG "1270.3666666666666" 05:13:26 DEBUG ] 05:13:26 DEBUG } 05:13:26 DEBUG ] 05:13:26 DEBUG } 05:13:26 DEBUG } 05:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722406" 05:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722406" 05:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712722406" 05:13:26 INFO [http_cmd]: http status code OK 05:13:26 DEBUG --- status code --- 05:13:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:26 DEBUG --- http response --- 05:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:13:26 DEBUG ----- output ----- 05:13:26 DEBUG { 05:13:26 DEBUG "status": "success", 05:13:26 DEBUG "data": { 05:13:26 DEBUG "resultType": "vector", 05:13:26 DEBUG "result": [ 05:13:26 DEBUG { 05:13:26 DEBUG "metric": { 05:13:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:13:26 DEBUG }, 05:13:26 DEBUG "value": [ 05:13:26 DEBUG 05:13:26 DEBUG 05:13:26 DEBUG 05:13:26 DEBUG [...] 05:13:26 DEBUG 05:13:26 DEBUG 05:13:26 DEBUG 05:13:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:13:26 DEBUG }, 05:13:26 DEBUG "value": [ 05:13:26 DEBUG 1712722406, 05:13:26 DEBUG "0" 05:13:26 DEBUG ] 05:13:26 DEBUG } 05:13:26 DEBUG ] 05:13:26 DEBUG } 05:13:26 DEBUG } 05:13:26 INFO [http_cmd]: http status code OK 05:13:26 DEBUG --- status code --- 05:13:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:26 DEBUG --- http response --- 05:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:13:26 DEBUG ----- output ----- 05:13:26 DEBUG { 05:13:26 DEBUG "status": "success", 05:13:26 DEBUG "data": { 05:13:26 DEBUG "resultType": "vector", 05:13:26 DEBUG "result": [ 05:13:26 DEBUG { 05:13:26 DEBUG "metric": { 05:13:26 DEBUG "pod": "ds-idrepo-0" 05:13:26 DEBUG }, 05:13:26 DEBUG "value": [ 05:13:26 DEBUG 05:13:26 DEBUG 05:13:26 DEBUG 05:13:26 DEBUG [...] 05:13:26 DEBUG 05:13:26 DEBUG 05:13:26 DEBUG 05:13:26 DEBUG "pod": "ds-idrepo-1" 05:13:26 DEBUG }, 05:13:26 DEBUG "value": [ 05:13:26 DEBUG 1712722406, 05:13:26 DEBUG "0" 05:13:26 DEBUG ] 05:13:26 DEBUG } 05:13:26 DEBUG ] 05:13:26 DEBUG } 05:13:26 DEBUG } 05:13:26 INFO [http_cmd]: http status code OK 05:13:26 DEBUG --- status code --- 05:13:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:26 DEBUG --- http response --- 05:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:13:26 DEBUG ----- output ----- 05:13:26 DEBUG { 05:13:26 DEBUG "status": "success", 05:13:26 DEBUG "data": { 05:13:26 DEBUG "resultType": "vector", 05:13:26 DEBUG "result": [ 05:13:26 DEBUG { 05:13:26 DEBUG "metric": { 05:13:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:13:26 DEBUG }, 05:13:26 DEBUG "value": [ 05:13:26 DEBUG 05:13:26 DEBUG 05:13:26 DEBUG 05:13:26 DEBUG [...] 05:13:26 DEBUG 05:13:26 DEBUG 05:13:26 DEBUG 05:13:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:13:26 DEBUG }, 05:13:26 DEBUG "value": [ 05:13:26 DEBUG 1712722406, 05:13:26 DEBUG "2483507200" 05:13:26 DEBUG ] 05:13:26 DEBUG } 05:13:26 DEBUG ] 05:13:26 DEBUG } 05:13:26 DEBUG } 05:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722406" 05:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722406" 05:13:26 INFO [http_cmd]: http status code OK 05:13:26 DEBUG --- status code --- 05:13:26 INFO [http_cmd]: http status code OK 05:13:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:26 DEBUG --- status code --- 05:13:26 DEBUG --- http response --- 05:13:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:13:26 DEBUG --- http response --- 05:13:26 DEBUG ----- output ----- 05:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:13:26 DEBUG { 05:13:26 DEBUG ----- output ----- 05:13:26 DEBUG "status": "success", 05:13:26 DEBUG { 05:13:26 DEBUG "data": { 05:13:26 DEBUG "status": "success", 05:13:26 DEBUG "resultType": "vector", 05:13:26 DEBUG "data": { 05:13:26 DEBUG "result": [ 05:13:26 DEBUG "resultType": "vector", 05:13:26 DEBUG { 05:13:26 DEBUG "result": [ 05:13:26 DEBUG "metric": { 05:13:26 DEBUG { 05:13:26 DEBUG "instance": "10.142.0.25:9100" 05:13:26 DEBUG }, 05:13:26 DEBUG "metric": { 05:13:26 DEBUG "value": [ 05:13:26 DEBUG "instance": "10.142.0.25:9100" 05:13:26 DEBUG 05:13:26 DEBUG }, 05:13:26 DEBUG 05:13:26 DEBUG "value": [ 05:13:26 DEBUG 05:13:26 DEBUG 05:13:26 DEBUG [...] 05:13:26 DEBUG 05:13:26 DEBUG 05:13:26 DEBUG 05:13:26 DEBUG 05:13:26 DEBUG [...] 05:13:26 DEBUG 05:13:26 DEBUG 05:13:26 DEBUG "instance": "10.142.0.77:9100" 05:13:26 DEBUG 05:13:26 DEBUG }, 05:13:26 DEBUG 05:13:26 DEBUG "value": [ 05:13:26 DEBUG "instance": "10.142.0.77:9100" 05:13:26 DEBUG 1712722406, 05:13:26 DEBUG }, 05:13:26 DEBUG "0.054999999999999355" 05:13:26 DEBUG "value": [ 05:13:26 DEBUG ] 05:13:26 DEBUG 1712722406, 05:13:26 DEBUG "37605839.03333333" 05:13:26 DEBUG } 05:13:26 DEBUG ] 05:13:26 DEBUG ] 05:13:26 DEBUG } 05:13:26 DEBUG } 05:13:26 DEBUG ] 05:13:26 DEBUG } 05:13:26 DEBUG } 05:13:26 DEBUG } 05:13:39 INFO 05:13:39 INFO [loop_until]: kubectl --namespace=xlou top pods 05:13:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:13:40 INFO [loop_until]: OK (rc = 0) 05:13:40 DEBUG --- stdout --- 05:13:40 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 9033m 8804Mi am-654c4fcbcf-9scrj 8499m 8419Mi am-654c4fcbcf-v5qb5 7463m 8399Mi ds-cts-0 7568m 13319Mi ds-cts-1 7201m 13318Mi ds-cts-2 6675m 12803Mi ds-idrepo-0 9563m 5621Mi ds-idrepo-1 311m 5668Mi ds-idrepo-2 336m 5698Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2212Mi lodemon-b776b7d88-thrqf 5m 70Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3559m 1649Mi 05:13:40 DEBUG --- stderr --- 05:13:40 DEBUG 05:13:46 INFO 05:13:46 INFO [loop_until]: kubectl --namespace=xlou top node 05:13:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:13:46 INFO [loop_until]: OK (rc = 0) 05:13:46 DEBUG --- stdout --- 05:13:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 102m 0% 2304Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8121m 51% 9278Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 71m 0% 3256Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10768m 67% 6506Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9567m 60% 9308Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9961m 62% 9648Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 78m 0% 853Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9750m 61% 6275Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 7205m 45% 13920Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 6857m 43% 13484Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 398m 2% 6359Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7487m 47% 13917Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 372m 2% 6325Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3745m 23% 5346Mi 9% 05:13:46 DEBUG --- stderr --- 05:13:46 DEBUG 05:14:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712722462" 05:14:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722462" 05:14:22 INFO [http_cmd]: http status code OK 05:14:22 DEBUG --- status code --- 05:14:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:22 DEBUG --- http response --- 05:14:22 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:14:22 ERROR [http_cmd]: ERROR 05:14:22 ERROR --- status code --- 05:14:22 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:14:22 ERROR --- http response --- 05:14:22 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:14:22 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:14:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722462" 05:14:22 INFO [http_cmd]: http status code OK 05:14:22 DEBUG --- status code --- 05:14:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:22 DEBUG --- http response --- 05:14:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:14:22 DEBUG ----- output ----- 05:14:22 DEBUG { 05:14:22 DEBUG "status": "success", 05:14:22 DEBUG "data": { 05:14:22 DEBUG "resultType": "vector", 05:14:22 DEBUG "result": [ 05:14:22 DEBUG { 05:14:22 DEBUG "metric": { 05:14:22 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:14:22 DEBUG }, 05:14:22 DEBUG "value": [ 05:14:22 DEBUG 05:14:22 DEBUG 05:14:22 DEBUG 05:14:22 DEBUG [...] 05:14:22 DEBUG 05:14:22 DEBUG 05:14:22 DEBUG 05:14:22 DEBUG "pod": "am-654c4fcbcf-487cv" 05:14:22 DEBUG }, 05:14:22 DEBUG "value": [ 05:14:22 DEBUG 1712722462, 05:14:22 DEBUG "0" 05:14:22 DEBUG ] 05:14:22 DEBUG } 05:14:22 DEBUG ] 05:14:22 DEBUG } 05:14:22 DEBUG } 05:14:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722462" 05:14:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722462" 05:14:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722462" 05:14:22 INFO [http_cmd]: http status code OK 05:14:22 DEBUG --- status code --- 05:14:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:22 DEBUG --- http response --- 05:14:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:14:22 DEBUG ----- output ----- 05:14:22 DEBUG { 05:14:22 DEBUG "status": "success", 05:14:22 DEBUG "data": { 05:14:22 DEBUG "resultType": "vector", 05:14:22 DEBUG "result": [ 05:14:22 DEBUG { 05:14:22 DEBUG "metric": { 05:14:22 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:14:22 DEBUG }, 05:14:22 DEBUG "value": [ 05:14:22 DEBUG 05:14:22 DEBUG 05:14:22 DEBUG 05:14:22 DEBUG [...] 05:14:22 DEBUG 05:14:22 DEBUG 05:14:22 DEBUG 05:14:22 DEBUG "pod": "am-654c4fcbcf-487cv" 05:14:22 DEBUG }, 05:14:22 DEBUG "value": [ 05:14:22 DEBUG 1712722462, 05:14:22 DEBUG "1264.5666666666666" 05:14:22 DEBUG ] 05:14:22 DEBUG } 05:14:22 DEBUG ] 05:14:22 DEBUG } 05:14:22 DEBUG } 05:14:22 INFO [http_cmd]: http status code OK 05:14:22 DEBUG --- status code --- 05:14:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:22 DEBUG --- http response --- 05:14:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:14:22 DEBUG ----- output ----- 05:14:22 DEBUG { 05:14:22 DEBUG "status": "success", 05:14:22 DEBUG "data": { 05:14:22 DEBUG "resultType": "vector", 05:14:22 DEBUG "result": [ 05:14:22 DEBUG { 05:14:22 DEBUG "metric": { 05:14:22 DEBUG "instance": "10.142.0.25:9100" 05:14:22 DEBUG }, 05:14:22 DEBUG "value": [ 05:14:22 DEBUG 05:14:22 DEBUG 05:14:22 DEBUG 05:14:22 DEBUG [...] 05:14:22 DEBUG 05:14:22 DEBUG 05:14:22 DEBUG 05:14:22 DEBUG "instance": "10.142.0.77:9100" 05:14:22 DEBUG }, 05:14:22 DEBUG "value": [ 05:14:22 DEBUG 1712722462, 05:14:22 DEBUG "259464020992" 05:14:22 DEBUG ] 05:14:22 DEBUG } 05:14:22 DEBUG ] 05:14:22 DEBUG } 05:14:22 DEBUG } 05:14:22 INFO [http_cmd]: http status code OK 05:14:22 DEBUG --- status code --- 05:14:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:22 DEBUG --- http response --- 05:14:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:14:22 DEBUG ----- output ----- 05:14:22 DEBUG { 05:14:22 DEBUG "status": "success", 05:14:22 DEBUG "data": { 05:14:22 DEBUG "resultType": "vector", 05:14:22 DEBUG "result": [ 05:14:22 DEBUG { 05:14:22 DEBUG "metric": { 05:14:22 DEBUG "pod": "ds-idrepo-0" 05:14:22 DEBUG }, 05:14:22 DEBUG "value": [ 05:14:22 DEBUG 05:14:22 DEBUG 05:14:22 DEBUG 05:14:22 DEBUG [...] 05:14:22 DEBUG 05:14:22 DEBUG 05:14:22 DEBUG 05:14:22 DEBUG "pod": "ds-cts-2" 05:14:22 DEBUG }, 05:14:22 DEBUG "value": [ 05:14:22 DEBUG 1712722462, 05:14:22 DEBUG "0" 05:14:22 DEBUG ] 05:14:22 DEBUG } 05:14:22 DEBUG ] 05:14:22 DEBUG } 05:14:22 DEBUG } 05:14:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722462" 05:14:22 INFO [http_cmd]: http status code OK 05:14:22 DEBUG --- status code --- 05:14:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:22 DEBUG --- http response --- 05:14:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:14:22 DEBUG ----- output ----- 05:14:22 DEBUG { 05:14:22 DEBUG "status": "success", 05:14:22 DEBUG "data": { 05:14:22 DEBUG "resultType": "vector", 05:14:22 DEBUG "result": [ 05:14:22 DEBUG { 05:14:22 DEBUG "metric": { 05:14:22 DEBUG "pod": "ds-idrepo-0" 05:14:22 DEBUG }, 05:14:22 DEBUG "value": [ 05:14:22 DEBUG 05:14:22 DEBUG 05:14:22 DEBUG 05:14:22 DEBUG [...] 05:14:22 DEBUG 05:14:22 DEBUG 05:14:22 DEBUG 05:14:22 DEBUG "pod": "ds-cts-2" 05:14:22 DEBUG }, 05:14:22 DEBUG "value": [ 05:14:22 DEBUG 1712722462, 05:14:22 DEBUG "28.933333333333334" 05:14:22 DEBUG ] 05:14:22 DEBUG } 05:14:22 DEBUG ] 05:14:22 DEBUG } 05:14:22 DEBUG } 05:14:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722462" 05:14:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722462" 05:14:23 INFO [http_cmd]: http status code OK 05:14:23 DEBUG --- status code --- 05:14:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:23 DEBUG --- http response --- 05:14:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:14:23 DEBUG ----- output ----- 05:14:23 DEBUG { 05:14:23 DEBUG "status": "success", 05:14:23 DEBUG "data": { 05:14:23 DEBUG "resultType": "vector", 05:14:23 DEBUG "result": [ 05:14:23 DEBUG { 05:14:23 DEBUG "metric": { 05:14:23 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:14:23 DEBUG }, 05:14:23 DEBUG "value": [ 05:14:23 DEBUG 05:14:23 DEBUG 05:14:23 DEBUG 05:14:23 DEBUG [...] 05:14:23 DEBUG 05:14:23 DEBUG 05:14:23 DEBUG 05:14:23 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 05:14:23 DEBUG }, 05:14:23 DEBUG "value": [ 05:14:23 DEBUG 1712722462, 05:14:23 DEBUG "0" 05:14:23 DEBUG ] 05:14:23 DEBUG } 05:14:23 DEBUG ] 05:14:23 DEBUG } 05:14:23 DEBUG } 05:14:23 INFO [http_cmd]: http status code OK 05:14:23 DEBUG --- status code --- 05:14:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:23 DEBUG --- http response --- 05:14:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:14:23 DEBUG ----- output ----- 05:14:23 DEBUG { 05:14:23 DEBUG "status": "success", 05:14:23 DEBUG "data": { 05:14:23 DEBUG "resultType": "vector", 05:14:23 DEBUG "result": [ 05:14:23 DEBUG { 05:14:23 DEBUG "metric": { 05:14:23 DEBUG "instance": "10.142.0.25:9100" 05:14:23 DEBUG }, 05:14:23 DEBUG "value": [ 05:14:23 DEBUG 05:14:23 DEBUG 05:14:23 DEBUG 05:14:23 DEBUG [...] 05:14:23 DEBUG 05:14:23 DEBUG 05:14:23 DEBUG 05:14:23 DEBUG "instance": "10.142.0.77:9100" 05:14:23 DEBUG }, 05:14:23 DEBUG "value": [ 05:14:23 DEBUG 1712722462, 05:14:23 DEBUG "37623063.56666666" 05:14:23 DEBUG ] 05:14:23 DEBUG } 05:14:23 DEBUG ] 05:14:23 DEBUG } 05:14:23 DEBUG } 05:14:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722463" 05:14:23 INFO [http_cmd]: http status code OK 05:14:23 DEBUG --- status code --- 05:14:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:23 DEBUG --- http response --- 05:14:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:14:23 DEBUG ----- output ----- 05:14:23 DEBUG { 05:14:23 DEBUG "status": "success", 05:14:23 DEBUG "data": { 05:14:23 DEBUG "resultType": "vector", 05:14:23 DEBUG "result": [ 05:14:23 DEBUG { 05:14:23 DEBUG "metric": { 05:14:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:14:23 DEBUG }, 05:14:23 DEBUG "value": [ 05:14:23 DEBUG 05:14:23 DEBUG 05:14:23 DEBUG 05:14:23 DEBUG [...] 05:14:23 DEBUG 05:14:23 DEBUG 05:14:23 DEBUG 05:14:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:14:23 DEBUG }, 05:14:23 DEBUG "value": [ 05:14:23 DEBUG 1712722463, 05:14:23 DEBUG "226718.5" 05:14:23 DEBUG ] 05:14:23 DEBUG } 05:14:23 DEBUG ] 05:14:23 DEBUG } 05:14:23 DEBUG } 05:14:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722463" 05:14:23 INFO [http_cmd]: http status code OK 05:14:23 DEBUG --- status code --- 05:14:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:23 DEBUG --- http response --- 05:14:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:14:23 DEBUG ----- output ----- 05:14:23 DEBUG { 05:14:23 DEBUG "status": "success", 05:14:23 DEBUG "data": { 05:14:23 DEBUG "resultType": "vector", 05:14:23 DEBUG "result": [ 05:14:23 DEBUG { 05:14:23 DEBUG "metric": { 05:14:23 DEBUG "pod": "ds-idrepo-0" 05:14:23 DEBUG }, 05:14:23 DEBUG "value": [ 05:14:23 DEBUG 05:14:23 DEBUG 05:14:23 DEBUG 05:14:23 DEBUG [...] 05:14:23 DEBUG 05:14:23 DEBUG 05:14:23 DEBUG 05:14:23 DEBUG "pod": "ds-idrepo-1" 05:14:23 DEBUG }, 05:14:23 DEBUG "value": [ 05:14:23 DEBUG 1712722463, 05:14:23 DEBUG "0" 05:14:23 DEBUG ] 05:14:23 DEBUG } 05:14:23 DEBUG ] 05:14:23 DEBUG } 05:14:23 DEBUG } 05:14:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722463" 05:14:23 INFO [http_cmd]: http status code OK 05:14:23 DEBUG --- status code --- 05:14:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:23 DEBUG --- http response --- 05:14:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:14:23 DEBUG ----- output ----- 05:14:23 DEBUG { 05:14:23 DEBUG "status": "success", 05:14:23 DEBUG "data": { 05:14:23 DEBUG "resultType": "vector", 05:14:23 DEBUG "result": [ 05:14:23 DEBUG { 05:14:23 DEBUG "metric": { 05:14:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:14:23 DEBUG }, 05:14:23 DEBUG "value": [ 05:14:23 DEBUG 05:14:23 DEBUG 05:14:23 DEBUG 05:14:23 DEBUG [...] 05:14:23 DEBUG 05:14:23 DEBUG 05:14:23 DEBUG 05:14:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:14:23 DEBUG }, 05:14:23 DEBUG "value": [ 05:14:23 DEBUG 1712722463, 05:14:23 DEBUG "1264.4333333333334" 05:14:23 DEBUG ] 05:14:23 DEBUG } 05:14:23 DEBUG ] 05:14:23 DEBUG } 05:14:23 DEBUG } 05:14:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722464" 05:14:24 INFO [http_cmd]: http status code OK 05:14:24 DEBUG --- status code --- 05:14:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:24 DEBUG --- http response --- 05:14:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:14:24 DEBUG ----- output ----- 05:14:24 DEBUG { 05:14:24 DEBUG "status": "success", 05:14:24 DEBUG "data": { 05:14:24 DEBUG "resultType": "vector", 05:14:24 DEBUG "result": [ 05:14:24 DEBUG { 05:14:24 DEBUG "metric": { 05:14:24 DEBUG "instance": "10.142.0.25:9100" 05:14:24 DEBUG }, 05:14:24 DEBUG "value": [ 05:14:24 DEBUG 05:14:24 DEBUG 05:14:24 DEBUG 05:14:24 DEBUG [...] 05:14:24 DEBUG 05:14:24 DEBUG 05:14:24 DEBUG 05:14:24 DEBUG "instance": "10.142.0.77:9100" 05:14:24 DEBUG }, 05:14:24 DEBUG "value": [ 05:14:24 DEBUG 1712722464, 05:14:24 DEBUG "0.149533333333331" 05:14:24 DEBUG ] 05:14:24 DEBUG } 05:14:24 DEBUG ] 05:14:24 DEBUG } 05:14:24 DEBUG } 05:14:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722464" 05:14:24 INFO [http_cmd]: http status code OK 05:14:24 DEBUG --- status code --- 05:14:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:24 DEBUG --- http response --- 05:14:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:14:24 DEBUG ----- output ----- 05:14:24 DEBUG { 05:14:24 DEBUG "status": "success", 05:14:24 DEBUG "data": { 05:14:24 DEBUG "resultType": "vector", 05:14:24 DEBUG "result": [ 05:14:24 DEBUG { 05:14:24 DEBUG "metric": { 05:14:24 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:14:24 DEBUG }, 05:14:24 DEBUG "value": [ 05:14:24 DEBUG 05:14:24 DEBUG 05:14:24 DEBUG 05:14:24 DEBUG [...] 05:14:24 DEBUG 05:14:24 DEBUG 05:14:24 DEBUG 05:14:24 DEBUG "pod": "ds-idrepo-2" 05:14:24 DEBUG }, 05:14:24 DEBUG "value": [ 05:14:24 DEBUG 1712722464, 05:14:24 DEBUG "172991736.22196394" 05:14:24 DEBUG ] 05:14:24 DEBUG } 05:14:24 DEBUG ] 05:14:24 DEBUG } 05:14:24 DEBUG } 05:14:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712722465" 05:14:25 INFO [http_cmd]: http status code OK 05:14:25 DEBUG --- status code --- 05:14:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:25 DEBUG --- http response --- 05:14:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:14:25 DEBUG ----- output ----- 05:14:25 DEBUG { 05:14:25 DEBUG "status": "success", 05:14:25 DEBUG "data": { 05:14:25 DEBUG "resultType": "vector", 05:14:25 DEBUG "result": [ 05:14:25 DEBUG { 05:14:25 DEBUG "metric": { 05:14:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:14:25 DEBUG }, 05:14:25 DEBUG "value": [ 05:14:25 DEBUG 05:14:25 DEBUG 05:14:25 DEBUG 05:14:25 DEBUG [...] 05:14:25 DEBUG 05:14:25 DEBUG 05:14:25 DEBUG 05:14:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:14:25 DEBUG }, 05:14:25 DEBUG "value": [ 05:14:25 DEBUG 1712722465, 05:14:25 DEBUG "3.6114620119723875" 05:14:25 DEBUG ] 05:14:25 DEBUG } 05:14:25 DEBUG ] 05:14:25 DEBUG } 05:14:25 DEBUG } 05:14:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712722465" 05:14:25 INFO [http_cmd]: http status code OK 05:14:25 DEBUG --- status code --- 05:14:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:25 DEBUG --- http response --- 05:14:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:14:25 DEBUG ----- output ----- 05:14:25 DEBUG { 05:14:25 DEBUG "status": "success", 05:14:25 DEBUG "data": { 05:14:25 DEBUG "resultType": "vector", 05:14:25 DEBUG "result": [ 05:14:25 DEBUG { 05:14:25 DEBUG "metric": { 05:14:25 DEBUG "pod": "ds-idrepo-0" 05:14:25 DEBUG }, 05:14:25 DEBUG "value": [ 05:14:25 DEBUG 05:14:25 DEBUG 05:14:25 DEBUG 05:14:25 DEBUG [...] 05:14:25 DEBUG 05:14:25 DEBUG 05:14:25 DEBUG 05:14:25 DEBUG "pod": "ds-cts-2" 05:14:25 DEBUG }, 05:14:25 DEBUG "value": [ 05:14:25 DEBUG 1712722465, 05:14:25 DEBUG "0" 05:14:25 DEBUG ] 05:14:25 DEBUG } 05:14:25 DEBUG ] 05:14:25 DEBUG } 05:14:25 DEBUG } 05:14:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722465" 05:14:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722465" 05:14:25 INFO [http_cmd]: http status code OK 05:14:25 DEBUG --- status code --- 05:14:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:25 DEBUG --- http response --- 05:14:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:14:25 DEBUG ----- output ----- 05:14:25 DEBUG { 05:14:25 DEBUG "status": "success", 05:14:25 DEBUG "data": { 05:14:25 DEBUG "resultType": "vector", 05:14:25 DEBUG "result": [ 05:14:25 DEBUG { 05:14:25 DEBUG "metric": { 05:14:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:14:25 DEBUG }, 05:14:25 DEBUG "value": [ 05:14:25 DEBUG 05:14:25 DEBUG 05:14:25 DEBUG 05:14:25 DEBUG [...] 05:14:25 DEBUG 05:14:25 DEBUG 05:14:25 DEBUG 05:14:25 DEBUG "pod": "ds-idrepo-2" 05:14:25 DEBUG }, 05:14:25 DEBUG "value": [ 05:14:25 DEBUG 1712722465, 05:14:25 DEBUG "0.164026129680978" 05:14:25 DEBUG ] 05:14:25 INFO [http_cmd]: http status code OK 05:14:25 DEBUG } 05:14:25 DEBUG --- status code --- 05:14:25 DEBUG ] 05:14:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:25 DEBUG } 05:14:25 DEBUG --- http response --- 05:14:25 DEBUG } 05:14:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:14:25 DEBUG ----- output ----- 05:14:25 DEBUG { 05:14:25 DEBUG "status": "success", 05:14:25 DEBUG "data": { 05:14:25 DEBUG "resultType": "vector", 05:14:25 DEBUG "result": [ 05:14:25 DEBUG { 05:14:25 DEBUG "metric": { 05:14:25 DEBUG "pod": "ds-idrepo-2" 05:14:25 DEBUG }, 05:14:25 DEBUG "value": [ 05:14:25 DEBUG 05:14:25 DEBUG 05:14:25 DEBUG 05:14:25 DEBUG [...] 05:14:25 DEBUG 05:14:25 DEBUG 05:14:25 DEBUG 05:14:25 DEBUG "pod": "ds-cts-2" 05:14:25 DEBUG }, 05:14:25 DEBUG "value": [ 05:14:25 DEBUG 1712722465, 05:14:25 DEBUG "0.00021086222222222226" 05:14:25 DEBUG ] 05:14:25 DEBUG } 05:14:25 DEBUG ] 05:14:25 DEBUG } 05:14:25 DEBUG } 05:14:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722465" 05:14:25 INFO [http_cmd]: http status code OK 05:14:25 DEBUG --- status code --- 05:14:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:25 DEBUG --- http response --- 05:14:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:14:25 DEBUG ----- output ----- 05:14:25 DEBUG { 05:14:25 DEBUG "status": "success", 05:14:25 DEBUG "data": { 05:14:25 DEBUG "resultType": "vector", 05:14:25 DEBUG "result": [ 05:14:25 DEBUG { 05:14:25 DEBUG "metric": { 05:14:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:14:25 DEBUG }, 05:14:25 DEBUG "value": [ 05:14:25 DEBUG 05:14:25 DEBUG 05:14:25 DEBUG 05:14:25 DEBUG [...] 05:14:25 DEBUG 05:14:25 DEBUG 05:14:25 DEBUG 05:14:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:14:25 DEBUG }, 05:14:25 DEBUG "value": [ 05:14:25 DEBUG 1712722465, 05:14:25 DEBUG "1264.6333333333332" 05:14:25 DEBUG ] 05:14:25 DEBUG } 05:14:25 DEBUG ] 05:14:25 DEBUG } 05:14:25 DEBUG } 05:14:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712722465" 05:14:25 INFO [http_cmd]: http status code OK 05:14:25 DEBUG --- status code --- 05:14:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:25 DEBUG --- http response --- 05:14:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:14:25 DEBUG ----- output ----- 05:14:25 DEBUG { 05:14:25 DEBUG "status": "success", 05:14:25 DEBUG "data": { 05:14:25 DEBUG "resultType": "vector", 05:14:25 DEBUG "result": [ 05:14:25 DEBUG { 05:14:25 DEBUG "metric": { 05:14:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:14:25 DEBUG }, 05:14:25 DEBUG "value": [ 05:14:25 DEBUG 05:14:25 DEBUG 05:14:25 DEBUG 05:14:25 DEBUG [...] 05:14:25 DEBUG 05:14:25 DEBUG 05:14:25 DEBUG 05:14:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:14:25 DEBUG }, 05:14:25 DEBUG "value": [ 05:14:25 DEBUG 1712722465, 05:14:25 DEBUG "0.04" 05:14:25 DEBUG ] 05:14:25 DEBUG } 05:14:25 DEBUG ] 05:14:25 DEBUG } 05:14:25 DEBUG } 05:14:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722465" 05:14:26 INFO [http_cmd]: http status code OK 05:14:26 DEBUG --- status code --- 05:14:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:26 DEBUG --- http response --- 05:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:14:26 DEBUG ----- output ----- 05:14:26 DEBUG { 05:14:26 DEBUG "status": "success", 05:14:26 DEBUG "data": { 05:14:26 DEBUG "resultType": "vector", 05:14:26 DEBUG "result": [ 05:14:26 DEBUG { 05:14:26 DEBUG "metric": { 05:14:26 DEBUG "pod": "ds-idrepo-0" 05:14:26 DEBUG }, 05:14:26 DEBUG "value": [ 05:14:26 DEBUG 05:14:26 DEBUG 05:14:26 DEBUG 05:14:26 DEBUG [...] 05:14:26 DEBUG 05:14:26 DEBUG 05:14:26 DEBUG 05:14:26 DEBUG "pod": "ds-idrepo-1" 05:14:26 DEBUG }, 05:14:26 DEBUG "value": [ 05:14:26 DEBUG 1712722465, 05:14:26 DEBUG "0" 05:14:26 DEBUG ] 05:14:26 DEBUG } 05:14:26 DEBUG ] 05:14:26 DEBUG } 05:14:26 DEBUG } 05:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722466" 05:14:26 INFO [http_cmd]: http status code OK 05:14:26 DEBUG --- status code --- 05:14:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:26 DEBUG --- http response --- 05:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:14:26 DEBUG ----- output ----- 05:14:26 DEBUG { 05:14:26 DEBUG "status": "success", 05:14:26 DEBUG "data": { 05:14:26 DEBUG "resultType": "vector", 05:14:26 DEBUG "result": [ 05:14:26 DEBUG { 05:14:26 DEBUG "metric": { 05:14:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:14:26 DEBUG }, 05:14:26 DEBUG "value": [ 05:14:26 DEBUG 05:14:26 DEBUG 05:14:26 DEBUG 05:14:26 DEBUG [...] 05:14:26 DEBUG 05:14:26 DEBUG 05:14:26 DEBUG 05:14:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:14:26 DEBUG }, 05:14:26 DEBUG "value": [ 05:14:26 DEBUG 1712722466, 05:14:26 DEBUG "1256.2333333333333" 05:14:26 DEBUG ] 05:14:26 DEBUG } 05:14:26 DEBUG ] 05:14:26 DEBUG } 05:14:26 DEBUG } 05:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722466" 05:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722466" 05:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712722466" 05:14:26 INFO [http_cmd]: http status code OK 05:14:26 DEBUG --- status code --- 05:14:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:26 DEBUG --- http response --- 05:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:14:26 DEBUG ----- output ----- 05:14:26 DEBUG { 05:14:26 DEBUG "status": "success", 05:14:26 DEBUG "data": { 05:14:26 INFO [http_cmd]: http status code OK 05:14:26 DEBUG "resultType": "vector", 05:14:26 DEBUG --- status code --- 05:14:26 DEBUG "result": [ 05:14:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:26 DEBUG { 05:14:26 DEBUG --- http response --- 05:14:26 DEBUG "metric": { 05:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:14:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:14:26 DEBUG ----- output ----- 05:14:26 DEBUG }, 05:14:26 DEBUG { 05:14:26 DEBUG "value": [ 05:14:26 DEBUG "status": "success", 05:14:26 DEBUG 05:14:26 DEBUG 05:14:26 DEBUG "data": { 05:14:26 DEBUG 05:14:26 DEBUG "resultType": "vector", 05:14:26 DEBUG [...] 05:14:26 DEBUG "result": [ 05:14:26 DEBUG 05:14:26 DEBUG { 05:14:26 DEBUG 05:14:26 DEBUG "metric": { 05:14:26 DEBUG 05:14:26 DEBUG "pod": "ds-idrepo-0" 05:14:26 DEBUG }, 05:14:26 DEBUG "pod": "ds-idrepo-2" 05:14:26 DEBUG "value": [ 05:14:26 DEBUG }, 05:14:26 DEBUG 05:14:26 DEBUG "value": [ 05:14:26 DEBUG 05:14:26 DEBUG 1712722466, 05:14:26 DEBUG 05:14:26 DEBUG [...] 05:14:26 DEBUG 05:14:26 DEBUG 05:14:26 DEBUG 05:14:26 DEBUG "pod": "ds-idrepo-1" 05:14:26 DEBUG }, 05:14:26 DEBUG "value": [ 05:14:26 DEBUG 1712722466, 05:14:26 DEBUG "0" 05:14:26 DEBUG ] 05:14:26 DEBUG } 05:14:26 DEBUG ] 05:14:26 DEBUG } 05:14:26 DEBUG } 05:14:26 INFO [http_cmd]: http status code OK 05:14:26 DEBUG --- status code --- 05:14:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:26 DEBUG --- http response --- 05:14:26 DEBUG "1885.7369691551185" 05:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:14:26 DEBUG ----- output ----- 05:14:26 DEBUG { 05:14:26 DEBUG "status": "success", 05:14:26 DEBUG "data": { 05:14:26 DEBUG "resultType": "vector", 05:14:26 DEBUG "result": [ 05:14:26 DEBUG { 05:14:26 DEBUG "metric": { 05:14:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:14:26 DEBUG }, 05:14:26 DEBUG "value": [ 05:14:26 DEBUG 05:14:26 DEBUG 05:14:26 DEBUG 05:14:26 DEBUG [...] 05:14:26 DEBUG 05:14:26 DEBUG 05:14:26 DEBUG 05:14:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:14:26 DEBUG }, 05:14:26 DEBUG "value": [ 05:14:26 DEBUG 1712722466, 05:14:26 DEBUG "2483777536" 05:14:26 DEBUG ] 05:14:26 DEBUG } 05:14:26 DEBUG ] 05:14:26 DEBUG } 05:14:26 DEBUG } 05:14:26 DEBUG ] 05:14:26 DEBUG } 05:14:26 DEBUG ] 05:14:26 DEBUG } 05:14:26 DEBUG } 05:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722466" 05:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722466" 05:14:26 INFO [http_cmd]: http status code OK 05:14:26 DEBUG --- status code --- 05:14:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:26 DEBUG --- http response --- 05:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:14:26 INFO [http_cmd]: http status code OK 05:14:26 DEBUG ----- output ----- 05:14:26 DEBUG --- status code --- 05:14:26 DEBUG { 05:14:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:14:26 DEBUG "status": "success", 05:14:26 DEBUG --- http response --- 05:14:26 DEBUG "data": { 05:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:14:26 DEBUG "resultType": "vector", 05:14:26 DEBUG ----- output ----- 05:14:26 DEBUG "result": [ 05:14:26 DEBUG { 05:14:26 DEBUG { 05:14:26 DEBUG "status": "success", 05:14:26 DEBUG "metric": { 05:14:26 DEBUG "data": { 05:14:26 DEBUG "instance": "10.142.0.25:9100" 05:14:26 DEBUG "resultType": "vector", 05:14:26 DEBUG }, 05:14:26 DEBUG "result": [ 05:14:26 DEBUG "value": [ 05:14:26 DEBUG { 05:14:26 DEBUG 05:14:26 DEBUG "metric": { 05:14:26 DEBUG 05:14:26 DEBUG "instance": "10.142.0.25:9100" 05:14:26 DEBUG 05:14:26 DEBUG }, 05:14:26 DEBUG [...] 05:14:26 DEBUG "value": [ 05:14:26 DEBUG 05:14:26 DEBUG 05:14:26 DEBUG 05:14:26 DEBUG 05:14:26 DEBUG 05:14:26 DEBUG 05:14:26 DEBUG "instance": "10.142.0.77:9100" 05:14:26 DEBUG [...] 05:14:26 DEBUG 05:14:26 DEBUG 05:14:26 DEBUG 05:14:26 DEBUG "instance": "10.142.0.77:9100" 05:14:26 DEBUG }, 05:14:26 DEBUG "value": [ 05:14:26 DEBUG 1712722466, 05:14:26 DEBUG "0.05600000000000004" 05:14:26 DEBUG ] 05:14:26 DEBUG } 05:14:26 DEBUG ] 05:14:26 DEBUG } 05:14:26 DEBUG } 05:14:26 DEBUG }, 05:14:26 DEBUG "value": [ 05:14:26 DEBUG 1712722466, 05:14:26 DEBUG "37624223.63333333" 05:14:26 DEBUG ] 05:14:26 DEBUG } 05:14:26 DEBUG ] 05:14:26 DEBUG } 05:14:26 DEBUG } 05:14:40 INFO 05:14:40 INFO [loop_until]: kubectl --namespace=xlou top pods 05:14:40 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:14:40 INFO [loop_until]: OK (rc = 0) 05:14:40 DEBUG --- stdout --- 05:14:40 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 8868m 8808Mi am-654c4fcbcf-9scrj 8532m 8421Mi am-654c4fcbcf-v5qb5 7267m 8400Mi ds-cts-0 7007m 13319Mi ds-cts-1 7392m 12438Mi ds-cts-2 6895m 13194Mi ds-idrepo-0 9571m 5621Mi ds-idrepo-1 317m 5668Mi ds-idrepo-2 375m 5698Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2212Mi lodemon-b776b7d88-thrqf 8m 70Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3618m 1650Mi 05:14:40 DEBUG --- stderr --- 05:14:40 DEBUG 05:14:46 INFO 05:14:46 INFO [loop_until]: kubectl --namespace=xlou top node 05:14:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:14:46 INFO [loop_until]: OK (rc = 0) 05:14:46 DEBUG --- stdout --- 05:14:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 104m 0% 2306Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8196m 51% 9275Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 81m 0% 3252Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10831m 68% 6490Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9590m 60% 9308Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9965m 62% 9649Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 78m 0% 851Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9422m 59% 6275Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 7170m 45% 12983Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 7067m 44% 13808Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 434m 2% 6359Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7460m 46% 13038Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 373m 2% 6324Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3697m 23% 5345Mi 9% 05:14:46 DEBUG --- stderr --- 05:14:46 DEBUG 05:15:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722522" 05:15:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712722522" 05:15:22 ERROR [http_cmd]: ERROR 05:15:22 ERROR --- status code --- 05:15:22 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:15:22 INFO [http_cmd]: http status code OK 05:15:22 ERROR --- http response --- 05:15:22 DEBUG --- status code --- 05:15:22 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:15:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:22 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:15:22 DEBUG --- http response --- 05:15:22 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:15:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722522" 05:15:22 INFO [http_cmd]: http status code OK 05:15:22 DEBUG --- status code --- 05:15:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:22 DEBUG --- http response --- 05:15:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:15:22 DEBUG ----- output ----- 05:15:22 DEBUG { 05:15:22 DEBUG "status": "success", 05:15:22 DEBUG "data": { 05:15:22 DEBUG "resultType": "vector", 05:15:22 DEBUG "result": [ 05:15:22 DEBUG { 05:15:22 DEBUG "metric": { 05:15:22 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:15:22 DEBUG }, 05:15:22 DEBUG "value": [ 05:15:22 DEBUG 05:15:22 DEBUG 05:15:22 DEBUG 05:15:22 DEBUG [...] 05:15:22 DEBUG 05:15:22 DEBUG 05:15:22 DEBUG 05:15:22 DEBUG "pod": "am-654c4fcbcf-487cv" 05:15:22 DEBUG }, 05:15:22 DEBUG "value": [ 05:15:22 DEBUG 1712722522, 05:15:22 DEBUG "0" 05:15:22 DEBUG ] 05:15:22 DEBUG } 05:15:22 DEBUG ] 05:15:22 DEBUG } 05:15:22 DEBUG } 05:15:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722522" 05:15:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722522" 05:15:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722522" 05:15:22 INFO [http_cmd]: http status code OK 05:15:22 DEBUG --- status code --- 05:15:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:22 DEBUG --- http response --- 05:15:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:15:22 DEBUG ----- output ----- 05:15:22 DEBUG { 05:15:22 DEBUG "status": "success", 05:15:22 DEBUG "data": { 05:15:22 DEBUG "resultType": "vector", 05:15:22 DEBUG "result": [ 05:15:22 DEBUG { 05:15:22 DEBUG "metric": { 05:15:22 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:15:22 DEBUG }, 05:15:22 DEBUG "value": [ 05:15:22 DEBUG 05:15:22 DEBUG 05:15:22 DEBUG 05:15:22 DEBUG [...] 05:15:22 DEBUG 05:15:22 DEBUG 05:15:22 DEBUG 05:15:22 INFO [http_cmd]: http status code OK 05:15:22 DEBUG "pod": "am-654c4fcbcf-487cv" 05:15:22 DEBUG --- status code --- 05:15:22 DEBUG }, 05:15:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:22 DEBUG "value": [ 05:15:22 DEBUG --- http response --- 05:15:22 DEBUG 1712722522, 05:15:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:15:22 DEBUG "1261.8" 05:15:22 DEBUG ----- output ----- 05:15:22 DEBUG ] 05:15:22 DEBUG { 05:15:22 DEBUG } 05:15:22 DEBUG "status": "success", 05:15:22 DEBUG ] 05:15:22 DEBUG "data": { 05:15:22 DEBUG } 05:15:22 DEBUG "resultType": "vector", 05:15:22 DEBUG } 05:15:22 DEBUG "result": [ 05:15:22 DEBUG { 05:15:22 DEBUG "metric": { 05:15:22 DEBUG "instance": "10.142.0.25:9100" 05:15:22 DEBUG }, 05:15:22 DEBUG "value": [ 05:15:22 INFO [http_cmd]: http status code OK 05:15:22 DEBUG 05:15:22 DEBUG --- status code --- 05:15:22 DEBUG 05:15:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:22 DEBUG 05:15:22 DEBUG --- http response --- 05:15:22 DEBUG [...] 05:15:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:15:22 DEBUG 05:15:22 DEBUG ----- output ----- 05:15:22 DEBUG 05:15:22 DEBUG { 05:15:22 DEBUG 05:15:22 DEBUG "status": "success", 05:15:22 DEBUG "instance": "10.142.0.77:9100" 05:15:22 DEBUG "data": { 05:15:22 DEBUG }, 05:15:22 DEBUG "resultType": "vector", 05:15:22 DEBUG "value": [ 05:15:22 DEBUG "result": [ 05:15:22 DEBUG 1712722522, 05:15:22 DEBUG { 05:15:22 DEBUG "261328555008" 05:15:22 DEBUG "metric": { 05:15:22 DEBUG ] 05:15:22 DEBUG "pod": "ds-idrepo-0" 05:15:22 DEBUG } 05:15:22 DEBUG }, 05:15:22 DEBUG ] 05:15:22 DEBUG "value": [ 05:15:22 DEBUG } 05:15:22 DEBUG 05:15:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722522" 05:15:22 DEBUG } 05:15:22 DEBUG 05:15:22 DEBUG 05:15:22 DEBUG [...] 05:15:22 DEBUG 05:15:22 DEBUG 05:15:22 DEBUG 05:15:22 DEBUG "pod": "ds-cts-2" 05:15:22 DEBUG }, 05:15:22 DEBUG "value": [ 05:15:22 DEBUG 1712722522, 05:15:22 DEBUG "0" 05:15:22 DEBUG ] 05:15:22 DEBUG } 05:15:22 DEBUG ] 05:15:22 DEBUG } 05:15:22 DEBUG } 05:15:22 INFO [http_cmd]: http status code OK 05:15:22 DEBUG --- status code --- 05:15:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:22 DEBUG --- http response --- 05:15:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:15:22 DEBUG ----- output ----- 05:15:22 DEBUG { 05:15:22 DEBUG "status": "success", 05:15:22 DEBUG "data": { 05:15:22 DEBUG "resultType": "vector", 05:15:22 DEBUG "result": [ 05:15:22 DEBUG { 05:15:22 DEBUG "metric": { 05:15:22 DEBUG "pod": "ds-idrepo-0" 05:15:22 DEBUG }, 05:15:22 DEBUG "value": [ 05:15:22 DEBUG 05:15:22 DEBUG 05:15:22 DEBUG 05:15:22 DEBUG [...] 05:15:22 DEBUG 05:15:22 DEBUG 05:15:22 DEBUG 05:15:22 DEBUG "pod": "ds-cts-2" 05:15:22 DEBUG }, 05:15:22 DEBUG "value": [ 05:15:22 DEBUG 1712722522, 05:15:22 DEBUG "29.866666666666667" 05:15:22 DEBUG ] 05:15:22 DEBUG } 05:15:22 DEBUG ] 05:15:22 DEBUG } 05:15:22 DEBUG } 05:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722523" 05:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722523" 05:15:23 INFO [http_cmd]: http status code OK 05:15:23 DEBUG --- status code --- 05:15:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:23 DEBUG --- http response --- 05:15:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:15:23 INFO [http_cmd]: http status code OK 05:15:23 DEBUG ----- output ----- 05:15:23 DEBUG { 05:15:23 DEBUG "status": "success", 05:15:23 DEBUG "data": { 05:15:23 DEBUG "resultType": "vector", 05:15:23 DEBUG "result": [ 05:15:23 DEBUG { 05:15:23 DEBUG "metric": { 05:15:23 DEBUG "instance": "10.142.0.25:9100" 05:15:23 DEBUG }, 05:15:23 DEBUG "value": [ 05:15:23 DEBUG 05:15:23 DEBUG 05:15:23 DEBUG 05:15:23 DEBUG [...] 05:15:23 DEBUG 05:15:23 DEBUG 05:15:23 DEBUG 05:15:23 DEBUG "instance": "10.142.0.77:9100" 05:15:23 DEBUG }, 05:15:23 DEBUG "value": [ 05:15:23 DEBUG 1712722523, 05:15:23 DEBUG "37641455.2" 05:15:23 DEBUG ] 05:15:23 DEBUG } 05:15:23 DEBUG ] 05:15:23 DEBUG } 05:15:23 DEBUG } 05:15:23 DEBUG --- status code --- 05:15:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:23 DEBUG --- http response --- 05:15:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:15:23 DEBUG ----- output ----- 05:15:23 DEBUG { 05:15:23 DEBUG "status": "success", 05:15:23 DEBUG "data": { 05:15:23 DEBUG "resultType": "vector", 05:15:23 DEBUG "result": [ 05:15:23 DEBUG { 05:15:23 DEBUG "metric": { 05:15:23 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:15:23 DEBUG }, 05:15:23 DEBUG "value": [ 05:15:23 DEBUG 05:15:23 DEBUG 05:15:23 DEBUG 05:15:23 DEBUG [...] 05:15:23 DEBUG 05:15:23 DEBUG 05:15:23 DEBUG 05:15:23 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:15:23 DEBUG }, 05:15:23 DEBUG "value": [ 05:15:23 DEBUG 1712722523, 05:15:23 DEBUG "0" 05:15:23 DEBUG ] 05:15:23 DEBUG } 05:15:23 DEBUG ] 05:15:23 DEBUG } 05:15:23 DEBUG } 05:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722523" 05:15:23 INFO [http_cmd]: http status code OK 05:15:23 DEBUG --- status code --- 05:15:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:23 DEBUG --- http response --- 05:15:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:15:23 DEBUG ----- output ----- 05:15:23 DEBUG { 05:15:23 DEBUG "status": "success", 05:15:23 DEBUG "data": { 05:15:23 DEBUG "resultType": "vector", 05:15:23 DEBUG "result": [ 05:15:23 DEBUG { 05:15:23 DEBUG "metric": { 05:15:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:15:23 DEBUG }, 05:15:23 DEBUG "value": [ 05:15:23 DEBUG 05:15:23 DEBUG 05:15:23 DEBUG 05:15:23 DEBUG [...] 05:15:23 DEBUG 05:15:23 DEBUG 05:15:23 DEBUG 05:15:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:15:23 DEBUG }, 05:15:23 DEBUG "value": [ 05:15:23 DEBUG 1712722523, 05:15:23 DEBUG "227188.5" 05:15:23 DEBUG ] 05:15:23 DEBUG } 05:15:23 DEBUG ] 05:15:23 DEBUG } 05:15:23 DEBUG } 05:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722523" 05:15:23 INFO [http_cmd]: http status code OK 05:15:23 DEBUG --- status code --- 05:15:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:23 DEBUG --- http response --- 05:15:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:15:23 DEBUG ----- output ----- 05:15:23 DEBUG { 05:15:23 DEBUG "status": "success", 05:15:23 DEBUG "data": { 05:15:23 DEBUG "resultType": "vector", 05:15:23 DEBUG "result": [ 05:15:23 DEBUG { 05:15:23 DEBUG "metric": { 05:15:23 DEBUG "pod": "ds-idrepo-0" 05:15:23 DEBUG }, 05:15:23 DEBUG "value": [ 05:15:23 DEBUG 05:15:23 DEBUG 05:15:23 DEBUG 05:15:23 DEBUG [...] 05:15:23 DEBUG 05:15:23 DEBUG 05:15:23 DEBUG 05:15:23 DEBUG "pod": "ds-idrepo-1" 05:15:23 DEBUG }, 05:15:23 DEBUG "value": [ 05:15:23 DEBUG 1712722523, 05:15:23 DEBUG "0" 05:15:23 DEBUG ] 05:15:23 DEBUG } 05:15:23 DEBUG ] 05:15:23 DEBUG } 05:15:23 DEBUG } 05:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722523" 05:15:23 INFO [http_cmd]: http status code OK 05:15:23 DEBUG --- status code --- 05:15:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:23 DEBUG --- http response --- 05:15:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:15:23 DEBUG ----- output ----- 05:15:23 DEBUG { 05:15:23 DEBUG "status": "success", 05:15:23 DEBUG "data": { 05:15:23 DEBUG "resultType": "vector", 05:15:23 DEBUG "result": [ 05:15:23 DEBUG { 05:15:23 DEBUG "metric": { 05:15:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:15:23 DEBUG }, 05:15:23 DEBUG "value": [ 05:15:23 DEBUG 05:15:23 DEBUG 05:15:23 DEBUG 05:15:23 DEBUG [...] 05:15:23 DEBUG 05:15:23 DEBUG 05:15:23 DEBUG 05:15:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:15:23 DEBUG }, 05:15:23 DEBUG "value": [ 05:15:23 DEBUG 1712722523, 05:15:23 DEBUG "1262.9666666666667" 05:15:23 DEBUG ] 05:15:23 DEBUG } 05:15:23 DEBUG ] 05:15:23 DEBUG } 05:15:23 DEBUG } 05:15:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722524" 05:15:24 INFO [http_cmd]: http status code OK 05:15:24 DEBUG --- status code --- 05:15:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:24 DEBUG --- http response --- 05:15:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:15:24 DEBUG ----- output ----- 05:15:24 DEBUG { 05:15:24 DEBUG "status": "success", 05:15:24 DEBUG "data": { 05:15:24 DEBUG "resultType": "vector", 05:15:24 DEBUG "result": [ 05:15:24 DEBUG { 05:15:24 DEBUG "metric": { 05:15:24 DEBUG "instance": "10.142.0.25:9100" 05:15:24 DEBUG }, 05:15:24 DEBUG "value": [ 05:15:24 DEBUG 05:15:24 DEBUG 05:15:24 DEBUG 05:15:24 DEBUG [...] 05:15:24 DEBUG 05:15:24 DEBUG 05:15:24 DEBUG 05:15:24 DEBUG "instance": "10.142.0.77:9100" 05:15:24 DEBUG }, 05:15:24 DEBUG "value": [ 05:15:24 DEBUG 1712722524, 05:15:24 DEBUG "0.034033333333334304" 05:15:24 DEBUG ] 05:15:24 DEBUG } 05:15:24 DEBUG ] 05:15:24 DEBUG } 05:15:24 DEBUG } 05:15:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722524" 05:15:24 INFO [http_cmd]: http status code OK 05:15:24 DEBUG --- status code --- 05:15:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:24 DEBUG --- http response --- 05:15:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:15:24 DEBUG ----- output ----- 05:15:24 DEBUG { 05:15:24 DEBUG "status": "success", 05:15:24 DEBUG "data": { 05:15:24 DEBUG "resultType": "vector", 05:15:24 DEBUG "result": [ 05:15:24 DEBUG { 05:15:24 DEBUG "metric": { 05:15:24 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:15:24 DEBUG }, 05:15:24 DEBUG "value": [ 05:15:24 DEBUG 05:15:24 DEBUG 05:15:24 DEBUG 05:15:24 DEBUG [...] 05:15:24 DEBUG 05:15:24 DEBUG 05:15:24 DEBUG 05:15:24 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:15:24 DEBUG }, 05:15:24 DEBUG "value": [ 05:15:24 DEBUG 1712722524, 05:15:24 DEBUG "32345.55406480366" 05:15:24 DEBUG ] 05:15:24 DEBUG } 05:15:24 DEBUG ] 05:15:24 DEBUG } 05:15:24 DEBUG } 05:15:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712722525" 05:15:25 INFO [http_cmd]: http status code OK 05:15:25 DEBUG --- status code --- 05:15:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:25 DEBUG --- http response --- 05:15:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:15:25 DEBUG ----- output ----- 05:15:25 DEBUG { 05:15:25 DEBUG "status": "success", 05:15:25 DEBUG "data": { 05:15:25 DEBUG "resultType": "vector", 05:15:25 DEBUG "result": [ 05:15:25 DEBUG { 05:15:25 DEBUG "metric": { 05:15:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:15:25 DEBUG }, 05:15:25 DEBUG "value": [ 05:15:25 DEBUG 05:15:25 DEBUG 05:15:25 DEBUG 05:15:25 DEBUG [...] 05:15:25 DEBUG 05:15:25 DEBUG 05:15:25 DEBUG 05:15:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:15:25 DEBUG }, 05:15:25 DEBUG "value": [ 05:15:25 DEBUG 1712722525, 05:15:25 DEBUG "3.600900666238707" 05:15:25 DEBUG ] 05:15:25 DEBUG } 05:15:25 DEBUG ] 05:15:25 DEBUG } 05:15:25 DEBUG } 05:15:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712722525" 05:15:25 INFO [http_cmd]: http status code OK 05:15:25 DEBUG --- status code --- 05:15:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:25 DEBUG --- http response --- 05:15:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:15:25 DEBUG ----- output ----- 05:15:25 DEBUG { 05:15:25 DEBUG "status": "success", 05:15:25 DEBUG "data": { 05:15:25 DEBUG "resultType": "vector", 05:15:25 DEBUG "result": [ 05:15:25 DEBUG { 05:15:25 DEBUG "metric": { 05:15:25 DEBUG "pod": "ds-idrepo-0" 05:15:25 DEBUG }, 05:15:25 DEBUG "value": [ 05:15:25 DEBUG 05:15:25 DEBUG 05:15:25 DEBUG 05:15:25 DEBUG [...] 05:15:25 DEBUG 05:15:25 DEBUG 05:15:25 DEBUG 05:15:25 DEBUG "pod": "ds-cts-2" 05:15:25 DEBUG }, 05:15:25 DEBUG "value": [ 05:15:25 DEBUG 1712722525, 05:15:25 DEBUG "0" 05:15:25 DEBUG ] 05:15:25 DEBUG } 05:15:25 DEBUG ] 05:15:25 DEBUG } 05:15:25 DEBUG } 05:15:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722525" 05:15:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722525" 05:15:25 INFO [http_cmd]: http status code OK 05:15:25 DEBUG --- status code --- 05:15:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:25 DEBUG --- http response --- 05:15:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:15:25 DEBUG ----- output ----- 05:15:25 DEBUG { 05:15:25 DEBUG "status": "success", 05:15:25 DEBUG "data": { 05:15:25 DEBUG "resultType": "vector", 05:15:25 DEBUG "result": [ 05:15:25 DEBUG { 05:15:25 DEBUG "metric": { 05:15:25 DEBUG "pod": "ds-idrepo-2" 05:15:25 DEBUG }, 05:15:25 DEBUG "value": [ 05:15:25 DEBUG 05:15:25 DEBUG 05:15:25 DEBUG 05:15:25 DEBUG [...] 05:15:25 DEBUG 05:15:25 DEBUG 05:15:25 DEBUG 05:15:25 DEBUG "pod": "ds-cts-2" 05:15:25 DEBUG }, 05:15:25 DEBUG "value": [ 05:15:25 DEBUG 1712722525, 05:15:25 DEBUG "0.00002010777777777778" 05:15:25 DEBUG ] 05:15:25 DEBUG } 05:15:25 DEBUG ] 05:15:25 DEBUG } 05:15:25 DEBUG } 05:15:25 INFO [http_cmd]: http status code OK 05:15:25 DEBUG --- status code --- 05:15:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:25 DEBUG --- http response --- 05:15:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:15:25 DEBUG ----- output ----- 05:15:25 DEBUG { 05:15:25 DEBUG "status": "success", 05:15:25 DEBUG "data": { 05:15:25 DEBUG "resultType": "vector", 05:15:25 DEBUG "result": [ 05:15:25 DEBUG { 05:15:25 DEBUG "metric": { 05:15:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:15:25 DEBUG }, 05:15:25 DEBUG "value": [ 05:15:25 DEBUG 05:15:25 DEBUG 05:15:25 DEBUG 05:15:25 DEBUG [...] 05:15:25 DEBUG 05:15:25 DEBUG 05:15:25 DEBUG 05:15:25 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:15:25 DEBUG }, 05:15:25 DEBUG "value": [ 05:15:25 DEBUG 1712722525, 05:15:25 DEBUG "2.4118776135290587" 05:15:25 DEBUG ] 05:15:25 DEBUG } 05:15:25 DEBUG ] 05:15:25 DEBUG } 05:15:25 DEBUG } 05:15:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722525" 05:15:25 INFO [http_cmd]: http status code OK 05:15:25 DEBUG --- status code --- 05:15:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:25 DEBUG --- http response --- 05:15:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:15:25 DEBUG ----- output ----- 05:15:25 DEBUG { 05:15:25 DEBUG "status": "success", 05:15:25 DEBUG "data": { 05:15:25 DEBUG "resultType": "vector", 05:15:25 DEBUG "result": [ 05:15:25 DEBUG { 05:15:25 DEBUG "metric": { 05:15:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:15:25 DEBUG }, 05:15:25 DEBUG "value": [ 05:15:25 DEBUG 05:15:25 DEBUG 05:15:25 DEBUG 05:15:25 DEBUG [...] 05:15:25 DEBUG 05:15:25 DEBUG 05:15:25 DEBUG 05:15:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:15:25 DEBUG }, 05:15:25 DEBUG "value": [ 05:15:25 DEBUG 1712722525, 05:15:25 DEBUG "1263.1" 05:15:25 DEBUG ] 05:15:25 DEBUG } 05:15:25 DEBUG ] 05:15:25 DEBUG } 05:15:25 DEBUG } 05:15:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712722525" 05:15:25 INFO [http_cmd]: http status code OK 05:15:25 DEBUG --- status code --- 05:15:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:25 DEBUG --- http response --- 05:15:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:15:25 DEBUG ----- output ----- 05:15:25 DEBUG { 05:15:25 DEBUG "status": "success", 05:15:25 DEBUG "data": { 05:15:25 DEBUG "resultType": "vector", 05:15:25 DEBUG "result": [ 05:15:25 DEBUG { 05:15:25 DEBUG "metric": { 05:15:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:15:25 DEBUG }, 05:15:25 DEBUG "value": [ 05:15:25 DEBUG 05:15:25 DEBUG 05:15:25 DEBUG 05:15:25 DEBUG [...] 05:15:25 DEBUG 05:15:25 DEBUG 05:15:25 DEBUG 05:15:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:15:25 DEBUG }, 05:15:25 DEBUG "value": [ 05:15:25 DEBUG 1712722525, 05:15:25 DEBUG "0.04" 05:15:25 DEBUG ] 05:15:25 DEBUG } 05:15:25 DEBUG ] 05:15:25 DEBUG } 05:15:25 DEBUG } 05:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722526" 05:15:26 INFO [http_cmd]: http status code OK 05:15:26 DEBUG --- status code --- 05:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:26 DEBUG --- http response --- 05:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:15:26 DEBUG ----- output ----- 05:15:26 DEBUG { 05:15:26 DEBUG "status": "success", 05:15:26 DEBUG "data": { 05:15:26 DEBUG "resultType": "vector", 05:15:26 DEBUG "result": [ 05:15:26 DEBUG { 05:15:26 DEBUG "metric": { 05:15:26 DEBUG "pod": "ds-idrepo-0" 05:15:26 DEBUG }, 05:15:26 DEBUG "value": [ 05:15:26 DEBUG 05:15:26 DEBUG 05:15:26 DEBUG 05:15:26 DEBUG [...] 05:15:26 DEBUG 05:15:26 DEBUG 05:15:26 DEBUG 05:15:26 DEBUG "pod": "ds-idrepo-1" 05:15:26 DEBUG }, 05:15:26 DEBUG "value": [ 05:15:26 DEBUG 1712722526, 05:15:26 DEBUG "0" 05:15:26 DEBUG ] 05:15:26 DEBUG } 05:15:26 DEBUG ] 05:15:26 DEBUG } 05:15:26 DEBUG } 05:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722526" 05:15:26 INFO [http_cmd]: http status code OK 05:15:26 DEBUG --- status code --- 05:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:26 DEBUG --- http response --- 05:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:15:26 DEBUG ----- output ----- 05:15:26 DEBUG { 05:15:26 DEBUG "status": "success", 05:15:26 DEBUG "data": { 05:15:26 DEBUG "resultType": "vector", 05:15:26 DEBUG "result": [ 05:15:26 DEBUG { 05:15:26 DEBUG "metric": { 05:15:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:15:26 DEBUG }, 05:15:26 DEBUG "value": [ 05:15:26 DEBUG 05:15:26 DEBUG 05:15:26 DEBUG 05:15:26 DEBUG [...] 05:15:26 DEBUG 05:15:26 DEBUG 05:15:26 DEBUG 05:15:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:15:26 DEBUG }, 05:15:26 DEBUG "value": [ 05:15:26 DEBUG 1712722526, 05:15:26 DEBUG "1247.9333333333334" 05:15:26 DEBUG ] 05:15:26 DEBUG } 05:15:26 DEBUG ] 05:15:26 DEBUG } 05:15:26 DEBUG } 05:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722526" 05:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712722526" 05:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722526" 05:15:26 INFO [http_cmd]: http status code OK 05:15:26 DEBUG --- status code --- 05:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:26 DEBUG --- http response --- 05:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:15:26 DEBUG ----- output ----- 05:15:26 DEBUG { 05:15:26 DEBUG "status": "success", 05:15:26 DEBUG "data": { 05:15:26 DEBUG "resultType": "vector", 05:15:26 DEBUG "result": [ 05:15:26 DEBUG { 05:15:26 DEBUG "metric": { 05:15:26 DEBUG "pod": "ds-idrepo-0" 05:15:26 DEBUG }, 05:15:26 DEBUG "value": [ 05:15:26 DEBUG 05:15:26 DEBUG 05:15:26 INFO [http_cmd]: http status code OK 05:15:26 DEBUG 05:15:26 DEBUG --- status code --- 05:15:26 DEBUG [...] 05:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:26 DEBUG 05:15:26 DEBUG --- http response --- 05:15:26 DEBUG 05:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:15:26 DEBUG 05:15:26 DEBUG ----- output ----- 05:15:26 DEBUG "pod": "ds-idrepo-1" 05:15:26 DEBUG { 05:15:26 DEBUG }, 05:15:26 DEBUG "status": "success", 05:15:26 DEBUG "value": [ 05:15:26 DEBUG "data": { 05:15:26 DEBUG 1712722526, 05:15:26 DEBUG "resultType": "vector", 05:15:26 DEBUG "result": [ 05:15:26 DEBUG "0" 05:15:26 DEBUG { 05:15:26 DEBUG ] 05:15:26 DEBUG "metric": { 05:15:26 DEBUG } 05:15:26 DEBUG ] 05:15:26 DEBUG } 05:15:26 DEBUG } 05:15:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:15:26 DEBUG }, 05:15:26 DEBUG "value": [ 05:15:26 DEBUG 05:15:26 DEBUG 05:15:26 DEBUG 05:15:26 DEBUG [...] 05:15:26 DEBUG 05:15:26 INFO [http_cmd]: http status code OK 05:15:26 DEBUG 05:15:26 DEBUG --- status code --- 05:15:26 DEBUG 05:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:15:26 DEBUG --- http response --- 05:15:26 DEBUG }, 05:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:15:26 DEBUG "value": [ 05:15:26 DEBUG ----- output ----- 05:15:26 DEBUG 1712722526, 05:15:26 DEBUG { 05:15:26 DEBUG "2487943168" 05:15:26 DEBUG "status": "success", 05:15:26 DEBUG ] 05:15:26 DEBUG "data": { 05:15:26 DEBUG } 05:15:26 DEBUG "resultType": "vector", 05:15:26 DEBUG ] 05:15:26 DEBUG "result": [ 05:15:26 DEBUG } 05:15:26 DEBUG { 05:15:26 DEBUG } 05:15:26 DEBUG "metric": { 05:15:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:15:26 DEBUG }, 05:15:26 DEBUG "value": [ 05:15:26 DEBUG 05:15:26 DEBUG 05:15:26 DEBUG 05:15:26 DEBUG [...] 05:15:26 DEBUG 05:15:26 DEBUG 05:15:26 DEBUG 05:15:26 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:15:26 DEBUG }, 05:15:26 DEBUG "value": [ 05:15:26 DEBUG 1712722526, 05:15:26 DEBUG "0" 05:15:26 DEBUG ] 05:15:26 DEBUG } 05:15:26 DEBUG ] 05:15:26 DEBUG } 05:15:26 DEBUG } 05:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722526" 05:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722526" 05:15:26 INFO [http_cmd]: http status code OK 05:15:26 DEBUG --- status code --- 05:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:26 DEBUG --- http response --- 05:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:15:26 DEBUG ----- output ----- 05:15:26 INFO [http_cmd]: http status code OK 05:15:26 DEBUG { 05:15:26 DEBUG --- status code --- 05:15:26 DEBUG "status": "success", 05:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:15:26 DEBUG "data": { 05:15:26 DEBUG --- http response --- 05:15:26 DEBUG "resultType": "vector", 05:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:15:26 DEBUG "result": [ 05:15:26 DEBUG ----- output ----- 05:15:26 DEBUG { 05:15:26 DEBUG { 05:15:26 DEBUG "metric": { 05:15:26 DEBUG "status": "success", 05:15:26 DEBUG "instance": "10.142.0.25:9100" 05:15:26 DEBUG }, 05:15:26 DEBUG "data": { 05:15:26 DEBUG "value": [ 05:15:26 DEBUG "resultType": "vector", 05:15:26 DEBUG 05:15:26 DEBUG "result": [ 05:15:26 DEBUG 05:15:26 DEBUG { 05:15:26 DEBUG 05:15:26 DEBUG "metric": { 05:15:26 DEBUG [...] 05:15:26 DEBUG "instance": "10.142.0.25:9100" 05:15:26 DEBUG 05:15:26 DEBUG }, 05:15:26 DEBUG 05:15:26 DEBUG "value": [ 05:15:26 DEBUG 05:15:26 DEBUG 05:15:26 DEBUG 05:15:26 DEBUG "instance": "10.142.0.77:9100" 05:15:26 DEBUG 05:15:26 DEBUG }, 05:15:26 DEBUG [...] 05:15:26 DEBUG "value": [ 05:15:26 DEBUG 05:15:26 DEBUG 1712722526, 05:15:26 DEBUG 05:15:26 DEBUG "37642893.46666667" 05:15:26 DEBUG 05:15:26 DEBUG ] 05:15:26 DEBUG "instance": "10.142.0.77:9100" 05:15:26 DEBUG } 05:15:26 DEBUG }, 05:15:26 DEBUG ] 05:15:26 DEBUG "value": [ 05:15:26 DEBUG } 05:15:26 DEBUG 1712722526, 05:15:26 DEBUG } 05:15:26 DEBUG "0.013999999999999993" 05:15:26 DEBUG ] 05:15:26 DEBUG } 05:15:26 DEBUG ] 05:15:26 DEBUG } 05:15:26 DEBUG } 05:15:40 INFO 05:15:40 INFO [loop_until]: kubectl --namespace=xlou top pods 05:15:40 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:15:40 INFO [loop_until]: OK (rc = 0) 05:15:40 DEBUG --- stdout --- 05:15:40 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 8923m 8809Mi am-654c4fcbcf-9scrj 8543m 8418Mi am-654c4fcbcf-v5qb5 7346m 8397Mi ds-cts-0 7078m 12950Mi ds-cts-1 7504m 13128Mi ds-cts-2 6830m 13291Mi ds-idrepo-0 9563m 5621Mi ds-idrepo-1 318m 5668Mi ds-idrepo-2 344m 5698Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2212Mi lodemon-b776b7d88-thrqf 4m 70Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3524m 1651Mi 05:15:40 DEBUG --- stderr --- 05:15:40 DEBUG 05:15:46 INFO 05:15:46 INFO [loop_until]: kubectl --namespace=xlou top node 05:15:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:15:46 INFO [loop_until]: OK (rc = 0) 05:15:46 DEBUG --- stdout --- 05:15:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 96m 0% 2306Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8170m 51% 9271Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 80m 0% 3255Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10833m 68% 6505Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9249m 58% 9309Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9902m 62% 9647Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 76m 0% 853Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9781m 61% 6276Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 7377m 46% 13600Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 7092m 44% 13888Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 405m 2% 6355Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7453m 46% 13621Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 365m 2% 6325Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3701m 23% 5347Mi 9% 05:15:46 DEBUG --- stderr --- 05:15:46 DEBUG 05:16:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712722582" 05:16:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722582" 05:16:22 ERROR [http_cmd]: ERROR 05:16:22 INFO [http_cmd]: http status code OK 05:16:22 ERROR --- status code --- 05:16:22 DEBUG --- status code --- 05:16:22 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:16:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:22 ERROR --- http response --- 05:16:22 DEBUG --- http response --- 05:16:22 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:16:22 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:16:22 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:16:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722582" 05:16:22 INFO [http_cmd]: http status code OK 05:16:22 DEBUG --- status code --- 05:16:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:22 DEBUG --- http response --- 05:16:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:16:22 DEBUG ----- output ----- 05:16:22 DEBUG { 05:16:22 DEBUG "status": "success", 05:16:22 DEBUG "data": { 05:16:22 DEBUG "resultType": "vector", 05:16:22 DEBUG "result": [ 05:16:22 DEBUG { 05:16:22 DEBUG "metric": { 05:16:22 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:16:22 DEBUG }, 05:16:22 DEBUG "value": [ 05:16:22 DEBUG 05:16:22 DEBUG 05:16:22 DEBUG 05:16:22 DEBUG [...] 05:16:22 DEBUG 05:16:22 DEBUG 05:16:22 DEBUG 05:16:22 DEBUG "pod": "am-654c4fcbcf-487cv" 05:16:22 DEBUG }, 05:16:22 DEBUG "value": [ 05:16:22 DEBUG 1712722582, 05:16:22 DEBUG "0" 05:16:22 DEBUG ] 05:16:22 DEBUG } 05:16:22 DEBUG ] 05:16:22 DEBUG } 05:16:22 DEBUG } 05:16:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722582" 05:16:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722582" 05:16:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722582" 05:16:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722582" 05:16:22 INFO [http_cmd]: http status code OK 05:16:22 INFO [http_cmd]: http status code OK 05:16:22 DEBUG --- status code --- 05:16:22 DEBUG --- status code --- 05:16:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:22 DEBUG --- http response --- 05:16:22 DEBUG --- http response --- 05:16:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:16:22 DEBUG ----- output ----- 05:16:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:16:22 INFO [http_cmd]: http status code OK 05:16:22 DEBUG { 05:16:22 DEBUG ----- output ----- 05:16:22 DEBUG --- status code --- 05:16:22 DEBUG "status": "success", 05:16:22 DEBUG { 05:16:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:22 DEBUG "data": { 05:16:22 DEBUG "status": "success", 05:16:22 DEBUG --- http response --- 05:16:22 DEBUG "resultType": "vector", 05:16:22 DEBUG "data": { 05:16:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:16:22 DEBUG "result": [ 05:16:22 DEBUG { 05:16:22 DEBUG ----- output ----- 05:16:22 DEBUG "metric": { 05:16:22 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:16:22 DEBUG { 05:16:22 DEBUG }, 05:16:22 DEBUG "value": [ 05:16:22 DEBUG "status": "success", 05:16:22 DEBUG 05:16:22 DEBUG "resultType": "vector", 05:16:22 DEBUG "data": { 05:16:22 DEBUG 05:16:22 DEBUG "result": [ 05:16:22 DEBUG "resultType": "vector", 05:16:22 DEBUG 05:16:22 DEBUG [...] 05:16:22 DEBUG "result": [ 05:16:22 DEBUG { 05:16:22 DEBUG 05:16:22 DEBUG { 05:16:22 DEBUG "metric": { 05:16:22 DEBUG 05:16:22 DEBUG "metric": { 05:16:22 INFO [http_cmd]: http status code OK 05:16:22 DEBUG "instance": "10.142.0.25:9100" 05:16:22 DEBUG 05:16:22 DEBUG "pod": "ds-idrepo-0" 05:16:22 DEBUG }, 05:16:22 DEBUG "value": [ 05:16:22 DEBUG 05:16:22 DEBUG --- status code --- 05:16:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:22 DEBUG 05:16:22 DEBUG }, 05:16:22 DEBUG --- http response --- 05:16:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:16:22 DEBUG 05:16:22 DEBUG "value": [ 05:16:22 DEBUG ----- output ----- 05:16:22 DEBUG { 05:16:22 DEBUG "pod": "am-654c4fcbcf-487cv" 05:16:22 DEBUG }, 05:16:22 DEBUG "status": "success", 05:16:22 DEBUG [...] 05:16:22 DEBUG 05:16:22 DEBUG 05:16:22 DEBUG "data": { 05:16:22 DEBUG "value": [ 05:16:22 DEBUG 1712722582, 05:16:22 DEBUG 05:16:22 DEBUG "resultType": "vector", 05:16:22 DEBUG 05:16:22 DEBUG "1259.6" 05:16:22 DEBUG 05:16:22 DEBUG "result": [ 05:16:22 DEBUG 05:16:22 DEBUG "pod": "ds-cts-2" 05:16:22 DEBUG [...] 05:16:22 DEBUG { 05:16:22 DEBUG ] 05:16:22 DEBUG }, 05:16:22 DEBUG 05:16:22 DEBUG "metric": { 05:16:22 DEBUG } 05:16:22 DEBUG "value": [ 05:16:22 DEBUG 05:16:22 DEBUG "pod": "ds-idrepo-0" 05:16:22 DEBUG ] 05:16:22 DEBUG 1712722582, 05:16:22 DEBUG 05:16:22 DEBUG }, 05:16:22 DEBUG } 05:16:22 DEBUG "0" 05:16:22 DEBUG "instance": "10.142.0.77:9100" 05:16:22 DEBUG "value": [ 05:16:22 DEBUG } 05:16:22 DEBUG ] 05:16:22 DEBUG }, 05:16:22 DEBUG 05:16:22 DEBUG } 05:16:22 DEBUG "value": [ 05:16:22 DEBUG 05:16:22 DEBUG ] 05:16:22 DEBUG 1712722582, 05:16:22 DEBUG 05:16:22 DEBUG } 05:16:22 DEBUG "263101023232" 05:16:22 DEBUG ] 05:16:22 DEBUG } 05:16:22 DEBUG } 05:16:22 DEBUG ] 05:16:22 DEBUG [...] 05:16:22 DEBUG 05:16:22 DEBUG 05:16:22 DEBUG } 05:16:22 DEBUG 05:16:22 DEBUG "pod": "ds-cts-2" 05:16:22 DEBUG } 05:16:22 DEBUG }, 05:16:22 DEBUG "value": [ 05:16:22 DEBUG 1712722582, 05:16:22 DEBUG "29.9" 05:16:22 DEBUG ] 05:16:22 DEBUG } 05:16:22 DEBUG ] 05:16:22 DEBUG } 05:16:22 DEBUG } 05:16:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722583" 05:16:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722583" 05:16:23 INFO [http_cmd]: http status code OK 05:16:23 INFO [http_cmd]: http status code OK 05:16:23 DEBUG --- status code --- 05:16:23 DEBUG --- status code --- 05:16:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:23 DEBUG --- http response --- 05:16:23 DEBUG --- http response --- 05:16:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:16:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 05:16:23 DEBUG ----- output ----- 05:16:23 DEBUG ----- output ----- 05:16:23 DEBUG { 05:16:23 DEBUG { 05:16:23 DEBUG "status": "success", 05:16:23 DEBUG "status": "success", 05:16:23 DEBUG "data": { 05:16:23 DEBUG "data": { 05:16:23 DEBUG "resultType": "vector", 05:16:23 DEBUG "resultType": "vector", 05:16:23 DEBUG "result": [ 05:16:23 DEBUG "result": [ 05:16:23 DEBUG { 05:16:23 DEBUG { 05:16:23 DEBUG "metric": { 05:16:23 DEBUG "metric": { 05:16:23 DEBUG "instance": "10.142.0.25:9100" 05:16:23 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:16:23 DEBUG }, 05:16:23 DEBUG }, 05:16:23 DEBUG "value": [ 05:16:23 DEBUG "value": [ 05:16:23 DEBUG 05:16:23 DEBUG 05:16:23 DEBUG 05:16:23 DEBUG 05:16:23 DEBUG 05:16:23 DEBUG 05:16:23 DEBUG [...] 05:16:23 DEBUG [...] 05:16:23 DEBUG 05:16:23 DEBUG 05:16:23 DEBUG 05:16:23 DEBUG 05:16:23 DEBUG 05:16:23 DEBUG 05:16:23 DEBUG "instance": "10.142.0.77:9100" 05:16:23 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:16:23 DEBUG }, 05:16:23 DEBUG }, 05:16:23 DEBUG "value": [ 05:16:23 DEBUG "value": [ 05:16:23 DEBUG 1712722583, 05:16:23 DEBUG 1712722583, 05:16:23 DEBUG "37598997.96666667" 05:16:23 DEBUG "0" 05:16:23 DEBUG ] 05:16:23 DEBUG ] 05:16:23 DEBUG } 05:16:23 DEBUG } 05:16:23 DEBUG ] 05:16:23 DEBUG ] 05:16:23 DEBUG } 05:16:23 DEBUG } 05:16:23 DEBUG } 05:16:23 DEBUG } 05:16:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722583" 05:16:23 INFO [http_cmd]: http status code OK 05:16:23 DEBUG --- status code --- 05:16:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:23 DEBUG --- http response --- 05:16:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:16:23 DEBUG ----- output ----- 05:16:23 DEBUG { 05:16:23 DEBUG "status": "success", 05:16:23 DEBUG "data": { 05:16:23 DEBUG "resultType": "vector", 05:16:23 DEBUG "result": [ 05:16:23 DEBUG { 05:16:23 DEBUG "metric": { 05:16:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:16:23 DEBUG }, 05:16:23 DEBUG "value": [ 05:16:23 DEBUG 05:16:23 DEBUG 05:16:23 DEBUG 05:16:23 DEBUG [...] 05:16:23 DEBUG 05:16:23 DEBUG 05:16:23 DEBUG 05:16:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:16:23 DEBUG }, 05:16:23 DEBUG "value": [ 05:16:23 DEBUG 1712722583, 05:16:23 DEBUG "227514.5" 05:16:23 DEBUG ] 05:16:23 DEBUG } 05:16:23 DEBUG ] 05:16:23 DEBUG } 05:16:23 DEBUG } 05:16:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722583" 05:16:23 INFO [http_cmd]: http status code OK 05:16:23 DEBUG --- status code --- 05:16:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:23 DEBUG --- http response --- 05:16:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:16:23 DEBUG ----- output ----- 05:16:23 DEBUG { 05:16:23 DEBUG "status": "success", 05:16:23 DEBUG "data": { 05:16:23 DEBUG "resultType": "vector", 05:16:23 DEBUG "result": [ 05:16:23 DEBUG { 05:16:23 DEBUG "metric": { 05:16:23 DEBUG "pod": "ds-idrepo-0" 05:16:23 DEBUG }, 05:16:23 DEBUG "value": [ 05:16:23 DEBUG 05:16:23 DEBUG 05:16:23 DEBUG 05:16:23 DEBUG [...] 05:16:23 DEBUG 05:16:23 DEBUG 05:16:23 DEBUG 05:16:23 DEBUG "pod": "ds-idrepo-1" 05:16:23 DEBUG }, 05:16:23 DEBUG "value": [ 05:16:23 DEBUG 1712722583, 05:16:23 DEBUG "0" 05:16:23 DEBUG ] 05:16:23 DEBUG } 05:16:23 DEBUG ] 05:16:23 DEBUG } 05:16:23 DEBUG } 05:16:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722584" 05:16:24 INFO [http_cmd]: http status code OK 05:16:24 DEBUG --- status code --- 05:16:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:24 DEBUG --- http response --- 05:16:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:16:24 DEBUG ----- output ----- 05:16:24 DEBUG { 05:16:24 DEBUG "status": "success", 05:16:24 DEBUG "data": { 05:16:24 DEBUG "resultType": "vector", 05:16:24 DEBUG "result": [ 05:16:24 DEBUG { 05:16:24 DEBUG "metric": { 05:16:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:16:24 DEBUG }, 05:16:24 DEBUG "value": [ 05:16:24 DEBUG 05:16:24 DEBUG 05:16:24 DEBUG 05:16:24 DEBUG [...] 05:16:24 DEBUG 05:16:24 DEBUG 05:16:24 DEBUG 05:16:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:16:24 DEBUG }, 05:16:24 DEBUG "value": [ 05:16:24 DEBUG 1712722584, 05:16:24 DEBUG "1255.9333333333334" 05:16:24 DEBUG ] 05:16:24 DEBUG } 05:16:24 DEBUG ] 05:16:24 DEBUG } 05:16:24 DEBUG } 05:16:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722584" 05:16:24 INFO [http_cmd]: http status code OK 05:16:24 DEBUG --- status code --- 05:16:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:24 DEBUG --- http response --- 05:16:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:16:24 DEBUG ----- output ----- 05:16:24 DEBUG { 05:16:24 DEBUG "status": "success", 05:16:24 DEBUG "data": { 05:16:24 DEBUG "resultType": "vector", 05:16:24 DEBUG "result": [ 05:16:24 DEBUG { 05:16:24 DEBUG "metric": { 05:16:24 DEBUG "instance": "10.142.0.25:9100" 05:16:24 DEBUG }, 05:16:24 DEBUG "value": [ 05:16:24 DEBUG 05:16:24 DEBUG 05:16:24 DEBUG 05:16:24 DEBUG [...] 05:16:24 DEBUG 05:16:24 DEBUG 05:16:24 DEBUG 05:16:24 DEBUG "instance": "10.142.0.77:9100" 05:16:24 DEBUG }, 05:16:24 DEBUG "value": [ 05:16:24 DEBUG 1712722584, 05:16:24 DEBUG "0.038233333333330205" 05:16:24 DEBUG ] 05:16:24 DEBUG } 05:16:24 DEBUG ] 05:16:24 DEBUG } 05:16:24 DEBUG } 05:16:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722585" 05:16:25 INFO [http_cmd]: http status code OK 05:16:25 DEBUG --- status code --- 05:16:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:25 DEBUG --- http response --- 05:16:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:16:25 DEBUG ----- output ----- 05:16:25 DEBUG { 05:16:25 DEBUG "status": "success", 05:16:25 DEBUG "data": { 05:16:25 DEBUG "resultType": "vector", 05:16:25 DEBUG "result": [ 05:16:25 DEBUG { 05:16:25 DEBUG "metric": { 05:16:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:16:25 DEBUG }, 05:16:25 DEBUG "value": [ 05:16:25 DEBUG 05:16:25 DEBUG 05:16:25 DEBUG 05:16:25 DEBUG [...] 05:16:25 DEBUG 05:16:25 DEBUG 05:16:25 DEBUG 05:16:25 DEBUG "pod": "ds-idrepo-1" 05:16:25 DEBUG }, 05:16:25 DEBUG "value": [ 05:16:25 DEBUG 1712722585, 05:16:25 DEBUG "3924.039003250271" 05:16:25 DEBUG ] 05:16:25 DEBUG } 05:16:25 DEBUG ] 05:16:25 DEBUG } 05:16:25 DEBUG } 05:16:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712722585" 05:16:25 INFO [http_cmd]: http status code OK 05:16:25 DEBUG --- status code --- 05:16:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:25 DEBUG --- http response --- 05:16:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:16:25 DEBUG ----- output ----- 05:16:25 DEBUG { 05:16:25 DEBUG "status": "success", 05:16:25 DEBUG "data": { 05:16:25 DEBUG "resultType": "vector", 05:16:25 DEBUG "result": [ 05:16:25 DEBUG { 05:16:25 DEBUG "metric": { 05:16:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:16:25 DEBUG }, 05:16:25 DEBUG "value": [ 05:16:25 DEBUG 05:16:25 DEBUG 05:16:25 DEBUG 05:16:25 DEBUG [...] 05:16:25 DEBUG 05:16:25 DEBUG 05:16:25 DEBUG 05:16:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:16:25 DEBUG }, 05:16:25 DEBUG "value": [ 05:16:25 DEBUG 1712722585, 05:16:25 DEBUG "3.6287818156548526" 05:16:25 DEBUG ] 05:16:25 DEBUG } 05:16:25 DEBUG ] 05:16:25 DEBUG } 05:16:25 DEBUG } 05:16:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712722585" 05:16:25 INFO [http_cmd]: http status code OK 05:16:25 DEBUG --- status code --- 05:16:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:25 DEBUG --- http response --- 05:16:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:16:25 DEBUG ----- output ----- 05:16:25 DEBUG { 05:16:25 DEBUG "status": "success", 05:16:25 DEBUG "data": { 05:16:25 DEBUG "resultType": "vector", 05:16:25 DEBUG "result": [ 05:16:25 DEBUG { 05:16:25 DEBUG "metric": { 05:16:25 DEBUG "pod": "ds-idrepo-0" 05:16:25 DEBUG }, 05:16:25 DEBUG "value": [ 05:16:25 DEBUG 05:16:25 DEBUG 05:16:25 DEBUG 05:16:25 DEBUG [...] 05:16:25 DEBUG 05:16:25 DEBUG 05:16:25 DEBUG 05:16:25 DEBUG "pod": "ds-cts-2" 05:16:25 DEBUG }, 05:16:25 DEBUG "value": [ 05:16:25 DEBUG 1712722585, 05:16:25 DEBUG "0" 05:16:25 DEBUG ] 05:16:25 DEBUG } 05:16:25 DEBUG ] 05:16:25 DEBUG } 05:16:25 DEBUG } 05:16:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722585" 05:16:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722585" 05:16:25 INFO [http_cmd]: http status code OK 05:16:25 DEBUG --- status code --- 05:16:25 INFO [http_cmd]: http status code OK 05:16:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:25 DEBUG --- status code --- 05:16:25 DEBUG --- http response --- 05:16:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:16:25 DEBUG --- http response --- 05:16:25 DEBUG ----- output ----- 05:16:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:16:25 DEBUG { 05:16:25 DEBUG ----- output ----- 05:16:25 DEBUG "status": "success", 05:16:25 DEBUG { 05:16:25 DEBUG "data": { 05:16:25 DEBUG "status": "success", 05:16:25 DEBUG "resultType": "vector", 05:16:25 DEBUG "data": { 05:16:25 DEBUG "result": [ 05:16:25 DEBUG "resultType": "vector", 05:16:25 DEBUG { 05:16:25 DEBUG "result": [ 05:16:25 DEBUG "metric": { 05:16:25 DEBUG { 05:16:25 DEBUG "pod": "ds-idrepo-2" 05:16:25 DEBUG "metric": { 05:16:25 DEBUG }, 05:16:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:16:25 DEBUG "value": [ 05:16:25 DEBUG }, 05:16:25 DEBUG 05:16:25 DEBUG "value": [ 05:16:25 DEBUG 05:16:25 DEBUG 05:16:25 DEBUG 05:16:25 DEBUG 05:16:25 DEBUG [...] 05:16:25 DEBUG 05:16:25 DEBUG 05:16:25 DEBUG [...] 05:16:25 DEBUG 05:16:25 DEBUG 05:16:25 DEBUG 05:16:25 DEBUG 05:16:25 DEBUG "pod": "ds-cts-2" 05:16:25 DEBUG 05:16:25 DEBUG }, 05:16:25 DEBUG "pod": "ds-idrepo-1" 05:16:25 DEBUG "value": [ 05:16:25 DEBUG }, 05:16:25 DEBUG 1712722585, 05:16:25 DEBUG "value": [ 05:16:25 DEBUG "0.00002010777777777778" 05:16:25 DEBUG 1712722585, 05:16:25 DEBUG ] 05:16:25 DEBUG "0.19996089518276236" 05:16:25 DEBUG } 05:16:25 DEBUG ] 05:16:25 DEBUG ] 05:16:25 DEBUG } 05:16:25 DEBUG } 05:16:25 DEBUG ] 05:16:25 DEBUG } 05:16:25 DEBUG } 05:16:25 DEBUG } 05:16:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722585" 05:16:25 INFO [http_cmd]: http status code OK 05:16:25 DEBUG --- status code --- 05:16:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:25 DEBUG --- http response --- 05:16:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:16:25 DEBUG ----- output ----- 05:16:25 DEBUG { 05:16:25 DEBUG "status": "success", 05:16:25 DEBUG "data": { 05:16:25 DEBUG "resultType": "vector", 05:16:25 DEBUG "result": [ 05:16:25 DEBUG { 05:16:25 DEBUG "metric": { 05:16:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:16:25 DEBUG }, 05:16:25 DEBUG "value": [ 05:16:25 DEBUG 05:16:25 DEBUG 05:16:25 DEBUG 05:16:25 DEBUG [...] 05:16:25 DEBUG 05:16:25 DEBUG 05:16:25 DEBUG 05:16:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:16:25 DEBUG }, 05:16:25 DEBUG "value": [ 05:16:25 DEBUG 1712722585, 05:16:25 DEBUG "1256.0666666666666" 05:16:25 DEBUG ] 05:16:25 DEBUG } 05:16:25 DEBUG ] 05:16:25 DEBUG } 05:16:25 DEBUG } 05:16:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712722585" 05:16:25 INFO [http_cmd]: http status code OK 05:16:25 DEBUG --- status code --- 05:16:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:25 DEBUG --- http response --- 05:16:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:16:25 DEBUG ----- output ----- 05:16:25 DEBUG { 05:16:25 DEBUG "status": "success", 05:16:25 DEBUG "data": { 05:16:25 DEBUG "resultType": "vector", 05:16:25 DEBUG "result": [ 05:16:25 DEBUG { 05:16:25 DEBUG "metric": { 05:16:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:16:25 DEBUG }, 05:16:25 DEBUG "value": [ 05:16:25 DEBUG 05:16:25 DEBUG 05:16:25 DEBUG 05:16:25 DEBUG [...] 05:16:25 DEBUG 05:16:25 DEBUG 05:16:25 DEBUG 05:16:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:16:25 DEBUG }, 05:16:25 DEBUG "value": [ 05:16:25 DEBUG 1712722585, 05:16:25 DEBUG "0.04" 05:16:25 DEBUG ] 05:16:25 DEBUG } 05:16:25 DEBUG ] 05:16:25 DEBUG } 05:16:25 DEBUG } 05:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722586" 05:16:26 INFO [http_cmd]: http status code OK 05:16:26 DEBUG --- status code --- 05:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:26 DEBUG --- http response --- 05:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:16:26 DEBUG ----- output ----- 05:16:26 DEBUG { 05:16:26 DEBUG "status": "success", 05:16:26 DEBUG "data": { 05:16:26 DEBUG "resultType": "vector", 05:16:26 DEBUG "result": [ 05:16:26 DEBUG { 05:16:26 DEBUG "metric": { 05:16:26 DEBUG "pod": "ds-idrepo-0" 05:16:26 DEBUG }, 05:16:26 DEBUG "value": [ 05:16:26 DEBUG 05:16:26 DEBUG 05:16:26 DEBUG 05:16:26 DEBUG [...] 05:16:26 DEBUG 05:16:26 DEBUG 05:16:26 DEBUG 05:16:26 DEBUG "pod": "ds-idrepo-1" 05:16:26 DEBUG }, 05:16:26 DEBUG "value": [ 05:16:26 DEBUG 1712722586, 05:16:26 DEBUG "0" 05:16:26 DEBUG ] 05:16:26 DEBUG } 05:16:26 DEBUG ] 05:16:26 DEBUG } 05:16:26 DEBUG } 05:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722586" 05:16:26 INFO [http_cmd]: http status code OK 05:16:26 DEBUG --- status code --- 05:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:26 DEBUG --- http response --- 05:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:16:26 DEBUG ----- output ----- 05:16:26 DEBUG { 05:16:26 DEBUG "status": "success", 05:16:26 DEBUG "data": { 05:16:26 DEBUG "resultType": "vector", 05:16:26 DEBUG "result": [ 05:16:26 DEBUG { 05:16:26 DEBUG "metric": { 05:16:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:16:26 DEBUG }, 05:16:26 DEBUG "value": [ 05:16:26 DEBUG 05:16:26 DEBUG 05:16:26 DEBUG 05:16:26 DEBUG [...] 05:16:26 DEBUG 05:16:26 DEBUG 05:16:26 DEBUG 05:16:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:16:26 DEBUG }, 05:16:26 DEBUG "value": [ 05:16:26 DEBUG 1712722586, 05:16:26 DEBUG "1266.9666666666667" 05:16:26 DEBUG ] 05:16:26 DEBUG } 05:16:26 DEBUG ] 05:16:26 DEBUG } 05:16:26 DEBUG } 05:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722586" 05:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712722586" 05:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722586" 05:16:26 INFO [http_cmd]: http status code OK 05:16:26 DEBUG --- status code --- 05:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:26 DEBUG --- http response --- 05:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:16:26 DEBUG ----- output ----- 05:16:26 DEBUG { 05:16:26 DEBUG "status": "success", 05:16:26 DEBUG "data": { 05:16:26 DEBUG "resultType": "vector", 05:16:26 DEBUG "result": [ 05:16:26 DEBUG { 05:16:26 DEBUG "metric": { 05:16:26 DEBUG "pod": "ds-idrepo-0" 05:16:26 DEBUG }, 05:16:26 DEBUG "value": [ 05:16:26 DEBUG 05:16:26 DEBUG 05:16:26 DEBUG 05:16:26 DEBUG [...] 05:16:26 DEBUG 05:16:26 DEBUG 05:16:26 DEBUG 05:16:26 DEBUG "pod": "ds-idrepo-1" 05:16:26 DEBUG }, 05:16:26 DEBUG "value": [ 05:16:26 DEBUG 1712722586, 05:16:26 DEBUG "0" 05:16:26 DEBUG ] 05:16:26 DEBUG } 05:16:26 DEBUG ] 05:16:26 DEBUG } 05:16:26 DEBUG } 05:16:26 INFO [http_cmd]: http status code OK 05:16:26 DEBUG --- status code --- 05:16:26 INFO [http_cmd]: http status code OK 05:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:26 DEBUG --- status code --- 05:16:26 DEBUG --- http response --- 05:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:16:26 DEBUG --- http response --- 05:16:26 DEBUG ----- output ----- 05:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:16:26 DEBUG { 05:16:26 DEBUG ----- output ----- 05:16:26 DEBUG "status": "success", 05:16:26 DEBUG { 05:16:26 DEBUG "data": { 05:16:26 DEBUG "status": "success", 05:16:26 DEBUG "resultType": "vector", 05:16:26 DEBUG "data": { 05:16:26 DEBUG "result": [ 05:16:26 DEBUG "resultType": "vector", 05:16:26 DEBUG { 05:16:26 DEBUG "result": [ 05:16:26 DEBUG "metric": { 05:16:26 DEBUG { 05:16:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:16:26 DEBUG "metric": { 05:16:26 DEBUG }, 05:16:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:16:26 DEBUG "value": [ 05:16:26 DEBUG }, 05:16:26 DEBUG 05:16:26 DEBUG "value": [ 05:16:26 DEBUG 05:16:26 DEBUG 05:16:26 DEBUG 05:16:26 DEBUG 05:16:26 DEBUG [...] 05:16:26 DEBUG 05:16:26 DEBUG 05:16:26 DEBUG [...] 05:16:26 DEBUG 05:16:26 DEBUG 05:16:26 DEBUG 05:16:26 DEBUG 05:16:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:16:26 DEBUG 05:16:26 DEBUG }, 05:16:26 DEBUG "pod": "ds-idrepo-1" 05:16:26 DEBUG "value": [ 05:16:26 DEBUG }, 05:16:26 DEBUG 1712722586, 05:16:26 DEBUG "value": [ 05:16:26 DEBUG "2488688640" 05:16:26 DEBUG 1712722586, 05:16:26 DEBUG ] 05:16:26 DEBUG "1790.1509570241965" 05:16:26 DEBUG } 05:16:26 DEBUG ] 05:16:26 DEBUG ] 05:16:26 DEBUG } 05:16:26 DEBUG } 05:16:26 DEBUG ] 05:16:26 DEBUG } 05:16:26 DEBUG } 05:16:26 DEBUG } 05:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722586" 05:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722586" 05:16:26 INFO [http_cmd]: http status code OK 05:16:26 DEBUG --- status code --- 05:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:26 DEBUG --- http response --- 05:16:26 INFO [http_cmd]: http status code OK 05:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:16:26 DEBUG --- status code --- 05:16:26 DEBUG ----- output ----- 05:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:16:26 DEBUG { 05:16:26 DEBUG "status": "success", 05:16:26 DEBUG --- http response --- 05:16:26 DEBUG "data": { 05:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:16:26 DEBUG "resultType": "vector", 05:16:26 DEBUG ----- output ----- 05:16:26 DEBUG "result": [ 05:16:26 DEBUG { 05:16:26 DEBUG { 05:16:26 DEBUG "status": "success", 05:16:26 DEBUG "metric": { 05:16:26 DEBUG "data": { 05:16:26 DEBUG "instance": "10.142.0.25:9100" 05:16:26 DEBUG "resultType": "vector", 05:16:26 DEBUG }, 05:16:26 DEBUG "result": [ 05:16:26 DEBUG "value": [ 05:16:26 DEBUG { 05:16:26 DEBUG 05:16:26 DEBUG "metric": { 05:16:26 DEBUG 05:16:26 DEBUG "instance": "10.142.0.25:9100" 05:16:26 DEBUG 05:16:26 DEBUG }, 05:16:26 DEBUG [...] 05:16:26 DEBUG "value": [ 05:16:26 DEBUG 05:16:26 DEBUG 05:16:26 DEBUG 05:16:26 DEBUG 05:16:26 DEBUG 05:16:26 DEBUG 05:16:26 DEBUG "instance": "10.142.0.77:9100" 05:16:26 DEBUG [...] 05:16:26 DEBUG }, 05:16:26 DEBUG 05:16:26 DEBUG "value": [ 05:16:26 DEBUG 05:16:26 DEBUG 1712722586, 05:16:26 DEBUG 05:16:26 DEBUG "37600418.266666666" 05:16:26 DEBUG "instance": "10.142.0.77:9100" 05:16:26 DEBUG ] 05:16:26 DEBUG }, 05:16:26 DEBUG } 05:16:26 DEBUG "value": [ 05:16:26 DEBUG ] 05:16:26 DEBUG 1712722586, 05:16:26 DEBUG } 05:16:26 DEBUG "0.018666666666666207" 05:16:26 DEBUG } 05:16:26 DEBUG ] 05:16:26 DEBUG } 05:16:26 DEBUG ] 05:16:26 DEBUG } 05:16:26 DEBUG } 05:16:40 INFO 05:16:40 INFO [loop_until]: kubectl --namespace=xlou top pods 05:16:40 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:16:40 INFO [loop_until]: OK (rc = 0) 05:16:40 DEBUG --- stdout --- 05:16:40 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 8964m 8809Mi am-654c4fcbcf-9scrj 8718m 8423Mi am-654c4fcbcf-v5qb5 7608m 8400Mi ds-cts-0 7350m 13260Mi ds-cts-1 7804m 13227Mi ds-cts-2 6764m 12766Mi ds-idrepo-0 9571m 5621Mi ds-idrepo-1 322m 5668Mi ds-idrepo-2 351m 5698Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2212Mi lodemon-b776b7d88-thrqf 8m 70Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3616m 1652Mi 05:16:40 DEBUG --- stderr --- 05:16:40 DEBUG 05:16:46 INFO 05:16:46 INFO [loop_until]: kubectl --namespace=xlou top node 05:16:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:16:46 INFO [loop_until]: OK (rc = 0) 05:16:46 DEBUG --- stdout --- 05:16:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 93m 0% 2305Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8258m 51% 9278Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 86m 0% 3268Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10811m 68% 6516Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9599m 60% 9309Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9541m 60% 9648Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 77m 0% 854Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9691m 60% 6277Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 7500m 47% 13825Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 6826m 42% 13387Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 399m 2% 6355Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7598m 47% 13876Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 377m 2% 6326Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3699m 23% 5346Mi 9% 05:16:46 DEBUG --- stderr --- 05:16:46 DEBUG 05:17:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722642" 05:17:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712722642" 05:17:22 INFO [http_cmd]: http status code OK 05:17:22 DEBUG --- status code --- 05:17:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:22 DEBUG --- http response --- 05:17:22 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:17:22 ERROR [http_cmd]: ERROR 05:17:22 ERROR --- status code --- 05:17:22 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:17:22 ERROR --- http response --- 05:17:22 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:17:22 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:17:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722642" 05:17:22 INFO [http_cmd]: http status code OK 05:17:22 DEBUG --- status code --- 05:17:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:22 DEBUG --- http response --- 05:17:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:17:22 DEBUG ----- output ----- 05:17:22 DEBUG { 05:17:22 DEBUG "status": "success", 05:17:22 DEBUG "data": { 05:17:22 DEBUG "resultType": "vector", 05:17:22 DEBUG "result": [ 05:17:22 DEBUG { 05:17:22 DEBUG "metric": { 05:17:22 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:17:22 DEBUG }, 05:17:22 DEBUG "value": [ 05:17:22 DEBUG 05:17:22 DEBUG 05:17:22 DEBUG 05:17:22 DEBUG [...] 05:17:22 DEBUG 05:17:22 DEBUG 05:17:22 DEBUG 05:17:22 DEBUG "pod": "am-654c4fcbcf-487cv" 05:17:22 DEBUG }, 05:17:22 DEBUG "value": [ 05:17:22 DEBUG 1712722642, 05:17:22 DEBUG "0" 05:17:22 DEBUG ] 05:17:22 DEBUG } 05:17:22 DEBUG ] 05:17:22 DEBUG } 05:17:22 DEBUG } 05:17:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722642" 05:17:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722642" 05:17:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722642" 05:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722642" 05:17:23 INFO [http_cmd]: http status code OK 05:17:23 DEBUG --- status code --- 05:17:23 INFO [http_cmd]: http status code OK 05:17:23 INFO [http_cmd]: http status code OK 05:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:23 DEBUG --- status code --- 05:17:23 DEBUG --- status code --- 05:17:23 DEBUG --- http response --- 05:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:17:23 DEBUG --- http response --- 05:17:23 DEBUG --- http response --- 05:17:23 INFO [http_cmd]: http status code OK 05:17:23 DEBUG ----- output ----- 05:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:17:23 DEBUG --- status code --- 05:17:23 DEBUG { 05:17:23 DEBUG ----- output ----- 05:17:23 DEBUG ----- output ----- 05:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:23 DEBUG "status": "success", 05:17:23 DEBUG { 05:17:23 DEBUG { 05:17:23 DEBUG --- http response --- 05:17:23 DEBUG "data": { 05:17:23 DEBUG "resultType": "vector", 05:17:23 DEBUG "status": "success", 05:17:23 DEBUG "data": { 05:17:23 DEBUG "resultType": "vector", 05:17:23 DEBUG "result": [ 05:17:23 DEBUG { 05:17:23 DEBUG "metric": { 05:17:23 DEBUG "instance": "10.142.0.25:9100" 05:17:23 DEBUG }, 05:17:23 DEBUG "value": [ 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG [...] 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG "instance": "10.142.0.77:9100" 05:17:23 DEBUG }, 05:17:23 DEBUG "value": [ 05:17:23 DEBUG 1712722642, 05:17:23 DEBUG "265038705664" 05:17:23 DEBUG ] 05:17:23 DEBUG } 05:17:23 DEBUG ] 05:17:23 DEBUG } 05:17:23 DEBUG } 05:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:17:23 DEBUG ----- output ----- 05:17:23 DEBUG { 05:17:23 DEBUG "status": "success", 05:17:23 DEBUG "data": { 05:17:23 DEBUG "resultType": "vector", 05:17:23 DEBUG "result": [ 05:17:23 DEBUG { 05:17:23 DEBUG "metric": { 05:17:23 DEBUG "pod": "ds-idrepo-0" 05:17:23 DEBUG }, 05:17:23 DEBUG "value": [ 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG [...] 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG "pod": "ds-cts-2" 05:17:23 DEBUG }, 05:17:23 DEBUG "value": [ 05:17:23 DEBUG 1712722642, 05:17:23 DEBUG "29.666666666666668" 05:17:23 DEBUG ] 05:17:23 DEBUG } 05:17:23 DEBUG ] 05:17:23 DEBUG } 05:17:23 DEBUG } 05:17:23 DEBUG "status": "success", 05:17:23 DEBUG "data": { 05:17:23 DEBUG "resultType": "vector", 05:17:23 DEBUG "result": [ 05:17:23 DEBUG { 05:17:23 DEBUG "metric": { 05:17:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:17:23 DEBUG }, 05:17:23 DEBUG "value": [ 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG [...] 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:17:23 DEBUG }, 05:17:23 DEBUG "value": [ 05:17:23 DEBUG 1712722642, 05:17:23 DEBUG "1266.4666666666667" 05:17:23 DEBUG ] 05:17:23 DEBUG } 05:17:23 DEBUG ] 05:17:23 DEBUG } 05:17:23 DEBUG } 05:17:23 DEBUG "result": [ 05:17:23 DEBUG { 05:17:23 DEBUG "metric": { 05:17:23 DEBUG "pod": "ds-idrepo-0" 05:17:23 DEBUG }, 05:17:23 DEBUG "value": [ 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG [...] 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG "pod": "ds-cts-2" 05:17:23 DEBUG }, 05:17:23 DEBUG "value": [ 05:17:23 DEBUG 1712722642, 05:17:23 DEBUG "0" 05:17:23 DEBUG ] 05:17:23 DEBUG } 05:17:23 DEBUG ] 05:17:23 DEBUG } 05:17:23 DEBUG } 05:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722643" 05:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722643" 05:17:23 INFO [http_cmd]: http status code OK 05:17:23 DEBUG --- status code --- 05:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:23 DEBUG --- http response --- 05:17:23 INFO [http_cmd]: http status code OK 05:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 05:17:23 DEBUG --- status code --- 05:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:23 DEBUG --- http response --- 05:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:17:23 DEBUG ----- output ----- 05:17:23 DEBUG ----- output ----- 05:17:23 DEBUG { 05:17:23 DEBUG { 05:17:23 DEBUG "status": "success", 05:17:23 DEBUG "status": "success", 05:17:23 DEBUG "data": { 05:17:23 DEBUG "data": { 05:17:23 DEBUG "resultType": "vector", 05:17:23 DEBUG "resultType": "vector", 05:17:23 DEBUG "result": [ 05:17:23 DEBUG "result": [ 05:17:23 DEBUG { 05:17:23 DEBUG { 05:17:23 DEBUG "metric": { 05:17:23 DEBUG "metric": { 05:17:23 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:17:23 DEBUG "instance": "10.142.0.25:9100" 05:17:23 DEBUG }, 05:17:23 DEBUG "value": [ 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG [...] 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:17:23 DEBUG }, 05:17:23 DEBUG "value": [ 05:17:23 DEBUG 1712722643, 05:17:23 DEBUG "0" 05:17:23 DEBUG ] 05:17:23 DEBUG } 05:17:23 DEBUG ] 05:17:23 DEBUG } 05:17:23 DEBUG } 05:17:23 DEBUG }, 05:17:23 DEBUG "value": [ 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG [...] 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG "instance": "10.142.0.77:9100" 05:17:23 DEBUG }, 05:17:23 DEBUG "value": [ 05:17:23 DEBUG 1712722643, 05:17:23 DEBUG "37586017.3" 05:17:23 DEBUG ] 05:17:23 DEBUG } 05:17:23 DEBUG ] 05:17:23 DEBUG } 05:17:23 DEBUG } 05:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722643" 05:17:23 INFO [http_cmd]: http status code OK 05:17:23 DEBUG --- status code --- 05:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:23 DEBUG --- http response --- 05:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:17:23 DEBUG ----- output ----- 05:17:23 DEBUG { 05:17:23 DEBUG "status": "success", 05:17:23 DEBUG "data": { 05:17:23 DEBUG "resultType": "vector", 05:17:23 DEBUG "result": [ 05:17:23 DEBUG { 05:17:23 DEBUG "metric": { 05:17:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:17:23 DEBUG }, 05:17:23 DEBUG "value": [ 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG [...] 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:17:23 DEBUG }, 05:17:23 DEBUG "value": [ 05:17:23 DEBUG 1712722643, 05:17:23 DEBUG "227337" 05:17:23 DEBUG ] 05:17:23 DEBUG } 05:17:23 DEBUG ] 05:17:23 DEBUG } 05:17:23 DEBUG } 05:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722643" 05:17:23 INFO [http_cmd]: http status code OK 05:17:23 DEBUG --- status code --- 05:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:23 DEBUG --- http response --- 05:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:17:23 DEBUG ----- output ----- 05:17:23 DEBUG { 05:17:23 DEBUG "status": "success", 05:17:23 DEBUG "data": { 05:17:23 DEBUG "resultType": "vector", 05:17:23 DEBUG "result": [ 05:17:23 DEBUG { 05:17:23 DEBUG "metric": { 05:17:23 DEBUG "pod": "ds-idrepo-0" 05:17:23 DEBUG }, 05:17:23 DEBUG "value": [ 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG [...] 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG 05:17:23 DEBUG "pod": "ds-idrepo-1" 05:17:23 DEBUG }, 05:17:23 DEBUG "value": [ 05:17:23 DEBUG 1712722643, 05:17:23 DEBUG "0" 05:17:23 DEBUG ] 05:17:23 DEBUG } 05:17:23 DEBUG ] 05:17:23 DEBUG } 05:17:23 DEBUG } 05:17:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722644" 05:17:24 INFO [http_cmd]: http status code OK 05:17:24 DEBUG --- status code --- 05:17:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:24 DEBUG --- http response --- 05:17:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:17:24 DEBUG ----- output ----- 05:17:24 DEBUG { 05:17:24 DEBUG "status": "success", 05:17:24 DEBUG "data": { 05:17:24 DEBUG "resultType": "vector", 05:17:24 DEBUG "result": [ 05:17:24 DEBUG { 05:17:24 DEBUG "metric": { 05:17:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:17:24 DEBUG }, 05:17:24 DEBUG "value": [ 05:17:24 DEBUG 05:17:24 DEBUG 05:17:24 DEBUG 05:17:24 DEBUG [...] 05:17:24 DEBUG 05:17:24 DEBUG 05:17:24 DEBUG 05:17:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:17:24 DEBUG }, 05:17:24 DEBUG "value": [ 05:17:24 DEBUG 1712722644, 05:17:24 DEBUG "1269.5" 05:17:24 DEBUG ] 05:17:24 DEBUG } 05:17:24 DEBUG ] 05:17:24 DEBUG } 05:17:24 DEBUG } 05:17:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722644" 05:17:24 INFO [http_cmd]: http status code OK 05:17:24 DEBUG --- status code --- 05:17:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:24 DEBUG --- http response --- 05:17:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:17:24 DEBUG ----- output ----- 05:17:24 DEBUG { 05:17:24 DEBUG "status": "success", 05:17:24 DEBUG "data": { 05:17:24 DEBUG "resultType": "vector", 05:17:24 DEBUG "result": [ 05:17:24 DEBUG { 05:17:24 DEBUG "metric": { 05:17:24 DEBUG "instance": "10.142.0.25:9100" 05:17:24 DEBUG }, 05:17:24 DEBUG "value": [ 05:17:24 DEBUG 05:17:24 DEBUG 05:17:24 DEBUG 05:17:24 DEBUG [...] 05:17:24 DEBUG 05:17:24 DEBUG 05:17:24 DEBUG 05:17:24 DEBUG "instance": "10.142.0.77:9100" 05:17:24 DEBUG }, 05:17:24 DEBUG "value": [ 05:17:24 DEBUG 1712722644, 05:17:24 DEBUG "0.15536666666666693" 05:17:24 DEBUG ] 05:17:24 DEBUG } 05:17:24 DEBUG ] 05:17:24 DEBUG } 05:17:24 DEBUG } 05:17:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722645" 05:17:25 INFO [http_cmd]: http status code OK 05:17:25 DEBUG --- status code --- 05:17:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:25 DEBUG --- http response --- 05:17:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:17:25 DEBUG ----- output ----- 05:17:25 DEBUG { 05:17:25 DEBUG "status": "success", 05:17:25 DEBUG "data": { 05:17:25 DEBUG "resultType": "vector", 05:17:25 DEBUG "result": [ 05:17:25 DEBUG { 05:17:25 DEBUG "metric": { 05:17:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:17:25 DEBUG }, 05:17:25 DEBUG "value": [ 05:17:25 DEBUG 05:17:25 DEBUG 05:17:25 DEBUG 05:17:25 DEBUG [...] 05:17:25 DEBUG 05:17:25 DEBUG 05:17:25 DEBUG 05:17:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:17:25 DEBUG }, 05:17:25 DEBUG "value": [ 05:17:25 DEBUG 1712722645, 05:17:25 DEBUG "246667185.15025556" 05:17:25 DEBUG ] 05:17:25 DEBUG } 05:17:25 DEBUG ] 05:17:25 DEBUG } 05:17:25 DEBUG } 05:17:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712722645" 05:17:25 INFO [http_cmd]: http status code OK 05:17:25 DEBUG --- status code --- 05:17:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:25 DEBUG --- http response --- 05:17:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:17:25 DEBUG ----- output ----- 05:17:25 DEBUG { 05:17:25 DEBUG "status": "success", 05:17:25 DEBUG "data": { 05:17:25 DEBUG "resultType": "vector", 05:17:25 DEBUG "result": [ 05:17:25 DEBUG { 05:17:25 DEBUG "metric": { 05:17:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:17:25 DEBUG }, 05:17:25 DEBUG "value": [ 05:17:25 DEBUG 05:17:25 DEBUG 05:17:25 DEBUG 05:17:25 DEBUG [...] 05:17:25 DEBUG 05:17:25 DEBUG 05:17:25 DEBUG 05:17:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:17:25 DEBUG }, 05:17:25 DEBUG "value": [ 05:17:25 DEBUG 1712722645, 05:17:25 DEBUG "3.5902262952973274" 05:17:25 DEBUG ] 05:17:25 DEBUG } 05:17:25 DEBUG ] 05:17:25 DEBUG } 05:17:25 DEBUG } 05:17:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712722645" 05:17:25 INFO [http_cmd]: http status code OK 05:17:25 DEBUG --- status code --- 05:17:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:25 DEBUG --- http response --- 05:17:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:17:25 DEBUG ----- output ----- 05:17:25 DEBUG { 05:17:25 DEBUG "status": "success", 05:17:25 DEBUG "data": { 05:17:25 DEBUG "resultType": "vector", 05:17:25 DEBUG "result": [ 05:17:25 DEBUG { 05:17:25 DEBUG "metric": { 05:17:25 DEBUG "pod": "ds-idrepo-0" 05:17:25 DEBUG }, 05:17:25 DEBUG "value": [ 05:17:25 DEBUG 05:17:25 DEBUG 05:17:25 DEBUG 05:17:25 DEBUG [...] 05:17:25 DEBUG 05:17:25 DEBUG 05:17:25 DEBUG 05:17:25 DEBUG "pod": "ds-cts-2" 05:17:25 DEBUG }, 05:17:25 DEBUG "value": [ 05:17:25 DEBUG 1712722645, 05:17:25 DEBUG "0" 05:17:25 DEBUG ] 05:17:25 DEBUG } 05:17:25 DEBUG ] 05:17:25 DEBUG } 05:17:25 DEBUG } 05:17:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722645" 05:17:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722645" 05:17:25 INFO [http_cmd]: http status code OK 05:17:25 DEBUG --- status code --- 05:17:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:25 DEBUG --- http response --- 05:17:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:17:25 DEBUG ----- output ----- 05:17:25 DEBUG { 05:17:25 DEBUG "status": "success", 05:17:25 DEBUG "data": { 05:17:25 DEBUG "resultType": "vector", 05:17:25 INFO [http_cmd]: http status code OK 05:17:25 DEBUG "result": [ 05:17:25 DEBUG --- status code --- 05:17:25 DEBUG { 05:17:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:25 DEBUG "metric": { 05:17:25 DEBUG --- http response --- 05:17:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:17:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:17:25 DEBUG }, 05:17:25 DEBUG ----- output ----- 05:17:25 DEBUG "value": [ 05:17:25 DEBUG { 05:17:25 DEBUG "status": "success", 05:17:25 DEBUG "data": { 05:17:25 DEBUG "resultType": "vector", 05:17:25 DEBUG "result": [ 05:17:25 DEBUG 05:17:25 DEBUG { 05:17:25 DEBUG 05:17:25 DEBUG "metric": { 05:17:25 DEBUG 05:17:25 DEBUG "pod": "ds-idrepo-2" 05:17:25 DEBUG [...] 05:17:25 DEBUG }, 05:17:25 DEBUG "value": [ 05:17:25 DEBUG 05:17:25 DEBUG 05:17:25 DEBUG 05:17:25 DEBUG [...] 05:17:25 DEBUG 05:17:25 DEBUG 05:17:25 DEBUG 05:17:25 DEBUG "pod": "ds-cts-2" 05:17:25 DEBUG }, 05:17:25 DEBUG 05:17:25 DEBUG "value": [ 05:17:25 DEBUG 05:17:25 DEBUG 1712722645, 05:17:25 DEBUG 05:17:25 DEBUG "0.00013333333333333334" 05:17:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:17:25 DEBUG ] 05:17:25 DEBUG }, 05:17:25 DEBUG "value": [ 05:17:25 DEBUG } 05:17:25 DEBUG 1712722645, 05:17:25 DEBUG "3.750482384464226" 05:17:25 DEBUG ] 05:17:25 DEBUG ] 05:17:25 DEBUG } 05:17:25 DEBUG } 05:17:25 DEBUG } 05:17:25 DEBUG ] 05:17:25 DEBUG } 05:17:25 DEBUG } 05:17:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722645" 05:17:25 INFO [http_cmd]: http status code OK 05:17:25 DEBUG --- status code --- 05:17:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:25 DEBUG --- http response --- 05:17:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:17:25 DEBUG ----- output ----- 05:17:25 DEBUG { 05:17:25 DEBUG "status": "success", 05:17:25 DEBUG "data": { 05:17:25 DEBUG "resultType": "vector", 05:17:25 DEBUG "result": [ 05:17:25 DEBUG { 05:17:25 DEBUG "metric": { 05:17:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:17:25 DEBUG }, 05:17:25 DEBUG "value": [ 05:17:25 DEBUG 05:17:25 DEBUG 05:17:25 DEBUG 05:17:25 DEBUG [...] 05:17:25 DEBUG 05:17:25 DEBUG 05:17:25 DEBUG 05:17:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:17:25 DEBUG }, 05:17:25 DEBUG "value": [ 05:17:25 DEBUG 1712722645, 05:17:25 DEBUG "1269.4" 05:17:25 DEBUG ] 05:17:25 DEBUG } 05:17:25 DEBUG ] 05:17:25 DEBUG } 05:17:25 DEBUG } 05:17:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712722645" 05:17:25 INFO [http_cmd]: http status code OK 05:17:25 DEBUG --- status code --- 05:17:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:25 DEBUG --- http response --- 05:17:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:17:25 DEBUG ----- output ----- 05:17:25 DEBUG { 05:17:25 DEBUG "status": "success", 05:17:25 DEBUG "data": { 05:17:25 DEBUG "resultType": "vector", 05:17:25 DEBUG "result": [ 05:17:25 DEBUG { 05:17:25 DEBUG "metric": { 05:17:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:17:25 DEBUG }, 05:17:25 DEBUG "value": [ 05:17:25 DEBUG 05:17:25 DEBUG 05:17:25 DEBUG 05:17:25 DEBUG [...] 05:17:25 DEBUG 05:17:25 DEBUG 05:17:25 DEBUG 05:17:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:17:25 DEBUG }, 05:17:25 DEBUG "value": [ 05:17:25 DEBUG 1712722645, 05:17:25 DEBUG "0.04" 05:17:25 DEBUG ] 05:17:25 DEBUG } 05:17:25 DEBUG ] 05:17:25 DEBUG } 05:17:25 DEBUG } 05:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722646" 05:17:26 INFO [http_cmd]: http status code OK 05:17:26 DEBUG --- status code --- 05:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:26 DEBUG --- http response --- 05:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:17:26 DEBUG ----- output ----- 05:17:26 DEBUG { 05:17:26 DEBUG "status": "success", 05:17:26 DEBUG "data": { 05:17:26 DEBUG "resultType": "vector", 05:17:26 DEBUG "result": [ 05:17:26 DEBUG { 05:17:26 DEBUG "metric": { 05:17:26 DEBUG "pod": "ds-idrepo-0" 05:17:26 DEBUG }, 05:17:26 DEBUG "value": [ 05:17:26 DEBUG 05:17:26 DEBUG 05:17:26 DEBUG 05:17:26 DEBUG [...] 05:17:26 DEBUG 05:17:26 DEBUG 05:17:26 DEBUG 05:17:26 DEBUG "pod": "ds-idrepo-1" 05:17:26 DEBUG }, 05:17:26 DEBUG "value": [ 05:17:26 DEBUG 1712722646, 05:17:26 DEBUG "0" 05:17:26 DEBUG ] 05:17:26 DEBUG } 05:17:26 DEBUG ] 05:17:26 DEBUG } 05:17:26 DEBUG } 05:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722646" 05:17:26 INFO [http_cmd]: http status code OK 05:17:26 DEBUG --- status code --- 05:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:26 DEBUG --- http response --- 05:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:17:26 DEBUG ----- output ----- 05:17:26 DEBUG { 05:17:26 DEBUG "status": "success", 05:17:26 DEBUG "data": { 05:17:26 DEBUG "resultType": "vector", 05:17:26 DEBUG "result": [ 05:17:26 DEBUG { 05:17:26 DEBUG "metric": { 05:17:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:17:26 DEBUG }, 05:17:26 DEBUG "value": [ 05:17:26 DEBUG 05:17:26 DEBUG 05:17:26 DEBUG 05:17:26 DEBUG [...] 05:17:26 DEBUG 05:17:26 DEBUG 05:17:26 DEBUG 05:17:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:17:26 DEBUG }, 05:17:26 DEBUG "value": [ 05:17:26 DEBUG 1712722646, 05:17:26 DEBUG "1260.5333333333333" 05:17:26 DEBUG ] 05:17:26 DEBUG } 05:17:26 DEBUG ] 05:17:26 DEBUG } 05:17:26 DEBUG } 05:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722646" 05:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712722646" 05:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722646" 05:17:26 INFO [http_cmd]: http status code OK 05:17:26 DEBUG --- status code --- 05:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:26 DEBUG --- http response --- 05:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:17:26 DEBUG ----- output ----- 05:17:26 DEBUG { 05:17:26 DEBUG "status": "success", 05:17:26 DEBUG "data": { 05:17:26 DEBUG "resultType": "vector", 05:17:26 DEBUG "result": [ 05:17:26 DEBUG { 05:17:26 DEBUG "metric": { 05:17:26 DEBUG "pod": "ds-idrepo-0" 05:17:26 DEBUG }, 05:17:26 DEBUG "value": [ 05:17:26 DEBUG 05:17:26 DEBUG 05:17:26 DEBUG 05:17:26 DEBUG [...] 05:17:26 DEBUG 05:17:26 DEBUG 05:17:26 DEBUG 05:17:26 DEBUG "pod": "ds-idrepo-1" 05:17:26 DEBUG }, 05:17:26 DEBUG "value": [ 05:17:26 DEBUG 1712722646, 05:17:26 DEBUG "0" 05:17:26 DEBUG ] 05:17:26 DEBUG } 05:17:26 DEBUG ] 05:17:26 DEBUG } 05:17:26 DEBUG } 05:17:26 INFO [http_cmd]: http status code OK 05:17:26 DEBUG --- status code --- 05:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:26 DEBUG --- http response --- 05:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:17:26 DEBUG ----- output ----- 05:17:26 DEBUG { 05:17:26 DEBUG "status": "success", 05:17:26 DEBUG "data": { 05:17:26 INFO [http_cmd]: http status code OK 05:17:26 DEBUG "resultType": "vector", 05:17:26 DEBUG --- status code --- 05:17:26 DEBUG "result": [ 05:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:26 DEBUG { 05:17:26 DEBUG --- http response --- 05:17:26 DEBUG "metric": { 05:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:17:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:17:26 DEBUG ----- output ----- 05:17:26 DEBUG }, 05:17:26 DEBUG "value": [ 05:17:26 DEBUG 05:17:26 DEBUG { 05:17:26 DEBUG 05:17:26 DEBUG "status": "success", 05:17:26 DEBUG 05:17:26 DEBUG [...] 05:17:26 DEBUG "data": { 05:17:26 DEBUG 05:17:26 DEBUG "resultType": "vector", 05:17:26 DEBUG 05:17:26 DEBUG "result": [ 05:17:26 DEBUG 05:17:26 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:17:26 DEBUG { 05:17:26 DEBUG }, 05:17:26 DEBUG "metric": { 05:17:26 DEBUG "value": [ 05:17:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:17:26 DEBUG 1712722646, 05:17:26 DEBUG }, 05:17:26 DEBUG "0" 05:17:26 DEBUG "value": [ 05:17:26 DEBUG ] 05:17:26 DEBUG 05:17:26 DEBUG } 05:17:26 DEBUG 05:17:26 DEBUG ] 05:17:26 DEBUG 05:17:26 DEBUG } 05:17:26 DEBUG [...] 05:17:26 DEBUG } 05:17:26 DEBUG 05:17:26 DEBUG 05:17:26 DEBUG 05:17:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:17:26 DEBUG }, 05:17:26 DEBUG "value": [ 05:17:26 DEBUG 1712722646, 05:17:26 DEBUG "2489991168" 05:17:26 DEBUG ] 05:17:26 DEBUG } 05:17:26 DEBUG ] 05:17:26 DEBUG } 05:17:26 DEBUG } 05:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722646" 05:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722646" 05:17:26 INFO [http_cmd]: http status code OK 05:17:26 DEBUG --- status code --- 05:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:26 DEBUG --- http response --- 05:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:17:26 DEBUG ----- output ----- 05:17:26 DEBUG { 05:17:26 DEBUG "status": "success", 05:17:26 DEBUG "data": { 05:17:26 DEBUG "resultType": "vector", 05:17:26 DEBUG "result": [ 05:17:26 DEBUG { 05:17:26 DEBUG "metric": { 05:17:26 DEBUG "instance": "10.142.0.25:9100" 05:17:26 DEBUG }, 05:17:26 DEBUG "value": [ 05:17:26 DEBUG 05:17:26 DEBUG 05:17:26 DEBUG 05:17:26 DEBUG [...] 05:17:26 DEBUG 05:17:26 DEBUG 05:17:26 DEBUG 05:17:26 DEBUG "instance": "10.142.0.77:9100" 05:17:26 DEBUG }, 05:17:26 DEBUG "value": [ 05:17:26 DEBUG 1712722646, 05:17:26 DEBUG "0.057666666666666616" 05:17:26 DEBUG ] 05:17:26 DEBUG } 05:17:26 DEBUG ] 05:17:26 DEBUG } 05:17:26 DEBUG } 05:17:26 INFO [http_cmd]: http status code OK 05:17:26 DEBUG --- status code --- 05:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:17:26 DEBUG --- http response --- 05:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:17:26 DEBUG ----- output ----- 05:17:26 DEBUG { 05:17:26 DEBUG "status": "success", 05:17:26 DEBUG "data": { 05:17:26 DEBUG "resultType": "vector", 05:17:26 DEBUG "result": [ 05:17:26 DEBUG { 05:17:26 DEBUG "metric": { 05:17:26 DEBUG "instance": "10.142.0.25:9100" 05:17:26 DEBUG }, 05:17:26 DEBUG "value": [ 05:17:26 DEBUG 05:17:26 DEBUG 05:17:26 DEBUG 05:17:26 DEBUG [...] 05:17:26 DEBUG 05:17:26 DEBUG 05:17:26 DEBUG 05:17:26 DEBUG "instance": "10.142.0.77:9100" 05:17:26 DEBUG }, 05:17:26 DEBUG "value": [ 05:17:26 DEBUG 1712722646, 05:17:26 DEBUG "37587037.13333333" 05:17:26 DEBUG ] 05:17:26 DEBUG } 05:17:26 DEBUG ] 05:17:26 DEBUG } 05:17:26 DEBUG } 05:17:40 INFO 05:17:40 INFO [loop_until]: kubectl --namespace=xlou top pods 05:17:40 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:17:40 INFO [loop_until]: OK (rc = 0) 05:17:40 DEBUG --- stdout --- 05:17:40 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 9184m 8808Mi am-654c4fcbcf-9scrj 8554m 8425Mi am-654c4fcbcf-v5qb5 7470m 8401Mi ds-cts-0 7226m 13319Mi ds-cts-1 7594m 13320Mi ds-cts-2 6750m 12382Mi ds-idrepo-0 9715m 5621Mi ds-idrepo-1 319m 5668Mi ds-idrepo-2 381m 5698Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 4m 2212Mi lodemon-b776b7d88-thrqf 8m 70Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3625m 1653Mi 05:17:40 DEBUG --- stderr --- 05:17:40 DEBUG 05:17:46 INFO 05:17:46 INFO [loop_until]: kubectl --namespace=xlou top node 05:17:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:17:46 INFO [loop_until]: OK (rc = 0) 05:17:46 DEBUG --- stdout --- 05:17:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 92m 0% 2303Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8141m 51% 9278Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 79m 0% 3259Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10856m 68% 6480Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9564m 60% 9312Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9956m 62% 9648Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 80m 0% 863Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9768m 61% 6278Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 7279m 45% 13931Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 7100m 44% 13720Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 422m 2% 6358Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7270m 45% 13925Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 374m 2% 6330Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3693m 23% 5344Mi 9% 05:17:46 DEBUG --- stderr --- 05:17:46 DEBUG 05:18:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722702" 05:18:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712722702" 05:18:22 ERROR [http_cmd]: ERROR 05:18:22 ERROR --- status code --- 05:18:22 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:18:22 INFO [http_cmd]: http status code OK 05:18:22 ERROR --- http response --- 05:18:22 DEBUG --- status code --- 05:18:22 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:18:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:22 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:18:22 DEBUG --- http response --- 05:18:22 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:18:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722702" 05:18:22 INFO [http_cmd]: http status code OK 05:18:22 DEBUG --- status code --- 05:18:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:22 DEBUG --- http response --- 05:18:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:18:22 DEBUG ----- output ----- 05:18:22 DEBUG { 05:18:22 DEBUG "status": "success", 05:18:22 DEBUG "data": { 05:18:22 DEBUG "resultType": "vector", 05:18:22 DEBUG "result": [ 05:18:22 DEBUG { 05:18:22 DEBUG "metric": { 05:18:22 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:18:22 DEBUG }, 05:18:22 DEBUG "value": [ 05:18:22 DEBUG 05:18:22 DEBUG 05:18:22 DEBUG 05:18:22 DEBUG [...] 05:18:22 DEBUG 05:18:22 DEBUG 05:18:22 DEBUG 05:18:22 DEBUG "pod": "am-654c4fcbcf-487cv" 05:18:22 DEBUG }, 05:18:22 DEBUG "value": [ 05:18:22 DEBUG 1712722702, 05:18:22 DEBUG "0" 05:18:22 DEBUG ] 05:18:22 DEBUG } 05:18:22 DEBUG ] 05:18:22 DEBUG } 05:18:22 DEBUG } 05:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722703" 05:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722703" 05:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722703" 05:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722703" 05:18:23 INFO [http_cmd]: http status code OK 05:18:23 DEBUG --- status code --- 05:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:23 DEBUG --- http response --- 05:18:23 INFO [http_cmd]: http status code OK 05:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:18:23 INFO [http_cmd]: http status code OK 05:18:23 INFO [http_cmd]: http status code OK 05:18:23 DEBUG --- status code --- 05:18:23 DEBUG ----- output ----- 05:18:23 DEBUG --- status code --- 05:18:23 DEBUG --- status code --- 05:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:23 DEBUG { 05:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:23 DEBUG --- http response --- 05:18:23 DEBUG "status": "success", 05:18:23 DEBUG --- http response --- 05:18:23 DEBUG --- http response --- 05:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:18:23 DEBUG "data": { 05:18:23 DEBUG "resultType": "vector", 05:18:23 DEBUG "result": [ 05:18:23 DEBUG ----- output ----- 05:18:23 DEBUG { 05:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:18:23 DEBUG { 05:18:23 DEBUG "metric": { 05:18:23 DEBUG ----- output ----- 05:18:23 DEBUG ----- output ----- 05:18:23 DEBUG "status": "success", 05:18:23 DEBUG "instance": "10.142.0.25:9100" 05:18:23 DEBUG { 05:18:23 DEBUG { 05:18:23 DEBUG "data": { 05:18:23 DEBUG }, 05:18:23 DEBUG "status": "success", 05:18:23 DEBUG "status": "success", 05:18:23 DEBUG "resultType": "vector", 05:18:23 DEBUG "value": [ 05:18:23 DEBUG "data": { 05:18:23 DEBUG "data": { 05:18:23 DEBUG "result": [ 05:18:23 DEBUG 05:18:23 DEBUG 05:18:23 DEBUG "resultType": "vector", 05:18:23 DEBUG { 05:18:23 DEBUG 05:18:23 DEBUG [...] 05:18:23 DEBUG "result": [ 05:18:23 DEBUG "metric": { 05:18:23 DEBUG 05:18:23 DEBUG 05:18:23 DEBUG { 05:18:23 DEBUG "metric": { 05:18:23 DEBUG 05:18:23 DEBUG "instance": "10.142.0.77:9100" 05:18:23 DEBUG }, 05:18:23 DEBUG "value": [ 05:18:23 DEBUG "pod": "ds-idrepo-0" 05:18:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:18:23 DEBUG 1712722703, 05:18:23 DEBUG "resultType": "vector", 05:18:23 DEBUG }, 05:18:23 DEBUG }, 05:18:23 DEBUG "266763918336" 05:18:23 DEBUG "result": [ 05:18:23 DEBUG "value": [ 05:18:23 DEBUG "value": [ 05:18:23 DEBUG ] 05:18:23 DEBUG { 05:18:23 DEBUG 05:18:23 DEBUG 05:18:23 DEBUG } 05:18:23 DEBUG "metric": { 05:18:23 DEBUG 05:18:23 DEBUG 05:18:23 DEBUG ] 05:18:23 DEBUG "pod": "ds-idrepo-0" 05:18:23 DEBUG 05:18:23 DEBUG 05:18:23 DEBUG } 05:18:23 DEBUG }, 05:18:23 DEBUG [...] 05:18:23 DEBUG [...] 05:18:23 DEBUG } 05:18:23 DEBUG "value": [ 05:18:23 DEBUG 05:18:23 DEBUG 05:18:23 DEBUG 05:18:23 DEBUG 05:18:23 DEBUG 05:18:23 DEBUG 05:18:23 DEBUG 05:18:23 DEBUG 05:18:23 DEBUG 05:18:23 DEBUG "pod": "ds-cts-2" 05:18:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:18:23 DEBUG [...] 05:18:23 DEBUG 05:18:23 DEBUG }, 05:18:23 DEBUG 05:18:23 DEBUG }, 05:18:23 DEBUG "value": [ 05:18:23 DEBUG 05:18:23 DEBUG "pod": "ds-cts-2" 05:18:23 DEBUG 1712722703, 05:18:23 DEBUG }, 05:18:23 DEBUG "value": [ 05:18:23 DEBUG "1245.2666666666667" 05:18:23 DEBUG 1712722703, 05:18:23 DEBUG "0" 05:18:23 DEBUG ] 05:18:23 DEBUG } 05:18:23 DEBUG "value": [ 05:18:23 DEBUG 1712722703, 05:18:23 DEBUG ] 05:18:23 DEBUG "29.599999999999998" 05:18:23 DEBUG ] 05:18:23 DEBUG } 05:18:23 DEBUG } 05:18:23 DEBUG ] 05:18:23 DEBUG } 05:18:23 DEBUG ] 05:18:23 DEBUG } 05:18:23 DEBUG } 05:18:23 DEBUG } 05:18:23 DEBUG ] 05:18:23 DEBUG } 05:18:23 DEBUG } 05:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722703" 05:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722703" 05:18:23 INFO [http_cmd]: http status code OK 05:18:23 DEBUG --- status code --- 05:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:23 DEBUG --- http response --- 05:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:18:23 DEBUG ----- output ----- 05:18:23 DEBUG { 05:18:23 DEBUG "status": "success", 05:18:23 DEBUG "data": { 05:18:23 DEBUG "resultType": "vector", 05:18:23 DEBUG "result": [ 05:18:23 DEBUG { 05:18:23 DEBUG "metric": { 05:18:23 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:18:23 DEBUG }, 05:18:23 DEBUG "value": [ 05:18:23 DEBUG 05:18:23 DEBUG 05:18:23 DEBUG 05:18:23 DEBUG [...] 05:18:23 DEBUG 05:18:23 DEBUG 05:18:23 DEBUG 05:18:23 DEBUG "pod": "ds-idrepo-2" 05:18:23 DEBUG }, 05:18:23 DEBUG "value": [ 05:18:23 DEBUG 1712722703, 05:18:23 INFO [http_cmd]: http status code OK 05:18:23 DEBUG "0" 05:18:23 DEBUG --- status code --- 05:18:23 DEBUG ] 05:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:23 DEBUG } 05:18:23 DEBUG --- http response --- 05:18:23 DEBUG ] 05:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:18:23 DEBUG } 05:18:23 DEBUG ----- output ----- 05:18:23 DEBUG } 05:18:23 DEBUG { 05:18:23 DEBUG "status": "success", 05:18:23 DEBUG "data": { 05:18:23 DEBUG "resultType": "vector", 05:18:23 DEBUG "result": [ 05:18:23 DEBUG { 05:18:23 DEBUG "metric": { 05:18:23 DEBUG "instance": "10.142.0.25:9100" 05:18:23 DEBUG }, 05:18:23 DEBUG "value": [ 05:18:23 DEBUG 05:18:23 DEBUG 05:18:23 DEBUG 05:18:23 DEBUG [...] 05:18:23 DEBUG 05:18:23 DEBUG 05:18:23 DEBUG 05:18:23 DEBUG "instance": "10.142.0.77:9100" 05:18:23 DEBUG }, 05:18:23 DEBUG "value": [ 05:18:23 DEBUG 1712722703, 05:18:23 DEBUG "37447259.66666667" 05:18:23 DEBUG ] 05:18:23 DEBUG } 05:18:23 DEBUG ] 05:18:23 DEBUG } 05:18:23 DEBUG } 05:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722703" 05:18:23 INFO [http_cmd]: http status code OK 05:18:23 DEBUG --- status code --- 05:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:23 DEBUG --- http response --- 05:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:18:23 DEBUG ----- output ----- 05:18:23 DEBUG { 05:18:23 DEBUG "status": "success", 05:18:23 DEBUG "data": { 05:18:23 DEBUG "resultType": "vector", 05:18:23 DEBUG "result": [ 05:18:23 DEBUG { 05:18:23 DEBUG "metric": { 05:18:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:18:23 DEBUG }, 05:18:23 DEBUG "value": [ 05:18:23 DEBUG 05:18:23 DEBUG 05:18:23 DEBUG 05:18:23 DEBUG [...] 05:18:23 DEBUG 05:18:23 DEBUG 05:18:23 DEBUG 05:18:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:18:23 DEBUG }, 05:18:23 DEBUG "value": [ 05:18:23 DEBUG 1712722703, 05:18:23 DEBUG "227126.5" 05:18:23 DEBUG ] 05:18:23 DEBUG } 05:18:23 DEBUG ] 05:18:23 DEBUG } 05:18:23 DEBUG } 05:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722703" 05:18:23 INFO [http_cmd]: http status code OK 05:18:23 DEBUG --- status code --- 05:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:23 DEBUG --- http response --- 05:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:18:23 DEBUG ----- output ----- 05:18:23 DEBUG { 05:18:23 DEBUG "status": "success", 05:18:23 DEBUG "data": { 05:18:23 DEBUG "resultType": "vector", 05:18:23 DEBUG "result": [ 05:18:23 DEBUG { 05:18:23 DEBUG "metric": { 05:18:23 DEBUG "pod": "ds-idrepo-0" 05:18:23 DEBUG }, 05:18:23 DEBUG "value": [ 05:18:23 DEBUG 05:18:23 DEBUG 05:18:23 DEBUG 05:18:23 DEBUG [...] 05:18:23 DEBUG 05:18:23 DEBUG 05:18:23 DEBUG 05:18:23 DEBUG "pod": "ds-idrepo-1" 05:18:23 DEBUG }, 05:18:23 DEBUG "value": [ 05:18:23 DEBUG 1712722703, 05:18:23 DEBUG "0" 05:18:23 DEBUG ] 05:18:23 DEBUG } 05:18:23 DEBUG ] 05:18:23 DEBUG } 05:18:23 DEBUG } 05:18:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722704" 05:18:24 INFO [http_cmd]: http status code OK 05:18:24 DEBUG --- status code --- 05:18:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:24 DEBUG --- http response --- 05:18:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:18:24 DEBUG ----- output ----- 05:18:24 DEBUG { 05:18:24 DEBUG "status": "success", 05:18:24 DEBUG "data": { 05:18:24 DEBUG "resultType": "vector", 05:18:24 DEBUG "result": [ 05:18:24 DEBUG { 05:18:24 DEBUG "metric": { 05:18:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:18:24 DEBUG }, 05:18:24 DEBUG "value": [ 05:18:24 DEBUG 05:18:24 DEBUG 05:18:24 DEBUG 05:18:24 DEBUG [...] 05:18:24 DEBUG 05:18:24 DEBUG 05:18:24 DEBUG 05:18:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:18:24 DEBUG }, 05:18:24 DEBUG "value": [ 05:18:24 DEBUG 1712722704, 05:18:24 DEBUG "1245.3333333333333" 05:18:24 DEBUG ] 05:18:24 DEBUG } 05:18:24 DEBUG ] 05:18:24 DEBUG } 05:18:24 DEBUG } 05:18:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722704" 05:18:24 INFO [http_cmd]: http status code OK 05:18:24 DEBUG --- status code --- 05:18:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:24 DEBUG --- http response --- 05:18:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:18:24 DEBUG ----- output ----- 05:18:24 DEBUG { 05:18:24 DEBUG "status": "success", 05:18:24 DEBUG "data": { 05:18:24 DEBUG "resultType": "vector", 05:18:24 DEBUG "result": [ 05:18:24 DEBUG { 05:18:24 DEBUG "metric": { 05:18:24 DEBUG "instance": "10.142.0.25:9100" 05:18:24 DEBUG }, 05:18:24 DEBUG "value": [ 05:18:24 DEBUG 05:18:24 DEBUG 05:18:24 DEBUG 05:18:24 DEBUG [...] 05:18:24 DEBUG 05:18:24 DEBUG 05:18:24 DEBUG 05:18:24 DEBUG "instance": "10.142.0.77:9100" 05:18:24 DEBUG }, 05:18:24 DEBUG "value": [ 05:18:24 DEBUG 1712722704, 05:18:24 DEBUG "0.1454333333333333" 05:18:24 DEBUG ] 05:18:24 DEBUG } 05:18:24 DEBUG ] 05:18:24 DEBUG } 05:18:24 DEBUG } 05:18:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722705" 05:18:25 INFO [http_cmd]: http status code OK 05:18:25 DEBUG --- status code --- 05:18:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:25 DEBUG --- http response --- 05:18:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:18:25 DEBUG ----- output ----- 05:18:25 DEBUG { 05:18:25 DEBUG "status": "success", 05:18:25 DEBUG "data": { 05:18:25 DEBUG "resultType": "vector", 05:18:25 DEBUG "result": [ 05:18:25 DEBUG { 05:18:25 DEBUG "metric": { 05:18:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:18:25 DEBUG }, 05:18:25 DEBUG "value": [ 05:18:25 DEBUG 05:18:25 DEBUG 05:18:25 DEBUG 05:18:25 DEBUG [...] 05:18:25 DEBUG 05:18:25 DEBUG 05:18:25 DEBUG 05:18:25 DEBUG "pod": "ds-idrepo-2" 05:18:25 DEBUG }, 05:18:25 DEBUG "value": [ 05:18:25 DEBUG 1712722705, 05:18:25 DEBUG "5161.940768746062" 05:18:25 DEBUG ] 05:18:25 DEBUG } 05:18:25 DEBUG ] 05:18:25 DEBUG } 05:18:25 DEBUG } 05:18:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712722705" 05:18:25 INFO [http_cmd]: http status code OK 05:18:25 DEBUG --- status code --- 05:18:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:25 DEBUG --- http response --- 05:18:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:18:25 DEBUG ----- output ----- 05:18:25 DEBUG { 05:18:25 DEBUG "status": "success", 05:18:25 DEBUG "data": { 05:18:25 DEBUG "resultType": "vector", 05:18:25 DEBUG "result": [ 05:18:25 DEBUG { 05:18:25 DEBUG "metric": { 05:18:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:18:25 DEBUG }, 05:18:25 DEBUG "value": [ 05:18:25 DEBUG 05:18:25 DEBUG 05:18:25 DEBUG 05:18:25 DEBUG [...] 05:18:25 DEBUG 05:18:25 DEBUG 05:18:25 DEBUG 05:18:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:18:25 DEBUG }, 05:18:25 DEBUG "value": [ 05:18:25 DEBUG 1712722705, 05:18:25 DEBUG "3.6047044984221213" 05:18:25 DEBUG ] 05:18:25 DEBUG } 05:18:25 DEBUG ] 05:18:25 DEBUG } 05:18:25 DEBUG } 05:18:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712722705" 05:18:25 INFO [http_cmd]: http status code OK 05:18:25 DEBUG --- status code --- 05:18:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:25 DEBUG --- http response --- 05:18:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:18:25 DEBUG ----- output ----- 05:18:25 DEBUG { 05:18:25 DEBUG "status": "success", 05:18:25 DEBUG "data": { 05:18:25 DEBUG "resultType": "vector", 05:18:25 DEBUG "result": [ 05:18:25 DEBUG { 05:18:25 DEBUG "metric": { 05:18:25 DEBUG "pod": "ds-idrepo-0" 05:18:25 DEBUG }, 05:18:25 DEBUG "value": [ 05:18:25 DEBUG 05:18:25 DEBUG 05:18:25 DEBUG 05:18:25 DEBUG [...] 05:18:25 DEBUG 05:18:25 DEBUG 05:18:25 DEBUG 05:18:25 DEBUG "pod": "ds-cts-2" 05:18:25 DEBUG }, 05:18:25 DEBUG "value": [ 05:18:25 DEBUG 1712722705, 05:18:25 DEBUG "0" 05:18:25 DEBUG ] 05:18:25 DEBUG } 05:18:25 DEBUG ] 05:18:25 DEBUG } 05:18:25 DEBUG } 05:18:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722705" 05:18:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722705" 05:18:25 INFO [http_cmd]: http status code OK 05:18:25 DEBUG --- status code --- 05:18:25 INFO [http_cmd]: http status code OK 05:18:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:25 DEBUG --- status code --- 05:18:25 DEBUG --- http response --- 05:18:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:18:25 DEBUG --- http response --- 05:18:25 DEBUG ----- output ----- 05:18:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:18:25 DEBUG { 05:18:25 DEBUG ----- output ----- 05:18:25 DEBUG "status": "success", 05:18:25 DEBUG { 05:18:25 DEBUG "data": { 05:18:25 DEBUG "status": "success", 05:18:25 DEBUG "resultType": "vector", 05:18:25 DEBUG "data": { 05:18:25 DEBUG "result": [ 05:18:25 DEBUG "resultType": "vector", 05:18:25 DEBUG { 05:18:25 DEBUG "result": [ 05:18:25 DEBUG "metric": { 05:18:25 DEBUG { 05:18:25 DEBUG "pod": "ds-idrepo-2" 05:18:25 DEBUG "metric": { 05:18:25 DEBUG }, 05:18:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:18:25 DEBUG "value": [ 05:18:25 DEBUG }, 05:18:25 DEBUG 05:18:25 DEBUG "value": [ 05:18:25 DEBUG 05:18:25 DEBUG 05:18:25 DEBUG 05:18:25 DEBUG 05:18:25 DEBUG [...] 05:18:25 DEBUG 05:18:25 DEBUG 05:18:25 DEBUG [...] 05:18:25 DEBUG 05:18:25 DEBUG 05:18:25 DEBUG 05:18:25 DEBUG "pod": "ds-cts-2" 05:18:25 DEBUG 05:18:25 DEBUG }, 05:18:25 DEBUG 05:18:25 DEBUG "value": [ 05:18:25 DEBUG "pod": "ds-idrepo-2" 05:18:25 DEBUG 1712722705, 05:18:25 DEBUG }, 05:18:25 DEBUG "0.0001" 05:18:25 DEBUG "value": [ 05:18:25 DEBUG ] 05:18:25 DEBUG } 05:18:25 DEBUG 1712722705, 05:18:25 DEBUG ] 05:18:25 DEBUG "0.3243493384737925" 05:18:25 DEBUG } 05:18:25 DEBUG ] 05:18:25 DEBUG } 05:18:25 DEBUG } 05:18:25 DEBUG ] 05:18:25 DEBUG } 05:18:25 DEBUG } 05:18:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722705" 05:18:25 INFO [http_cmd]: http status code OK 05:18:25 DEBUG --- status code --- 05:18:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:25 DEBUG --- http response --- 05:18:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:18:25 DEBUG ----- output ----- 05:18:25 DEBUG { 05:18:25 DEBUG "status": "success", 05:18:25 DEBUG "data": { 05:18:25 DEBUG "resultType": "vector", 05:18:25 DEBUG "result": [ 05:18:25 DEBUG { 05:18:25 DEBUG "metric": { 05:18:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:18:25 DEBUG }, 05:18:25 DEBUG "value": [ 05:18:25 DEBUG 05:18:25 DEBUG 05:18:25 DEBUG 05:18:25 DEBUG [...] 05:18:25 DEBUG 05:18:25 DEBUG 05:18:25 DEBUG 05:18:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:18:25 DEBUG }, 05:18:25 DEBUG "value": [ 05:18:25 DEBUG 1712722705, 05:18:25 DEBUG "1245.6333333333332" 05:18:25 DEBUG ] 05:18:25 DEBUG } 05:18:25 DEBUG ] 05:18:25 DEBUG } 05:18:25 DEBUG } 05:18:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712722705" 05:18:25 INFO [http_cmd]: http status code OK 05:18:25 DEBUG --- status code --- 05:18:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:25 DEBUG --- http response --- 05:18:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:18:25 DEBUG ----- output ----- 05:18:25 DEBUG { 05:18:25 DEBUG "status": "success", 05:18:25 DEBUG "data": { 05:18:25 DEBUG "resultType": "vector", 05:18:25 DEBUG "result": [ 05:18:25 DEBUG { 05:18:25 DEBUG "metric": { 05:18:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:18:25 DEBUG }, 05:18:25 DEBUG "value": [ 05:18:25 DEBUG 05:18:25 DEBUG 05:18:25 DEBUG 05:18:25 DEBUG [...] 05:18:25 DEBUG 05:18:25 DEBUG 05:18:25 DEBUG 05:18:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:18:25 DEBUG }, 05:18:25 DEBUG "value": [ 05:18:25 DEBUG 1712722705, 05:18:25 DEBUG "0.04" 05:18:25 DEBUG ] 05:18:25 DEBUG } 05:18:25 DEBUG ] 05:18:25 DEBUG } 05:18:25 DEBUG } 05:18:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722706" 05:18:26 INFO [http_cmd]: http status code OK 05:18:26 DEBUG --- status code --- 05:18:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:26 DEBUG --- http response --- 05:18:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:18:26 DEBUG ----- output ----- 05:18:26 DEBUG { 05:18:26 DEBUG "status": "success", 05:18:26 DEBUG "data": { 05:18:26 DEBUG "resultType": "vector", 05:18:26 DEBUG "result": [ 05:18:26 DEBUG { 05:18:26 DEBUG "metric": { 05:18:26 DEBUG "pod": "ds-idrepo-0" 05:18:26 DEBUG }, 05:18:26 DEBUG "value": [ 05:18:26 DEBUG 05:18:26 DEBUG 05:18:26 DEBUG 05:18:26 DEBUG [...] 05:18:26 DEBUG 05:18:26 DEBUG 05:18:26 DEBUG 05:18:26 DEBUG "pod": "ds-idrepo-1" 05:18:26 DEBUG }, 05:18:26 DEBUG "value": [ 05:18:26 DEBUG 1712722706, 05:18:26 DEBUG "0" 05:18:26 DEBUG ] 05:18:26 DEBUG } 05:18:26 DEBUG ] 05:18:26 DEBUG } 05:18:26 DEBUG } 05:18:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722706" 05:18:26 INFO [http_cmd]: http status code OK 05:18:26 DEBUG --- status code --- 05:18:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:26 DEBUG --- http response --- 05:18:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:18:26 DEBUG ----- output ----- 05:18:26 DEBUG { 05:18:26 DEBUG "status": "success", 05:18:26 DEBUG "data": { 05:18:26 DEBUG "resultType": "vector", 05:18:26 DEBUG "result": [ 05:18:26 DEBUG { 05:18:26 DEBUG "metric": { 05:18:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:18:26 DEBUG }, 05:18:26 DEBUG "value": [ 05:18:26 DEBUG 05:18:26 DEBUG 05:18:26 DEBUG 05:18:26 DEBUG [...] 05:18:26 DEBUG 05:18:26 DEBUG 05:18:26 DEBUG 05:18:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:18:26 DEBUG }, 05:18:26 DEBUG "value": [ 05:18:26 DEBUG 1712722706, 05:18:26 DEBUG "1262.9666666666667" 05:18:26 DEBUG ] 05:18:26 DEBUG } 05:18:26 DEBUG ] 05:18:26 DEBUG } 05:18:26 DEBUG } 05:18:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722706" 05:18:26 INFO [http_cmd]: http status code OK 05:18:26 DEBUG --- status code --- 05:18:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:26 DEBUG --- http response --- 05:18:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:18:26 DEBUG ----- output ----- 05:18:26 DEBUG { 05:18:26 DEBUG "status": "success", 05:18:26 DEBUG "data": { 05:18:26 DEBUG "resultType": "vector", 05:18:26 DEBUG "result": [ 05:18:26 DEBUG { 05:18:26 DEBUG "metric": { 05:18:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722706" 05:18:26 DEBUG "pod": "ds-idrepo-0" 05:18:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712722706" 05:18:26 DEBUG }, 05:18:26 DEBUG "value": [ 05:18:26 DEBUG 05:18:26 DEBUG 05:18:26 DEBUG 05:18:26 DEBUG [...] 05:18:26 DEBUG 05:18:26 DEBUG 05:18:26 DEBUG 05:18:26 DEBUG "pod": "ds-idrepo-1" 05:18:26 DEBUG }, 05:18:26 DEBUG "value": [ 05:18:26 DEBUG 1712722706, 05:18:26 DEBUG "0" 05:18:26 DEBUG ] 05:18:26 DEBUG } 05:18:26 DEBUG ] 05:18:26 DEBUG } 05:18:26 DEBUG } 05:18:26 INFO [http_cmd]: http status code OK 05:18:26 DEBUG --- status code --- 05:18:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:26 DEBUG --- http response --- 05:18:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:18:26 DEBUG ----- output ----- 05:18:26 DEBUG { 05:18:26 DEBUG "status": "success", 05:18:26 DEBUG "data": { 05:18:26 DEBUG "resultType": "vector", 05:18:26 DEBUG "result": [ 05:18:26 DEBUG { 05:18:26 DEBUG "metric": { 05:18:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:18:26 DEBUG }, 05:18:26 DEBUG "value": [ 05:18:26 DEBUG 05:18:26 DEBUG 05:18:26 DEBUG 05:18:26 DEBUG [...] 05:18:26 DEBUG 05:18:26 DEBUG 05:18:26 DEBUG 05:18:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:18:26 DEBUG }, 05:18:26 DEBUG "value": [ 05:18:26 DEBUG 1712722706, 05:18:26 DEBUG "2486943744" 05:18:26 DEBUG ] 05:18:26 DEBUG } 05:18:26 DEBUG ] 05:18:26 DEBUG } 05:18:26 DEBUG } 05:18:26 INFO [http_cmd]: http status code OK 05:18:26 DEBUG --- status code --- 05:18:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:26 DEBUG --- http response --- 05:18:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:18:26 DEBUG ----- output ----- 05:18:26 DEBUG { 05:18:26 DEBUG "status": "success", 05:18:26 DEBUG "data": { 05:18:26 DEBUG "resultType": "vector", 05:18:26 DEBUG "result": [ 05:18:26 DEBUG { 05:18:26 DEBUG "metric": { 05:18:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:18:26 DEBUG }, 05:18:26 DEBUG "value": [ 05:18:26 DEBUG 05:18:26 DEBUG 05:18:26 DEBUG 05:18:26 DEBUG [...] 05:18:26 DEBUG 05:18:26 DEBUG 05:18:26 DEBUG 05:18:26 DEBUG "pod": "ds-idrepo-2" 05:18:26 DEBUG }, 05:18:26 DEBUG "value": [ 05:18:26 DEBUG 1712722706, 05:18:26 DEBUG "3803.535303286572" 05:18:26 DEBUG ] 05:18:26 DEBUG } 05:18:26 DEBUG ] 05:18:26 DEBUG } 05:18:26 DEBUG } 05:18:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722706" 05:18:26 INFO [http_cmd]: http status code OK 05:18:26 DEBUG --- status code --- 05:18:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:26 DEBUG --- http response --- 05:18:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:18:26 DEBUG ----- output ----- 05:18:26 DEBUG { 05:18:26 DEBUG "status": "success", 05:18:26 DEBUG "data": { 05:18:26 DEBUG "resultType": "vector", 05:18:26 DEBUG "result": [ 05:18:26 DEBUG { 05:18:26 DEBUG "metric": { 05:18:26 DEBUG "instance": "10.142.0.25:9100" 05:18:26 DEBUG }, 05:18:26 DEBUG "value": [ 05:18:26 DEBUG 05:18:26 DEBUG 05:18:26 DEBUG 05:18:26 DEBUG [...] 05:18:26 DEBUG 05:18:26 DEBUG 05:18:26 DEBUG 05:18:26 DEBUG "instance": "10.142.0.77:9100" 05:18:26 DEBUG }, 05:18:26 DEBUG "value": [ 05:18:26 DEBUG 1712722706, 05:18:26 DEBUG "0.0556666666666663" 05:18:26 DEBUG ] 05:18:26 DEBUG } 05:18:26 DEBUG ] 05:18:26 DEBUG } 05:18:26 DEBUG } 05:18:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722706" 05:18:26 INFO [http_cmd]: http status code OK 05:18:26 DEBUG --- status code --- 05:18:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:18:26 DEBUG --- http response --- 05:18:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:18:26 DEBUG ----- output ----- 05:18:26 DEBUG { 05:18:26 DEBUG "status": "success", 05:18:26 DEBUG "data": { 05:18:26 DEBUG "resultType": "vector", 05:18:26 DEBUG "result": [ 05:18:26 DEBUG { 05:18:26 DEBUG "metric": { 05:18:26 DEBUG "instance": "10.142.0.25:9100" 05:18:26 DEBUG }, 05:18:26 DEBUG "value": [ 05:18:26 DEBUG 05:18:26 DEBUG 05:18:26 DEBUG 05:18:26 DEBUG [...] 05:18:26 DEBUG 05:18:26 DEBUG 05:18:26 DEBUG 05:18:26 DEBUG "instance": "10.142.0.77:9100" 05:18:26 DEBUG }, 05:18:26 DEBUG "value": [ 05:18:26 DEBUG 1712722706, 05:18:26 DEBUG "37448554.93333334" 05:18:26 DEBUG ] 05:18:26 DEBUG } 05:18:26 DEBUG ] 05:18:26 DEBUG } 05:18:26 DEBUG } 05:18:40 INFO 05:18:40 INFO [loop_until]: kubectl --namespace=xlou top pods 05:18:40 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:18:40 INFO [loop_until]: OK (rc = 0) 05:18:40 DEBUG --- stdout --- 05:18:40 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 8649m 8805Mi am-654c4fcbcf-9scrj 8605m 8423Mi am-654c4fcbcf-v5qb5 7464m 8400Mi ds-cts-0 6789m 12807Mi ds-cts-1 7378m 12870Mi ds-cts-2 6829m 13145Mi ds-idrepo-0 9713m 5621Mi ds-idrepo-1 317m 5668Mi ds-idrepo-2 351m 5698Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 5m 2212Mi lodemon-b776b7d88-thrqf 8m 70Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3525m 1654Mi 05:18:40 DEBUG --- stderr --- 05:18:40 DEBUG 05:18:46 INFO 05:18:46 INFO [loop_until]: kubectl --namespace=xlou top node 05:18:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:18:47 INFO [loop_until]: OK (rc = 0) 05:18:47 DEBUG --- stdout --- 05:18:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 94m 0% 2300Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8212m 51% 9278Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 80m 0% 3255Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10766m 67% 6500Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9549m 60% 9310Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9834m 61% 9649Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 76m 0% 854Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9737m 61% 6280Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 7261m 45% 13448Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 6883m 43% 13807Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 375m 2% 6357Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7402m 46% 13460Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 376m 2% 6323Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3727m 23% 5345Mi 9% 05:18:47 DEBUG --- stderr --- 05:18:47 DEBUG 05:19:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712722762" 05:19:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722762" 05:19:22 INFO [http_cmd]: http status code OK 05:19:22 DEBUG --- status code --- 05:19:22 ERROR [http_cmd]: ERROR 05:19:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:22 ERROR --- status code --- 05:19:22 DEBUG --- http response --- 05:19:22 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:19:22 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:19:22 ERROR --- http response --- 05:19:22 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:19:22 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722763" 05:19:23 INFO [http_cmd]: http status code OK 05:19:23 DEBUG --- status code --- 05:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:23 DEBUG --- http response --- 05:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:19:23 DEBUG ----- output ----- 05:19:23 DEBUG { 05:19:23 DEBUG "status": "success", 05:19:23 DEBUG "data": { 05:19:23 DEBUG "resultType": "vector", 05:19:23 DEBUG "result": [ 05:19:23 DEBUG { 05:19:23 DEBUG "metric": { 05:19:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:19:23 DEBUG }, 05:19:23 DEBUG "value": [ 05:19:23 DEBUG 05:19:23 DEBUG 05:19:23 DEBUG 05:19:23 DEBUG [...] 05:19:23 DEBUG 05:19:23 DEBUG 05:19:23 DEBUG 05:19:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:19:23 DEBUG }, 05:19:23 DEBUG "value": [ 05:19:23 DEBUG 1712722763, 05:19:23 DEBUG "0" 05:19:23 DEBUG ] 05:19:23 DEBUG } 05:19:23 DEBUG ] 05:19:23 DEBUG } 05:19:23 DEBUG } 05:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722763" 05:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722763" 05:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722763" 05:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722763" 05:19:23 INFO [http_cmd]: http status code OK 05:19:23 DEBUG --- status code --- 05:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:23 DEBUG --- http response --- 05:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:19:23 DEBUG ----- output ----- 05:19:23 DEBUG { 05:19:23 DEBUG "status": "success", 05:19:23 DEBUG "data": { 05:19:23 DEBUG "resultType": "vector", 05:19:23 DEBUG "result": [ 05:19:23 DEBUG { 05:19:23 DEBUG "metric": { 05:19:23 DEBUG "pod": "ds-idrepo-0" 05:19:23 DEBUG }, 05:19:23 DEBUG "value": [ 05:19:23 DEBUG 05:19:23 INFO [http_cmd]: http status code OK 05:19:23 INFO [http_cmd]: http status code OK 05:19:23 INFO [http_cmd]: http status code OK 05:19:23 DEBUG 05:19:23 DEBUG --- status code --- 05:19:23 DEBUG --- status code --- 05:19:23 DEBUG --- status code --- 05:19:23 DEBUG 05:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:23 DEBUG [...] 05:19:23 DEBUG 05:19:23 DEBUG --- http response --- 05:19:23 DEBUG --- http response --- 05:19:23 DEBUG --- http response --- 05:19:23 DEBUG 05:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:19:23 DEBUG ----- output ----- 05:19:23 DEBUG { 05:19:23 DEBUG "status": "success", 05:19:23 DEBUG 05:19:23 DEBUG "data": { 05:19:23 DEBUG "resultType": "vector", 05:19:23 DEBUG "result": [ 05:19:23 DEBUG "pod": "ds-cts-2" 05:19:23 DEBUG { 05:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:19:23 DEBUG ----- output ----- 05:19:23 DEBUG }, 05:19:23 DEBUG "metric": { 05:19:23 DEBUG ----- output ----- 05:19:23 DEBUG { 05:19:23 DEBUG "value": [ 05:19:23 DEBUG "pod": "ds-idrepo-0" 05:19:23 DEBUG { 05:19:23 DEBUG "status": "success", 05:19:23 DEBUG "data": { 05:19:23 DEBUG }, 05:19:23 DEBUG "status": "success", 05:19:23 DEBUG 1712722763, 05:19:23 DEBUG "resultType": "vector", 05:19:23 DEBUG "value": [ 05:19:23 DEBUG "data": { 05:19:23 DEBUG "29.766666666666666" 05:19:23 DEBUG "result": [ 05:19:23 DEBUG 05:19:23 DEBUG "resultType": "vector", 05:19:23 DEBUG ] 05:19:23 DEBUG { 05:19:23 DEBUG 05:19:23 DEBUG "result": [ 05:19:23 DEBUG } 05:19:23 DEBUG "metric": { 05:19:23 DEBUG 05:19:23 DEBUG { 05:19:23 DEBUG ] 05:19:23 DEBUG } 05:19:23 DEBUG } 05:19:23 DEBUG "metric": { 05:19:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:19:23 DEBUG [...] 05:19:23 DEBUG "instance": "10.142.0.25:9100" 05:19:23 DEBUG }, 05:19:23 DEBUG 05:19:23 DEBUG "value": [ 05:19:23 DEBUG }, 05:19:23 DEBUG "value": [ 05:19:23 DEBUG 05:19:23 DEBUG 05:19:23 DEBUG 05:19:23 DEBUG 05:19:23 DEBUG 05:19:23 DEBUG 05:19:23 DEBUG "pod": "ds-cts-2" 05:19:23 DEBUG 05:19:23 DEBUG 05:19:23 DEBUG }, 05:19:23 DEBUG [...] 05:19:23 DEBUG [...] 05:19:23 DEBUG "value": [ 05:19:23 DEBUG 05:19:23 DEBUG 05:19:23 DEBUG 1712722763, 05:19:23 DEBUG 05:19:23 DEBUG 05:19:23 DEBUG "0" 05:19:23 DEBUG 05:19:23 DEBUG 05:19:23 DEBUG "instance": "10.142.0.77:9100" 05:19:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:19:23 DEBUG }, 05:19:23 DEBUG ] 05:19:23 DEBUG }, 05:19:23 DEBUG "value": [ 05:19:23 DEBUG } 05:19:23 DEBUG "value": [ 05:19:23 DEBUG 1712722763, 05:19:23 DEBUG ] 05:19:23 DEBUG 1712722763, 05:19:23 DEBUG "268645067776" 05:19:23 DEBUG } 05:19:23 DEBUG "1252.3666666666666" 05:19:23 DEBUG ] 05:19:23 DEBUG } 05:19:23 DEBUG ] 05:19:23 DEBUG } 05:19:23 DEBUG } 05:19:23 DEBUG ] 05:19:23 DEBUG ] 05:19:23 DEBUG } 05:19:23 DEBUG } 05:19:23 DEBUG } 05:19:23 DEBUG } 05:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722763" 05:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722763" 05:19:23 INFO [http_cmd]: http status code OK 05:19:23 INFO [http_cmd]: http status code OK 05:19:23 DEBUG --- status code --- 05:19:23 DEBUG --- status code --- 05:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:23 DEBUG --- http response --- 05:19:23 DEBUG --- http response --- 05:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:19:23 DEBUG ----- output ----- 05:19:23 DEBUG ----- output ----- 05:19:23 DEBUG { 05:19:23 DEBUG { 05:19:23 DEBUG "status": "success", 05:19:23 DEBUG "data": { 05:19:23 DEBUG "status": "success", 05:19:23 DEBUG "resultType": "vector", 05:19:23 DEBUG "data": { 05:19:23 DEBUG "result": [ 05:19:23 DEBUG { 05:19:23 DEBUG "resultType": "vector", 05:19:23 DEBUG "metric": { 05:19:23 DEBUG "result": [ 05:19:23 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:19:23 DEBUG { 05:19:23 DEBUG }, 05:19:23 DEBUG "metric": { 05:19:23 DEBUG "value": [ 05:19:23 DEBUG "instance": "10.142.0.25:9100" 05:19:23 DEBUG 05:19:23 DEBUG }, 05:19:23 DEBUG 05:19:23 DEBUG "value": [ 05:19:23 DEBUG 05:19:23 DEBUG 05:19:23 DEBUG [...] 05:19:23 DEBUG 05:19:23 DEBUG 05:19:23 DEBUG 05:19:23 DEBUG 05:19:23 DEBUG [...] 05:19:23 DEBUG 05:19:23 DEBUG 05:19:23 DEBUG "pod": "am-654c4fcbcf-9scrj" 05:19:23 DEBUG 05:19:23 DEBUG }, 05:19:23 DEBUG 05:19:23 DEBUG "value": [ 05:19:23 DEBUG "instance": "10.142.0.77:9100" 05:19:23 DEBUG 1712722763, 05:19:23 DEBUG "0" 05:19:23 DEBUG }, 05:19:23 DEBUG ] 05:19:23 DEBUG "value": [ 05:19:23 DEBUG } 05:19:23 DEBUG 1712722763, 05:19:23 DEBUG ] 05:19:23 DEBUG "37626137.16666667" 05:19:23 DEBUG } 05:19:23 DEBUG ] 05:19:23 DEBUG } 05:19:23 DEBUG } 05:19:23 DEBUG ] 05:19:23 DEBUG } 05:19:23 DEBUG } 05:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722763" 05:19:23 INFO [http_cmd]: http status code OK 05:19:23 DEBUG --- status code --- 05:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:23 DEBUG --- http response --- 05:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:19:23 DEBUG ----- output ----- 05:19:23 DEBUG { 05:19:23 DEBUG "status": "success", 05:19:23 DEBUG "data": { 05:19:23 DEBUG "resultType": "vector", 05:19:23 DEBUG "result": [ 05:19:23 DEBUG { 05:19:23 DEBUG "metric": { 05:19:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:19:23 DEBUG }, 05:19:23 DEBUG "value": [ 05:19:23 DEBUG 05:19:23 DEBUG 05:19:23 DEBUG 05:19:23 DEBUG [...] 05:19:23 DEBUG 05:19:23 DEBUG 05:19:23 DEBUG 05:19:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:19:23 DEBUG }, 05:19:23 DEBUG "value": [ 05:19:23 DEBUG 1712722763, 05:19:23 DEBUG "226257.5" 05:19:23 DEBUG ] 05:19:23 DEBUG } 05:19:23 DEBUG ] 05:19:23 DEBUG } 05:19:23 DEBUG } 05:19:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722764" 05:19:24 INFO [http_cmd]: http status code OK 05:19:24 DEBUG --- status code --- 05:19:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:24 DEBUG --- http response --- 05:19:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:19:24 DEBUG ----- output ----- 05:19:24 DEBUG { 05:19:24 DEBUG "status": "success", 05:19:24 DEBUG "data": { 05:19:24 DEBUG "resultType": "vector", 05:19:24 DEBUG "result": [ 05:19:24 DEBUG { 05:19:24 DEBUG "metric": { 05:19:24 DEBUG "pod": "ds-idrepo-0" 05:19:24 DEBUG }, 05:19:24 DEBUG "value": [ 05:19:24 DEBUG 05:19:24 DEBUG 05:19:24 DEBUG 05:19:24 DEBUG [...] 05:19:24 DEBUG 05:19:24 DEBUG 05:19:24 DEBUG 05:19:24 DEBUG "pod": "ds-idrepo-1" 05:19:24 DEBUG }, 05:19:24 DEBUG "value": [ 05:19:24 DEBUG 1712722764, 05:19:24 DEBUG "0" 05:19:24 DEBUG ] 05:19:24 DEBUG } 05:19:24 DEBUG ] 05:19:24 DEBUG } 05:19:24 DEBUG } 05:19:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722764" 05:19:24 INFO [http_cmd]: http status code OK 05:19:24 DEBUG --- status code --- 05:19:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:24 DEBUG --- http response --- 05:19:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:19:24 DEBUG ----- output ----- 05:19:24 DEBUG { 05:19:24 DEBUG "status": "success", 05:19:24 DEBUG "data": { 05:19:24 DEBUG "resultType": "vector", 05:19:24 DEBUG "result": [ 05:19:24 DEBUG { 05:19:24 DEBUG "metric": { 05:19:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:19:24 DEBUG }, 05:19:24 DEBUG "value": [ 05:19:24 DEBUG 05:19:24 DEBUG 05:19:24 DEBUG 05:19:24 DEBUG [...] 05:19:24 DEBUG 05:19:24 DEBUG 05:19:24 DEBUG 05:19:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:19:24 DEBUG }, 05:19:24 DEBUG "value": [ 05:19:24 DEBUG 1712722764, 05:19:24 DEBUG "1251.7666666666667" 05:19:24 DEBUG ] 05:19:24 DEBUG } 05:19:24 DEBUG ] 05:19:24 DEBUG } 05:19:24 DEBUG } 05:19:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722764" 05:19:24 INFO [http_cmd]: http status code OK 05:19:24 DEBUG --- status code --- 05:19:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:24 DEBUG --- http response --- 05:19:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:19:24 DEBUG ----- output ----- 05:19:24 DEBUG { 05:19:24 DEBUG "status": "success", 05:19:24 DEBUG "data": { 05:19:24 DEBUG "resultType": "vector", 05:19:24 DEBUG "result": [ 05:19:24 DEBUG { 05:19:24 DEBUG "metric": { 05:19:24 DEBUG "instance": "10.142.0.25:9100" 05:19:24 DEBUG }, 05:19:24 DEBUG "value": [ 05:19:24 DEBUG 05:19:24 DEBUG 05:19:24 DEBUG 05:19:24 DEBUG [...] 05:19:24 DEBUG 05:19:24 DEBUG 05:19:24 DEBUG 05:19:24 DEBUG "instance": "10.142.0.77:9100" 05:19:24 DEBUG }, 05:19:24 DEBUG "value": [ 05:19:24 DEBUG 1712722764, 05:19:24 DEBUG "0.03219999999999885" 05:19:24 DEBUG ] 05:19:24 DEBUG } 05:19:24 DEBUG ] 05:19:24 DEBUG } 05:19:24 DEBUG } 05:19:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722765" 05:19:25 INFO [http_cmd]: http status code OK 05:19:25 DEBUG --- status code --- 05:19:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:25 DEBUG --- http response --- 05:19:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:19:25 DEBUG ----- output ----- 05:19:25 DEBUG { 05:19:25 DEBUG "status": "success", 05:19:25 DEBUG "data": { 05:19:25 DEBUG "resultType": "vector", 05:19:25 DEBUG "result": [ 05:19:25 DEBUG { 05:19:25 DEBUG "metric": { 05:19:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:19:25 DEBUG }, 05:19:25 DEBUG "value": [ 05:19:25 DEBUG 05:19:25 DEBUG 05:19:25 DEBUG 05:19:25 DEBUG [...] 05:19:25 DEBUG 05:19:25 DEBUG 05:19:25 DEBUG 05:19:25 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:19:25 DEBUG }, 05:19:25 DEBUG "value": [ 05:19:25 DEBUG 1712722765, 05:19:25 DEBUG "48773374.186953165" 05:19:25 DEBUG ] 05:19:25 DEBUG } 05:19:25 DEBUG ] 05:19:25 DEBUG } 05:19:25 DEBUG } 05:19:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712722765" 05:19:25 INFO [http_cmd]: http status code OK 05:19:25 DEBUG --- status code --- 05:19:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:25 DEBUG --- http response --- 05:19:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:19:25 DEBUG ----- output ----- 05:19:25 DEBUG { 05:19:25 DEBUG "status": "success", 05:19:25 DEBUG "data": { 05:19:25 DEBUG "resultType": "vector", 05:19:25 DEBUG "result": [ 05:19:25 DEBUG { 05:19:25 DEBUG "metric": { 05:19:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:19:25 DEBUG }, 05:19:25 DEBUG "value": [ 05:19:25 DEBUG 05:19:25 DEBUG 05:19:25 DEBUG 05:19:25 DEBUG [...] 05:19:25 DEBUG 05:19:25 DEBUG 05:19:25 DEBUG 05:19:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:19:25 DEBUG }, 05:19:25 DEBUG "value": [ 05:19:25 DEBUG 1712722765, 05:19:25 DEBUG "3.6144549380782616" 05:19:25 DEBUG ] 05:19:25 DEBUG } 05:19:25 DEBUG ] 05:19:25 DEBUG } 05:19:25 DEBUG } 05:19:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712722765" 05:19:25 INFO [http_cmd]: http status code OK 05:19:25 DEBUG --- status code --- 05:19:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:25 DEBUG --- http response --- 05:19:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:19:25 DEBUG ----- output ----- 05:19:25 DEBUG { 05:19:25 DEBUG "status": "success", 05:19:25 DEBUG "data": { 05:19:25 DEBUG "resultType": "vector", 05:19:25 DEBUG "result": [ 05:19:25 DEBUG { 05:19:25 DEBUG "metric": { 05:19:25 DEBUG "pod": "ds-idrepo-0" 05:19:25 DEBUG }, 05:19:25 DEBUG "value": [ 05:19:25 DEBUG 05:19:25 DEBUG 05:19:25 DEBUG 05:19:25 DEBUG [...] 05:19:25 DEBUG 05:19:25 DEBUG 05:19:25 DEBUG 05:19:25 DEBUG "pod": "ds-cts-2" 05:19:25 DEBUG }, 05:19:25 DEBUG "value": [ 05:19:25 DEBUG 1712722765, 05:19:25 DEBUG "0" 05:19:25 DEBUG ] 05:19:25 DEBUG } 05:19:25 DEBUG ] 05:19:25 DEBUG } 05:19:25 DEBUG } 05:19:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722765" 05:19:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722765" 05:19:25 INFO [http_cmd]: http status code OK 05:19:25 INFO [http_cmd]: http status code OK 05:19:25 DEBUG --- status code --- 05:19:25 DEBUG --- status code --- 05:19:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:25 DEBUG --- http response --- 05:19:25 DEBUG --- http response --- 05:19:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:19:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:19:25 DEBUG ----- output ----- 05:19:25 DEBUG ----- output ----- 05:19:25 DEBUG { 05:19:25 DEBUG { 05:19:25 DEBUG "status": "success", 05:19:25 DEBUG "status": "success", 05:19:25 DEBUG "data": { 05:19:25 DEBUG "data": { 05:19:25 DEBUG "resultType": "vector", 05:19:25 DEBUG "resultType": "vector", 05:19:25 DEBUG "result": [ 05:19:25 DEBUG "result": [ 05:19:25 DEBUG { 05:19:25 DEBUG { 05:19:25 DEBUG "metric": { 05:19:25 DEBUG "metric": { 05:19:25 DEBUG "pod": "ds-idrepo-2" 05:19:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:19:25 DEBUG }, 05:19:25 DEBUG }, 05:19:25 DEBUG "value": [ 05:19:25 DEBUG "value": [ 05:19:25 DEBUG 05:19:25 DEBUG 05:19:25 DEBUG 05:19:25 DEBUG 05:19:25 DEBUG 05:19:25 DEBUG 05:19:25 DEBUG [...] 05:19:25 DEBUG [...] 05:19:25 DEBUG 05:19:25 DEBUG 05:19:25 DEBUG 05:19:25 DEBUG 05:19:25 DEBUG 05:19:25 DEBUG 05:19:25 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:19:25 DEBUG "pod": "ds-cts-2" 05:19:25 DEBUG }, 05:19:25 DEBUG }, 05:19:25 DEBUG "value": [ 05:19:25 DEBUG "value": [ 05:19:25 DEBUG 1712722765, 05:19:25 DEBUG 1712722765, 05:19:25 DEBUG "2.15570747888554" 05:19:25 DEBUG "0.00013387222222222223" 05:19:25 DEBUG ] 05:19:25 DEBUG ] 05:19:25 DEBUG } 05:19:25 DEBUG } 05:19:25 DEBUG ] 05:19:25 DEBUG ] 05:19:25 DEBUG } 05:19:25 DEBUG } 05:19:25 DEBUG } 05:19:25 DEBUG } 05:19:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722765" 05:19:25 INFO [http_cmd]: http status code OK 05:19:25 DEBUG --- status code --- 05:19:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:25 DEBUG --- http response --- 05:19:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:19:25 DEBUG ----- output ----- 05:19:25 DEBUG { 05:19:25 DEBUG "status": "success", 05:19:25 DEBUG "data": { 05:19:25 DEBUG "resultType": "vector", 05:19:25 DEBUG "result": [ 05:19:25 DEBUG { 05:19:25 DEBUG "metric": { 05:19:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:19:25 DEBUG }, 05:19:25 DEBUG "value": [ 05:19:25 DEBUG 05:19:25 DEBUG 05:19:25 DEBUG 05:19:25 DEBUG [...] 05:19:25 DEBUG 05:19:25 DEBUG 05:19:25 DEBUG 05:19:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:19:25 DEBUG }, 05:19:25 DEBUG "value": [ 05:19:25 DEBUG 1712722765, 05:19:25 DEBUG "1252.4666666666667" 05:19:25 DEBUG ] 05:19:25 DEBUG } 05:19:25 DEBUG ] 05:19:25 DEBUG } 05:19:25 DEBUG } 05:19:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712722766" 05:19:26 INFO [http_cmd]: http status code OK 05:19:26 DEBUG --- status code --- 05:19:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:26 DEBUG --- http response --- 05:19:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:19:26 DEBUG ----- output ----- 05:19:26 DEBUG { 05:19:26 DEBUG "status": "success", 05:19:26 DEBUG "data": { 05:19:26 DEBUG "resultType": "vector", 05:19:26 DEBUG "result": [ 05:19:26 DEBUG { 05:19:26 DEBUG "metric": { 05:19:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:19:26 DEBUG }, 05:19:26 DEBUG "value": [ 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG [...] 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:19:26 DEBUG }, 05:19:26 DEBUG "value": [ 05:19:26 DEBUG 1712722766, 05:19:26 DEBUG "0.0405" 05:19:26 DEBUG ] 05:19:26 DEBUG } 05:19:26 DEBUG ] 05:19:26 DEBUG } 05:19:26 DEBUG } 05:19:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722766" 05:19:26 INFO [http_cmd]: http status code OK 05:19:26 DEBUG --- status code --- 05:19:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:26 DEBUG --- http response --- 05:19:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:19:26 DEBUG ----- output ----- 05:19:26 DEBUG { 05:19:26 DEBUG "status": "success", 05:19:26 DEBUG "data": { 05:19:26 DEBUG "resultType": "vector", 05:19:26 DEBUG "result": [ 05:19:26 DEBUG { 05:19:26 DEBUG "metric": { 05:19:26 DEBUG "pod": "ds-idrepo-0" 05:19:26 DEBUG }, 05:19:26 DEBUG "value": [ 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG [...] 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG "pod": "ds-idrepo-1" 05:19:26 DEBUG }, 05:19:26 DEBUG "value": [ 05:19:26 DEBUG 1712722766, 05:19:26 DEBUG "0" 05:19:26 DEBUG ] 05:19:26 DEBUG } 05:19:26 DEBUG ] 05:19:26 DEBUG } 05:19:26 DEBUG } 05:19:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722766" 05:19:26 INFO [http_cmd]: http status code OK 05:19:26 DEBUG --- status code --- 05:19:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:26 DEBUG --- http response --- 05:19:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:19:26 DEBUG ----- output ----- 05:19:26 DEBUG { 05:19:26 DEBUG "status": "success", 05:19:26 DEBUG "data": { 05:19:26 DEBUG "resultType": "vector", 05:19:26 DEBUG "result": [ 05:19:26 DEBUG { 05:19:26 DEBUG "metric": { 05:19:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:19:26 DEBUG }, 05:19:26 DEBUG "value": [ 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG [...] 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:19:26 DEBUG }, 05:19:26 DEBUG "value": [ 05:19:26 DEBUG 1712722766, 05:19:26 DEBUG "1254.8333333333333" 05:19:26 DEBUG ] 05:19:26 DEBUG } 05:19:26 DEBUG ] 05:19:26 DEBUG } 05:19:26 DEBUG } 05:19:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722766" 05:19:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712722766" 05:19:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722766" 05:19:26 INFO [http_cmd]: http status code OK 05:19:26 DEBUG --- status code --- 05:19:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:26 DEBUG --- http response --- 05:19:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:19:26 DEBUG ----- output ----- 05:19:26 DEBUG { 05:19:26 DEBUG "status": "success", 05:19:26 DEBUG "data": { 05:19:26 DEBUG "resultType": "vector", 05:19:26 DEBUG "result": [ 05:19:26 DEBUG { 05:19:26 DEBUG "metric": { 05:19:26 DEBUG "pod": "ds-idrepo-0" 05:19:26 DEBUG }, 05:19:26 DEBUG "value": [ 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG [...] 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG "pod": "ds-idrepo-1" 05:19:26 DEBUG }, 05:19:26 DEBUG "value": [ 05:19:26 DEBUG 1712722766, 05:19:26 DEBUG "0" 05:19:26 DEBUG ] 05:19:26 DEBUG } 05:19:26 DEBUG ] 05:19:26 DEBUG } 05:19:26 DEBUG } 05:19:26 INFO [http_cmd]: http status code OK 05:19:26 DEBUG --- status code --- 05:19:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:26 DEBUG --- http response --- 05:19:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:19:26 DEBUG ----- output ----- 05:19:26 DEBUG { 05:19:26 DEBUG "status": "success", 05:19:26 DEBUG "data": { 05:19:26 DEBUG "resultType": "vector", 05:19:26 DEBUG "result": [ 05:19:26 DEBUG { 05:19:26 DEBUG "metric": { 05:19:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:19:26 DEBUG }, 05:19:26 DEBUG "value": [ 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG [...] 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:19:26 DEBUG }, 05:19:26 DEBUG "value": [ 05:19:26 DEBUG 1712722766, 05:19:26 INFO [http_cmd]: http status code OK 05:19:26 DEBUG "2484809728" 05:19:26 DEBUG --- status code --- 05:19:26 DEBUG ] 05:19:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:26 DEBUG } 05:19:26 DEBUG --- http response --- 05:19:26 DEBUG ] 05:19:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:19:26 DEBUG } 05:19:26 DEBUG ----- output ----- 05:19:26 DEBUG } 05:19:26 DEBUG { 05:19:26 DEBUG "status": "success", 05:19:26 DEBUG "data": { 05:19:26 DEBUG "resultType": "vector", 05:19:26 DEBUG "result": [ 05:19:26 DEBUG { 05:19:26 DEBUG "metric": { 05:19:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:19:26 DEBUG }, 05:19:26 DEBUG "value": [ 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG [...] 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:19:26 DEBUG }, 05:19:26 DEBUG "value": [ 05:19:26 DEBUG 1712722766, 05:19:26 DEBUG "0" 05:19:26 DEBUG ] 05:19:26 DEBUG } 05:19:26 DEBUG ] 05:19:26 DEBUG } 05:19:26 DEBUG } 05:19:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722766" 05:19:26 INFO [http_cmd]: http status code OK 05:19:26 DEBUG --- status code --- 05:19:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:26 DEBUG --- http response --- 05:19:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:19:26 DEBUG ----- output ----- 05:19:26 DEBUG { 05:19:26 DEBUG "status": "success", 05:19:26 DEBUG "data": { 05:19:26 DEBUG "resultType": "vector", 05:19:26 DEBUG "result": [ 05:19:26 DEBUG { 05:19:26 DEBUG "metric": { 05:19:26 DEBUG "instance": "10.142.0.25:9100" 05:19:26 DEBUG }, 05:19:26 DEBUG "value": [ 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG [...] 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG "instance": "10.142.0.77:9100" 05:19:26 DEBUG }, 05:19:26 DEBUG "value": [ 05:19:26 DEBUG 1712722766, 05:19:26 DEBUG "0.015333333333333537" 05:19:26 DEBUG ] 05:19:26 DEBUG } 05:19:26 DEBUG ] 05:19:26 DEBUG } 05:19:26 DEBUG } 05:19:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722766" 05:19:26 INFO [http_cmd]: http status code OK 05:19:26 DEBUG --- status code --- 05:19:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:19:26 DEBUG --- http response --- 05:19:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:19:26 DEBUG ----- output ----- 05:19:26 DEBUG { 05:19:26 DEBUG "status": "success", 05:19:26 DEBUG "data": { 05:19:26 DEBUG "resultType": "vector", 05:19:26 DEBUG "result": [ 05:19:26 DEBUG { 05:19:26 DEBUG "metric": { 05:19:26 DEBUG "instance": "10.142.0.25:9100" 05:19:26 DEBUG }, 05:19:26 DEBUG "value": [ 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG [...] 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG 05:19:26 DEBUG "instance": "10.142.0.77:9100" 05:19:26 DEBUG }, 05:19:26 DEBUG "value": [ 05:19:26 DEBUG 1712722766, 05:19:26 DEBUG "37625713" 05:19:26 DEBUG ] 05:19:26 DEBUG } 05:19:26 DEBUG ] 05:19:26 DEBUG } 05:19:26 DEBUG } 05:19:40 INFO 05:19:40 INFO [loop_until]: kubectl --namespace=xlou top pods 05:19:40 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:19:40 INFO [loop_until]: OK (rc = 0) 05:19:40 DEBUG --- stdout --- 05:19:40 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 8936m 8808Mi am-654c4fcbcf-9scrj 8765m 8418Mi am-654c4fcbcf-v5qb5 7324m 8403Mi ds-cts-0 7324m 13165Mi ds-cts-1 7401m 13155Mi ds-cts-2 6508m 12735Mi ds-idrepo-0 9409m 5621Mi ds-idrepo-1 326m 5669Mi ds-idrepo-2 352m 5698Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 7m 70Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3576m 1654Mi 05:19:40 DEBUG --- stderr --- 05:19:40 DEBUG 05:19:47 INFO 05:19:47 INFO [loop_until]: kubectl --namespace=xlou top node 05:19:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:19:47 INFO [loop_until]: OK (rc = 0) 05:19:47 DEBUG --- stdout --- 05:19:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 92m 0% 2305Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 7895m 49% 9276Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 82m 0% 3260Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10776m 67% 6514Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9260m 58% 9313Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9863m 62% 9646Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 79m 0% 848Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9454m 59% 6276Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 7345m 46% 13701Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 6646m 41% 13291Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 388m 2% 6359Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7567m 47% 13799Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 365m 2% 6337Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3688m 23% 5344Mi 9% 05:19:47 DEBUG --- stderr --- 05:19:47 DEBUG 05:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722823" 05:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712722823" 05:20:23 ERROR [http_cmd]: ERROR 05:20:23 ERROR --- status code --- 05:20:23 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:20:23 ERROR --- http response --- 05:20:23 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:20:23 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:20:23 INFO [http_cmd]: http status code OK 05:20:23 DEBUG --- status code --- 05:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:23 DEBUG --- http response --- 05:20:23 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722823" 05:20:23 INFO [http_cmd]: http status code OK 05:20:23 DEBUG --- status code --- 05:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:23 DEBUG --- http response --- 05:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:20:23 DEBUG ----- output ----- 05:20:23 DEBUG { 05:20:23 DEBUG "status": "success", 05:20:23 DEBUG "data": { 05:20:23 DEBUG "resultType": "vector", 05:20:23 DEBUG "result": [ 05:20:23 DEBUG { 05:20:23 DEBUG "metric": { 05:20:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:20:23 DEBUG }, 05:20:23 DEBUG "value": [ 05:20:23 DEBUG 05:20:23 DEBUG 05:20:23 DEBUG 05:20:23 DEBUG [...] 05:20:23 DEBUG 05:20:23 DEBUG 05:20:23 DEBUG 05:20:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:20:23 DEBUG }, 05:20:23 DEBUG "value": [ 05:20:23 DEBUG 1712722823, 05:20:23 DEBUG "0" 05:20:23 DEBUG ] 05:20:23 DEBUG } 05:20:23 DEBUG ] 05:20:23 DEBUG } 05:20:23 DEBUG } 05:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722823" 05:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722823" 05:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722823" 05:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722823" 05:20:23 INFO [http_cmd]: http status code OK 05:20:23 DEBUG --- status code --- 05:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:23 DEBUG --- http response --- 05:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:20:23 DEBUG ----- output ----- 05:20:23 DEBUG { 05:20:23 DEBUG "status": "success", 05:20:23 DEBUG "data": { 05:20:23 DEBUG "resultType": "vector", 05:20:23 DEBUG "result": [ 05:20:23 DEBUG { 05:20:23 DEBUG "metric": { 05:20:23 DEBUG "pod": "ds-idrepo-0" 05:20:23 DEBUG }, 05:20:23 DEBUG "value": [ 05:20:23 DEBUG 05:20:23 DEBUG 05:20:23 DEBUG 05:20:23 DEBUG [...] 05:20:23 DEBUG 05:20:23 DEBUG 05:20:23 DEBUG 05:20:23 DEBUG "pod": "ds-cts-2" 05:20:23 DEBUG }, 05:20:23 DEBUG "value": [ 05:20:23 DEBUG 1712722823, 05:20:23 DEBUG "29.933333333333334" 05:20:23 DEBUG ] 05:20:23 DEBUG } 05:20:23 DEBUG ] 05:20:23 DEBUG } 05:20:23 DEBUG } 05:20:23 INFO [http_cmd]: http status code OK 05:20:23 DEBUG --- status code --- 05:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:23 DEBUG --- http response --- 05:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:20:23 DEBUG ----- output ----- 05:20:23 DEBUG { 05:20:23 DEBUG "status": "success", 05:20:23 DEBUG "data": { 05:20:23 DEBUG "resultType": "vector", 05:20:23 DEBUG "result": [ 05:20:23 DEBUG { 05:20:23 DEBUG "metric": { 05:20:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:20:23 DEBUG }, 05:20:23 DEBUG "value": [ 05:20:23 DEBUG 05:20:23 DEBUG 05:20:23 DEBUG 05:20:23 DEBUG [...] 05:20:23 DEBUG 05:20:23 DEBUG 05:20:23 DEBUG 05:20:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:20:23 DEBUG }, 05:20:23 DEBUG "value": [ 05:20:23 DEBUG 1712722823, 05:20:23 DEBUG "1269.7" 05:20:23 DEBUG ] 05:20:23 DEBUG } 05:20:23 DEBUG ] 05:20:23 DEBUG } 05:20:23 DEBUG } 05:20:23 INFO [http_cmd]: http status code OK 05:20:23 INFO [http_cmd]: http status code OK 05:20:23 DEBUG --- status code --- 05:20:23 DEBUG --- status code --- 05:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:23 DEBUG --- http response --- 05:20:23 DEBUG --- http response --- 05:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:20:23 DEBUG ----- output ----- 05:20:23 DEBUG ----- output ----- 05:20:23 DEBUG { 05:20:23 DEBUG { 05:20:23 DEBUG "status": "success", 05:20:23 DEBUG "status": "success", 05:20:23 DEBUG "data": { 05:20:23 DEBUG "data": { 05:20:23 DEBUG "resultType": "vector", 05:20:23 DEBUG "resultType": "vector", 05:20:23 DEBUG "result": [ 05:20:23 DEBUG "result": [ 05:20:23 DEBUG { 05:20:23 DEBUG "metric": { 05:20:23 DEBUG "instance": "10.142.0.25:9100" 05:20:23 DEBUG }, 05:20:23 DEBUG "value": [ 05:20:23 DEBUG 05:20:23 DEBUG 05:20:23 DEBUG 05:20:23 DEBUG [...] 05:20:23 DEBUG 05:20:23 DEBUG 05:20:23 DEBUG 05:20:23 DEBUG "instance": "10.142.0.77:9100" 05:20:23 DEBUG }, 05:20:23 DEBUG "value": [ 05:20:23 DEBUG 1712722823, 05:20:23 DEBUG "270563183616" 05:20:23 DEBUG ] 05:20:23 DEBUG } 05:20:23 DEBUG ] 05:20:23 DEBUG } 05:20:23 DEBUG } 05:20:23 DEBUG { 05:20:23 DEBUG "metric": { 05:20:23 DEBUG "pod": "ds-idrepo-0" 05:20:23 DEBUG }, 05:20:23 DEBUG "value": [ 05:20:23 DEBUG 05:20:23 DEBUG 05:20:23 DEBUG 05:20:23 DEBUG [...] 05:20:23 DEBUG 05:20:23 DEBUG 05:20:23 DEBUG 05:20:23 DEBUG "pod": "ds-cts-2" 05:20:23 DEBUG }, 05:20:23 DEBUG "value": [ 05:20:23 DEBUG 1712722823, 05:20:23 DEBUG "0" 05:20:23 DEBUG ] 05:20:23 DEBUG } 05:20:23 DEBUG ] 05:20:23 DEBUG } 05:20:23 DEBUG } 05:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722823" 05:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722823" 05:20:23 INFO [http_cmd]: http status code OK 05:20:23 DEBUG --- status code --- 05:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:23 DEBUG --- http response --- 05:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:20:23 DEBUG ----- output ----- 05:20:23 INFO [http_cmd]: http status code OK 05:20:23 DEBUG { 05:20:23 DEBUG --- status code --- 05:20:23 DEBUG "status": "success", 05:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:23 DEBUG "data": { 05:20:23 DEBUG --- http response --- 05:20:23 DEBUG "resultType": "vector", 05:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:20:23 DEBUG "result": [ 05:20:23 DEBUG { 05:20:23 DEBUG ----- output ----- 05:20:23 DEBUG "metric": { 05:20:23 DEBUG { 05:20:23 DEBUG "instance": "10.142.0.25:9100" 05:20:23 DEBUG "status": "success", 05:20:23 DEBUG }, 05:20:23 DEBUG "data": { 05:20:23 DEBUG "value": [ 05:20:23 DEBUG "resultType": "vector", 05:20:23 DEBUG 05:20:23 DEBUG 05:20:23 DEBUG 05:20:23 DEBUG [...] 05:20:23 DEBUG "result": [ 05:20:23 DEBUG 05:20:23 DEBUG { 05:20:23 DEBUG 05:20:23 DEBUG "metric": { 05:20:23 DEBUG 05:20:23 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:20:23 DEBUG "instance": "10.142.0.77:9100" 05:20:23 DEBUG }, 05:20:23 DEBUG "value": [ 05:20:23 DEBUG 1712722823, 05:20:23 DEBUG }, 05:20:23 DEBUG "value": [ 05:20:23 DEBUG 05:20:23 DEBUG 05:20:23 DEBUG 05:20:23 DEBUG [...] 05:20:23 DEBUG 05:20:23 DEBUG 05:20:23 DEBUG 05:20:23 DEBUG "37680335.63333333" 05:20:23 DEBUG ] 05:20:23 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 05:20:23 DEBUG }, 05:20:23 DEBUG "value": [ 05:20:23 DEBUG 1712722823, 05:20:23 DEBUG } 05:20:23 DEBUG ] 05:20:23 DEBUG } 05:20:23 DEBUG } 05:20:23 DEBUG "0" 05:20:23 DEBUG ] 05:20:23 DEBUG } 05:20:23 DEBUG ] 05:20:23 DEBUG } 05:20:23 DEBUG } 05:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722823" 05:20:23 INFO [http_cmd]: http status code OK 05:20:23 DEBUG --- status code --- 05:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:23 DEBUG --- http response --- 05:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:20:23 DEBUG ----- output ----- 05:20:23 DEBUG { 05:20:23 DEBUG "status": "success", 05:20:23 DEBUG "data": { 05:20:23 DEBUG "resultType": "vector", 05:20:23 DEBUG "result": [ 05:20:23 DEBUG { 05:20:23 DEBUG "metric": { 05:20:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:20:23 DEBUG }, 05:20:23 DEBUG "value": [ 05:20:23 DEBUG 05:20:23 DEBUG 05:20:23 DEBUG 05:20:23 DEBUG [...] 05:20:23 DEBUG 05:20:23 DEBUG 05:20:23 DEBUG 05:20:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:20:23 DEBUG }, 05:20:23 DEBUG "value": [ 05:20:23 DEBUG 1712722823, 05:20:23 DEBUG "226388" 05:20:23 DEBUG ] 05:20:23 DEBUG } 05:20:23 DEBUG ] 05:20:23 DEBUG } 05:20:23 DEBUG } 05:20:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722824" 05:20:24 INFO [http_cmd]: http status code OK 05:20:24 DEBUG --- status code --- 05:20:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:24 DEBUG --- http response --- 05:20:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:20:24 DEBUG ----- output ----- 05:20:24 DEBUG { 05:20:24 DEBUG "status": "success", 05:20:24 DEBUG "data": { 05:20:24 DEBUG "resultType": "vector", 05:20:24 DEBUG "result": [ 05:20:24 DEBUG { 05:20:24 DEBUG "metric": { 05:20:24 DEBUG "pod": "ds-idrepo-0" 05:20:24 DEBUG }, 05:20:24 DEBUG "value": [ 05:20:24 DEBUG 05:20:24 DEBUG 05:20:24 DEBUG 05:20:24 DEBUG [...] 05:20:24 DEBUG 05:20:24 DEBUG 05:20:24 DEBUG 05:20:24 DEBUG "pod": "ds-idrepo-1" 05:20:24 DEBUG }, 05:20:24 DEBUG "value": [ 05:20:24 DEBUG 1712722824, 05:20:24 DEBUG "0" 05:20:24 DEBUG ] 05:20:24 DEBUG } 05:20:24 DEBUG ] 05:20:24 DEBUG } 05:20:24 DEBUG } 05:20:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722824" 05:20:24 INFO [http_cmd]: http status code OK 05:20:24 DEBUG --- status code --- 05:20:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:24 DEBUG --- http response --- 05:20:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:20:24 DEBUG ----- output ----- 05:20:24 DEBUG { 05:20:24 DEBUG "status": "success", 05:20:24 DEBUG "data": { 05:20:24 DEBUG "resultType": "vector", 05:20:24 DEBUG "result": [ 05:20:24 DEBUG { 05:20:24 DEBUG "metric": { 05:20:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:20:24 DEBUG }, 05:20:24 DEBUG "value": [ 05:20:24 DEBUG 05:20:24 DEBUG 05:20:24 DEBUG 05:20:24 DEBUG [...] 05:20:24 DEBUG 05:20:24 DEBUG 05:20:24 DEBUG 05:20:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:20:24 DEBUG }, 05:20:24 DEBUG "value": [ 05:20:24 DEBUG 1712722824, 05:20:24 DEBUG "1273.2" 05:20:24 DEBUG ] 05:20:24 DEBUG } 05:20:24 DEBUG ] 05:20:24 DEBUG } 05:20:24 DEBUG } 05:20:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722824" 05:20:24 INFO [http_cmd]: http status code OK 05:20:24 DEBUG --- status code --- 05:20:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:24 DEBUG --- http response --- 05:20:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:20:24 DEBUG ----- output ----- 05:20:24 DEBUG { 05:20:24 DEBUG "status": "success", 05:20:24 DEBUG "data": { 05:20:24 DEBUG "resultType": "vector", 05:20:24 DEBUG "result": [ 05:20:24 DEBUG { 05:20:24 DEBUG "metric": { 05:20:24 DEBUG "instance": "10.142.0.25:9100" 05:20:24 DEBUG }, 05:20:24 DEBUG "value": [ 05:20:24 DEBUG 05:20:24 DEBUG 05:20:24 DEBUG 05:20:24 DEBUG [...] 05:20:24 DEBUG 05:20:24 DEBUG 05:20:24 DEBUG 05:20:24 DEBUG "instance": "10.142.0.77:9100" 05:20:24 DEBUG }, 05:20:24 DEBUG "value": [ 05:20:24 DEBUG 1712722824, 05:20:24 DEBUG "0.13396666666666746" 05:20:24 DEBUG ] 05:20:24 DEBUG } 05:20:24 DEBUG ] 05:20:24 DEBUG } 05:20:24 DEBUG } 05:20:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722825" 05:20:25 INFO [http_cmd]: http status code OK 05:20:25 DEBUG --- status code --- 05:20:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:25 DEBUG --- http response --- 05:20:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:20:25 DEBUG ----- output ----- 05:20:25 DEBUG { 05:20:25 DEBUG "status": "success", 05:20:25 DEBUG "data": { 05:20:25 DEBUG "resultType": "vector", 05:20:25 DEBUG "result": [ 05:20:25 DEBUG { 05:20:25 DEBUG "metric": { 05:20:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:20:25 DEBUG }, 05:20:25 DEBUG "value": [ 05:20:25 DEBUG 05:20:25 DEBUG 05:20:25 DEBUG 05:20:25 DEBUG [...] 05:20:25 DEBUG 05:20:25 DEBUG 05:20:25 DEBUG 05:20:25 DEBUG "pod": "ds-idrepo-2" 05:20:25 DEBUG }, 05:20:25 DEBUG "value": [ 05:20:25 DEBUG 1712722825, 05:20:25 DEBUG "285.06803076173577" 05:20:25 DEBUG ] 05:20:25 DEBUG } 05:20:25 DEBUG ] 05:20:25 DEBUG } 05:20:25 DEBUG } 05:20:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712722825" 05:20:25 INFO [http_cmd]: http status code OK 05:20:25 DEBUG --- status code --- 05:20:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:25 DEBUG --- http response --- 05:20:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:20:25 DEBUG ----- output ----- 05:20:25 DEBUG { 05:20:25 DEBUG "status": "success", 05:20:25 DEBUG "data": { 05:20:25 DEBUG "resultType": "vector", 05:20:25 DEBUG "result": [ 05:20:25 DEBUG { 05:20:25 DEBUG "metric": { 05:20:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:20:25 DEBUG }, 05:20:25 DEBUG "value": [ 05:20:25 DEBUG 05:20:25 DEBUG 05:20:25 DEBUG 05:20:25 DEBUG [...] 05:20:25 DEBUG 05:20:25 DEBUG 05:20:25 DEBUG 05:20:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:20:25 DEBUG }, 05:20:25 DEBUG "value": [ 05:20:25 DEBUG 1712722825, 05:20:25 DEBUG "3.5989096190111174" 05:20:25 DEBUG ] 05:20:25 DEBUG } 05:20:25 DEBUG ] 05:20:25 DEBUG } 05:20:25 DEBUG } 05:20:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712722825" 05:20:25 INFO [http_cmd]: http status code OK 05:20:25 DEBUG --- status code --- 05:20:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:25 DEBUG --- http response --- 05:20:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:20:25 DEBUG ----- output ----- 05:20:25 DEBUG { 05:20:25 DEBUG "status": "success", 05:20:25 DEBUG "data": { 05:20:25 DEBUG "resultType": "vector", 05:20:25 DEBUG "result": [ 05:20:25 DEBUG { 05:20:25 DEBUG "metric": { 05:20:25 DEBUG "pod": "ds-idrepo-0" 05:20:25 DEBUG }, 05:20:25 DEBUG "value": [ 05:20:25 DEBUG 05:20:25 DEBUG 05:20:25 DEBUG 05:20:25 DEBUG [...] 05:20:25 DEBUG 05:20:25 DEBUG 05:20:25 DEBUG 05:20:25 DEBUG "pod": "ds-cts-2" 05:20:25 DEBUG }, 05:20:25 DEBUG "value": [ 05:20:25 DEBUG 1712722825, 05:20:25 DEBUG "0" 05:20:25 DEBUG ] 05:20:25 DEBUG } 05:20:25 DEBUG ] 05:20:25 DEBUG } 05:20:25 DEBUG } 05:20:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722825" 05:20:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722825" 05:20:25 INFO [http_cmd]: http status code OK 05:20:25 INFO [http_cmd]: http status code OK 05:20:25 DEBUG --- status code --- 05:20:25 DEBUG --- status code --- 05:20:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:25 DEBUG --- http response --- 05:20:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:20:25 DEBUG --- http response --- 05:20:25 DEBUG ----- output ----- 05:20:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:20:25 DEBUG { 05:20:25 DEBUG ----- output ----- 05:20:25 DEBUG "status": "success", 05:20:25 DEBUG { 05:20:25 DEBUG "data": { 05:20:25 DEBUG "status": "success", 05:20:25 DEBUG "resultType": "vector", 05:20:25 DEBUG "data": { 05:20:25 DEBUG "result": [ 05:20:25 DEBUG "resultType": "vector", 05:20:25 DEBUG { 05:20:25 DEBUG "result": [ 05:20:25 DEBUG "metric": { 05:20:25 DEBUG { 05:20:25 DEBUG "pod": "ds-idrepo-2" 05:20:25 DEBUG "metric": { 05:20:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:20:25 DEBUG }, 05:20:25 DEBUG }, 05:20:25 DEBUG "value": [ 05:20:25 DEBUG "value": [ 05:20:25 DEBUG 05:20:25 DEBUG 05:20:25 DEBUG 05:20:25 DEBUG 05:20:25 DEBUG 05:20:25 DEBUG [...] 05:20:25 DEBUG 05:20:25 DEBUG 05:20:25 DEBUG [...] 05:20:25 DEBUG 05:20:25 DEBUG 05:20:25 DEBUG "pod": "ds-cts-2" 05:20:25 DEBUG }, 05:20:25 DEBUG "value": [ 05:20:25 DEBUG 1712722825, 05:20:25 DEBUG "0.00011720555555555555" 05:20:25 DEBUG 05:20:25 DEBUG ] 05:20:25 DEBUG 05:20:25 DEBUG } 05:20:25 DEBUG 05:20:25 DEBUG ] 05:20:25 DEBUG "pod": "ds-idrepo-2" 05:20:25 DEBUG } 05:20:25 DEBUG }, 05:20:25 DEBUG } 05:20:25 DEBUG "value": [ 05:20:25 DEBUG 1712722825, 05:20:25 DEBUG "0.33080799672896694" 05:20:25 DEBUG ] 05:20:25 DEBUG } 05:20:25 DEBUG ] 05:20:25 DEBUG } 05:20:25 DEBUG } 05:20:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722826" 05:20:26 INFO [http_cmd]: http status code OK 05:20:26 DEBUG --- status code --- 05:20:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:26 DEBUG --- http response --- 05:20:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:20:26 DEBUG ----- output ----- 05:20:26 DEBUG { 05:20:26 DEBUG "status": "success", 05:20:26 DEBUG "data": { 05:20:26 DEBUG "resultType": "vector", 05:20:26 DEBUG "result": [ 05:20:26 DEBUG { 05:20:26 DEBUG "metric": { 05:20:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:20:26 DEBUG }, 05:20:26 DEBUG "value": [ 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG [...] 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:20:26 DEBUG }, 05:20:26 DEBUG "value": [ 05:20:26 DEBUG 1712722826, 05:20:26 DEBUG "1273.3" 05:20:26 DEBUG ] 05:20:26 DEBUG } 05:20:26 DEBUG ] 05:20:26 DEBUG } 05:20:26 DEBUG } 05:20:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712722826" 05:20:26 INFO [http_cmd]: http status code OK 05:20:26 DEBUG --- status code --- 05:20:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:26 DEBUG --- http response --- 05:20:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:20:26 DEBUG ----- output ----- 05:20:26 DEBUG { 05:20:26 DEBUG "status": "success", 05:20:26 DEBUG "data": { 05:20:26 DEBUG "resultType": "vector", 05:20:26 DEBUG "result": [ 05:20:26 DEBUG { 05:20:26 DEBUG "metric": { 05:20:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:20:26 DEBUG }, 05:20:26 DEBUG "value": [ 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG [...] 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:20:26 DEBUG }, 05:20:26 DEBUG "value": [ 05:20:26 DEBUG 1712722826, 05:20:26 DEBUG "0.041" 05:20:26 DEBUG ] 05:20:26 DEBUG } 05:20:26 DEBUG ] 05:20:26 DEBUG } 05:20:26 DEBUG } 05:20:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722826" 05:20:26 INFO [http_cmd]: http status code OK 05:20:26 DEBUG --- status code --- 05:20:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:26 DEBUG --- http response --- 05:20:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:20:26 DEBUG ----- output ----- 05:20:26 DEBUG { 05:20:26 DEBUG "status": "success", 05:20:26 DEBUG "data": { 05:20:26 DEBUG "resultType": "vector", 05:20:26 DEBUG "result": [ 05:20:26 DEBUG { 05:20:26 DEBUG "metric": { 05:20:26 DEBUG "pod": "ds-idrepo-0" 05:20:26 DEBUG }, 05:20:26 DEBUG "value": [ 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG [...] 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG "pod": "ds-idrepo-1" 05:20:26 DEBUG }, 05:20:26 DEBUG "value": [ 05:20:26 DEBUG 1712722826, 05:20:26 DEBUG "0" 05:20:26 DEBUG ] 05:20:26 DEBUG } 05:20:26 DEBUG ] 05:20:26 DEBUG } 05:20:26 DEBUG } 05:20:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722826" 05:20:26 INFO [http_cmd]: http status code OK 05:20:26 DEBUG --- status code --- 05:20:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:26 DEBUG --- http response --- 05:20:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:20:26 DEBUG ----- output ----- 05:20:26 DEBUG { 05:20:26 DEBUG "status": "success", 05:20:26 DEBUG "data": { 05:20:26 DEBUG "resultType": "vector", 05:20:26 DEBUG "result": [ 05:20:26 DEBUG { 05:20:26 DEBUG "metric": { 05:20:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:20:26 DEBUG }, 05:20:26 DEBUG "value": [ 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG [...] 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:20:26 DEBUG }, 05:20:26 DEBUG "value": [ 05:20:26 DEBUG 1712722826, 05:20:26 DEBUG "1268.6666666666667" 05:20:26 DEBUG ] 05:20:26 DEBUG } 05:20:26 DEBUG ] 05:20:26 DEBUG } 05:20:26 DEBUG } 05:20:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722826" 05:20:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712722826" 05:20:26 INFO [http_cmd]: http status code OK 05:20:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722826" 05:20:26 DEBUG --- status code --- 05:20:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:26 DEBUG --- http response --- 05:20:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:20:26 DEBUG ----- output ----- 05:20:26 DEBUG { 05:20:26 DEBUG "status": "success", 05:20:26 DEBUG "data": { 05:20:26 DEBUG "resultType": "vector", 05:20:26 DEBUG "result": [ 05:20:26 DEBUG { 05:20:26 DEBUG "metric": { 05:20:26 DEBUG "pod": "ds-idrepo-0" 05:20:26 DEBUG }, 05:20:26 DEBUG "value": [ 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG [...] 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG "pod": "ds-idrepo-1" 05:20:26 DEBUG }, 05:20:26 DEBUG "value": [ 05:20:26 DEBUG 1712722826, 05:20:26 DEBUG "0" 05:20:26 DEBUG ] 05:20:26 DEBUG } 05:20:26 DEBUG ] 05:20:26 DEBUG } 05:20:26 DEBUG } 05:20:26 INFO [http_cmd]: http status code OK 05:20:26 DEBUG --- status code --- 05:20:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:26 DEBUG --- http response --- 05:20:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:20:26 DEBUG ----- output ----- 05:20:26 DEBUG { 05:20:26 DEBUG "status": "success", 05:20:26 DEBUG "data": { 05:20:26 DEBUG "resultType": "vector", 05:20:26 DEBUG "result": [ 05:20:26 DEBUG { 05:20:26 DEBUG "metric": { 05:20:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:20:26 DEBUG }, 05:20:26 DEBUG "value": [ 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG [...] 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:20:26 DEBUG }, 05:20:26 DEBUG "value": [ 05:20:26 DEBUG 1712722826, 05:20:26 DEBUG "2484989952" 05:20:26 DEBUG ] 05:20:26 DEBUG } 05:20:26 DEBUG ] 05:20:26 DEBUG } 05:20:26 DEBUG } 05:20:26 INFO [http_cmd]: http status code OK 05:20:26 DEBUG --- status code --- 05:20:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:26 DEBUG --- http response --- 05:20:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 05:20:26 DEBUG ----- output ----- 05:20:26 DEBUG { 05:20:26 DEBUG "status": "success", 05:20:26 DEBUG "data": { 05:20:26 DEBUG "resultType": "vector", 05:20:26 DEBUG "result": [ 05:20:26 DEBUG { 05:20:26 DEBUG "metric": { 05:20:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:20:26 DEBUG }, 05:20:26 DEBUG "value": [ 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG [...] 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG "pod": "ds-idrepo-2" 05:20:26 DEBUG }, 05:20:26 DEBUG "value": [ 05:20:26 DEBUG 1712722826, 05:20:26 DEBUG "3705.884399902565" 05:20:26 DEBUG ] 05:20:26 DEBUG } 05:20:26 DEBUG ] 05:20:26 DEBUG } 05:20:26 DEBUG } 05:20:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722826" 05:20:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722826" 05:20:26 INFO [http_cmd]: http status code OK 05:20:26 DEBUG --- status code --- 05:20:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:26 DEBUG --- http response --- 05:20:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:20:26 DEBUG ----- output ----- 05:20:26 DEBUG { 05:20:26 DEBUG "status": "success", 05:20:26 DEBUG "data": { 05:20:26 DEBUG "resultType": "vector", 05:20:26 DEBUG "result": [ 05:20:26 DEBUG { 05:20:26 DEBUG "metric": { 05:20:26 DEBUG "instance": "10.142.0.25:9100" 05:20:26 DEBUG }, 05:20:26 DEBUG "value": [ 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG [...] 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG "instance": "10.142.0.77:9100" 05:20:26 DEBUG }, 05:20:26 DEBUG "value": [ 05:20:26 DEBUG 1712722826, 05:20:26 DEBUG "0.04600000000000013" 05:20:26 DEBUG ] 05:20:26 DEBUG } 05:20:26 DEBUG ] 05:20:26 DEBUG } 05:20:26 DEBUG } 05:20:26 INFO [http_cmd]: http status code OK 05:20:26 DEBUG --- status code --- 05:20:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:20:26 DEBUG --- http response --- 05:20:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:20:26 DEBUG ----- output ----- 05:20:26 DEBUG { 05:20:26 DEBUG "status": "success", 05:20:26 DEBUG "data": { 05:20:26 DEBUG "resultType": "vector", 05:20:26 DEBUG "result": [ 05:20:26 DEBUG { 05:20:26 DEBUG "metric": { 05:20:26 DEBUG "instance": "10.142.0.25:9100" 05:20:26 DEBUG }, 05:20:26 DEBUG "value": [ 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG [...] 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG 05:20:26 DEBUG "instance": "10.142.0.77:9100" 05:20:26 DEBUG }, 05:20:26 DEBUG "value": [ 05:20:26 DEBUG 1712722826, 05:20:26 DEBUG "37679769.733333334" 05:20:26 DEBUG ] 05:20:26 DEBUG } 05:20:26 DEBUG ] 05:20:26 DEBUG } 05:20:26 DEBUG } 05:20:40 INFO 05:20:40 INFO [loop_until]: kubectl --namespace=xlou top pods 05:20:40 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:20:41 INFO [loop_until]: OK (rc = 0) 05:20:41 DEBUG --- stdout --- 05:20:41 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 9173m 8804Mi am-654c4fcbcf-9scrj 8791m 8423Mi am-654c4fcbcf-v5qb5 7253m 8400Mi ds-cts-0 7293m 13314Mi ds-cts-1 7498m 13260Mi ds-cts-2 6653m 13259Mi ds-idrepo-0 9408m 5621Mi ds-idrepo-1 311m 5668Mi ds-idrepo-2 395m 5698Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 5m 2213Mi lodemon-b776b7d88-thrqf 2m 70Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3579m 1655Mi 05:20:41 DEBUG --- stderr --- 05:20:41 DEBUG 05:20:47 INFO 05:20:47 INFO [loop_until]: kubectl --namespace=xlou top node 05:20:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:20:47 INFO [loop_until]: OK (rc = 0) 05:20:47 DEBUG --- stdout --- 05:20:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 99m 0% 2308Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8088m 50% 9280Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 82m 0% 3254Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10742m 67% 6499Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9519m 59% 9307Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9648m 60% 9644Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 77m 0% 853Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9722m 61% 6278Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 7489m 47% 13891Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 6567m 41% 13910Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 388m 2% 6361Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7569m 47% 13869Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 377m 2% 6326Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3566m 22% 5340Mi 9% 05:20:47 DEBUG --- stderr --- 05:20:47 DEBUG 05:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712722883" 05:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722883" 05:21:23 ERROR [http_cmd]: ERROR 05:21:23 INFO [http_cmd]: http status code OK 05:21:23 ERROR --- status code --- 05:21:23 DEBUG --- status code --- 05:21:23 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:23 ERROR --- http response --- 05:21:23 DEBUG --- http response --- 05:21:23 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:21:23 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:21:23 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722883" 05:21:23 INFO [http_cmd]: http status code OK 05:21:23 DEBUG --- status code --- 05:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:23 DEBUG --- http response --- 05:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:21:23 DEBUG ----- output ----- 05:21:23 DEBUG { 05:21:23 DEBUG "status": "success", 05:21:23 DEBUG "data": { 05:21:23 DEBUG "resultType": "vector", 05:21:23 DEBUG "result": [ 05:21:23 DEBUG { 05:21:23 DEBUG "metric": { 05:21:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:21:23 DEBUG }, 05:21:23 DEBUG "value": [ 05:21:23 DEBUG 05:21:23 DEBUG 05:21:23 DEBUG 05:21:23 DEBUG [...] 05:21:23 DEBUG 05:21:23 DEBUG 05:21:23 DEBUG 05:21:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:21:23 DEBUG }, 05:21:23 DEBUG "value": [ 05:21:23 DEBUG 1712722883, 05:21:23 DEBUG "0" 05:21:23 DEBUG ] 05:21:23 DEBUG } 05:21:23 DEBUG ] 05:21:23 DEBUG } 05:21:23 DEBUG } 05:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722883" 05:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722883" 05:21:23 INFO [http_cmd]: http status code OK 05:21:23 DEBUG --- status code --- 05:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:23 DEBUG --- http response --- 05:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:21:23 DEBUG ----- output ----- 05:21:23 DEBUG { 05:21:23 DEBUG "status": "success", 05:21:23 DEBUG "data": { 05:21:23 DEBUG "resultType": "vector", 05:21:23 DEBUG "result": [ 05:21:23 DEBUG { 05:21:23 DEBUG "metric": { 05:21:23 DEBUG "pod": "ds-idrepo-0" 05:21:23 DEBUG }, 05:21:23 DEBUG "value": [ 05:21:23 DEBUG 05:21:23 DEBUG 05:21:23 DEBUG 05:21:23 DEBUG [...] 05:21:23 DEBUG 05:21:23 DEBUG 05:21:23 DEBUG 05:21:23 DEBUG "pod": "ds-cts-2" 05:21:23 DEBUG }, 05:21:23 DEBUG "value": [ 05:21:23 DEBUG 1712722883, 05:21:23 DEBUG "29.633333333333333" 05:21:23 DEBUG ] 05:21:23 DEBUG } 05:21:23 DEBUG ] 05:21:23 DEBUG } 05:21:23 DEBUG } 05:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722883" 05:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722883" 05:21:23 INFO [http_cmd]: http status code OK 05:21:23 DEBUG --- status code --- 05:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:23 DEBUG --- http response --- 05:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:21:23 DEBUG ----- output ----- 05:21:23 INFO [http_cmd]: http status code OK 05:21:23 DEBUG --- status code --- 05:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:23 DEBUG --- http response --- 05:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:21:23 DEBUG ----- output ----- 05:21:23 DEBUG { 05:21:23 DEBUG "status": "success", 05:21:23 DEBUG "data": { 05:21:23 DEBUG "resultType": "vector", 05:21:23 DEBUG "result": [ 05:21:23 DEBUG { 05:21:23 DEBUG "metric": { 05:21:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:21:23 DEBUG }, 05:21:23 DEBUG "value": [ 05:21:23 DEBUG 05:21:23 DEBUG 05:21:23 DEBUG 05:21:23 DEBUG [...] 05:21:23 DEBUG 05:21:23 DEBUG 05:21:23 DEBUG 05:21:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:21:23 DEBUG }, 05:21:23 DEBUG "value": [ 05:21:23 DEBUG 1712722883, 05:21:23 DEBUG "1263.6666666666667" 05:21:23 DEBUG ] 05:21:23 DEBUG } 05:21:23 DEBUG ] 05:21:23 DEBUG } 05:21:23 DEBUG } 05:21:23 INFO [http_cmd]: http status code OK 05:21:23 DEBUG --- status code --- 05:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:23 DEBUG --- http response --- 05:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:21:23 DEBUG ----- output ----- 05:21:23 DEBUG { 05:21:23 DEBUG "status": "success", 05:21:23 DEBUG "data": { 05:21:23 DEBUG "resultType": "vector", 05:21:23 DEBUG "result": [ 05:21:23 DEBUG { 05:21:23 DEBUG "metric": { 05:21:23 DEBUG "pod": "ds-idrepo-0" 05:21:23 DEBUG }, 05:21:23 DEBUG "value": [ 05:21:23 DEBUG 05:21:23 DEBUG 05:21:23 DEBUG 05:21:23 DEBUG [...] 05:21:23 DEBUG 05:21:23 DEBUG 05:21:23 DEBUG 05:21:23 DEBUG "pod": "ds-cts-2" 05:21:23 DEBUG }, 05:21:23 DEBUG "value": [ 05:21:23 DEBUG 1712722883, 05:21:23 DEBUG "0" 05:21:23 DEBUG ] 05:21:23 DEBUG { 05:21:23 DEBUG } 05:21:23 DEBUG "status": "success", 05:21:23 DEBUG ] 05:21:23 DEBUG "data": { 05:21:23 DEBUG } 05:21:23 DEBUG } 05:21:23 DEBUG "resultType": "vector", 05:21:23 DEBUG "result": [ 05:21:23 DEBUG { 05:21:23 DEBUG "metric": { 05:21:23 DEBUG "instance": "10.142.0.25:9100" 05:21:23 DEBUG }, 05:21:23 DEBUG "value": [ 05:21:23 DEBUG 05:21:23 DEBUG 05:21:23 DEBUG 05:21:23 DEBUG [...] 05:21:23 DEBUG 05:21:23 DEBUG 05:21:23 DEBUG 05:21:23 DEBUG "instance": "10.142.0.77:9100" 05:21:23 DEBUG }, 05:21:23 DEBUG "value": [ 05:21:23 DEBUG 1712722883, 05:21:23 DEBUG "272311305216" 05:21:23 DEBUG ] 05:21:23 DEBUG } 05:21:23 DEBUG ] 05:21:23 DEBUG } 05:21:23 DEBUG } 05:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722883" 05:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722883" 05:21:23 INFO [http_cmd]: http status code OK 05:21:23 DEBUG --- status code --- 05:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:23 DEBUG --- http response --- 05:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:21:23 INFO [http_cmd]: http status code OK 05:21:23 DEBUG ----- output ----- 05:21:23 DEBUG --- status code --- 05:21:23 DEBUG { 05:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:23 DEBUG "status": "success", 05:21:23 DEBUG --- http response --- 05:21:23 DEBUG "data": { 05:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:21:23 DEBUG "resultType": "vector", 05:21:23 DEBUG ----- output ----- 05:21:23 DEBUG "result": [ 05:21:23 DEBUG { 05:21:23 DEBUG { 05:21:23 DEBUG "status": "success", 05:21:23 DEBUG "metric": { 05:21:23 DEBUG "data": { 05:21:23 DEBUG "instance": "10.142.0.25:9100" 05:21:23 DEBUG "resultType": "vector", 05:21:23 DEBUG }, 05:21:23 DEBUG "result": [ 05:21:23 DEBUG "value": [ 05:21:23 DEBUG { 05:21:23 DEBUG 05:21:23 DEBUG "metric": { 05:21:23 DEBUG 05:21:23 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:21:23 DEBUG 05:21:23 DEBUG }, 05:21:23 DEBUG [...] 05:21:23 DEBUG "value": [ 05:21:23 DEBUG 05:21:23 DEBUG 05:21:23 DEBUG 05:21:23 DEBUG 05:21:23 DEBUG 05:21:23 DEBUG 05:21:23 DEBUG "instance": "10.142.0.77:9100" 05:21:23 DEBUG [...] 05:21:23 DEBUG }, 05:21:23 DEBUG 05:21:23 DEBUG "value": [ 05:21:23 DEBUG 05:21:23 DEBUG 1712722883, 05:21:23 DEBUG 05:21:23 DEBUG "37609419.66666667" 05:21:23 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 05:21:23 DEBUG ] 05:21:23 DEBUG }, 05:21:23 DEBUG } 05:21:23 DEBUG "value": [ 05:21:23 DEBUG ] 05:21:23 DEBUG 1712722883, 05:21:23 DEBUG } 05:21:23 DEBUG "0" 05:21:23 DEBUG } 05:21:23 DEBUG ] 05:21:23 DEBUG } 05:21:23 DEBUG ] 05:21:23 DEBUG } 05:21:23 DEBUG } 05:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722883" 05:21:23 INFO [http_cmd]: http status code OK 05:21:23 DEBUG --- status code --- 05:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:23 DEBUG --- http response --- 05:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:21:23 DEBUG ----- output ----- 05:21:23 DEBUG { 05:21:23 DEBUG "status": "success", 05:21:23 DEBUG "data": { 05:21:23 DEBUG "resultType": "vector", 05:21:23 DEBUG "result": [ 05:21:23 DEBUG { 05:21:23 DEBUG "metric": { 05:21:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:21:23 DEBUG }, 05:21:23 DEBUG "value": [ 05:21:23 DEBUG 05:21:23 DEBUG 05:21:23 DEBUG 05:21:23 DEBUG [...] 05:21:23 DEBUG 05:21:23 DEBUG 05:21:23 DEBUG 05:21:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:21:23 DEBUG }, 05:21:23 DEBUG "value": [ 05:21:23 DEBUG 1712722883, 05:21:23 DEBUG "226798" 05:21:23 DEBUG ] 05:21:23 DEBUG } 05:21:23 DEBUG ] 05:21:23 DEBUG } 05:21:23 DEBUG } 05:21:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722884" 05:21:24 INFO [http_cmd]: http status code OK 05:21:24 DEBUG --- status code --- 05:21:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:24 DEBUG --- http response --- 05:21:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:21:24 DEBUG ----- output ----- 05:21:24 DEBUG { 05:21:24 DEBUG "status": "success", 05:21:24 DEBUG "data": { 05:21:24 DEBUG "resultType": "vector", 05:21:24 DEBUG "result": [ 05:21:24 DEBUG { 05:21:24 DEBUG "metric": { 05:21:24 DEBUG "pod": "ds-idrepo-0" 05:21:24 DEBUG }, 05:21:24 DEBUG "value": [ 05:21:24 DEBUG 05:21:24 DEBUG 05:21:24 DEBUG 05:21:24 DEBUG [...] 05:21:24 DEBUG 05:21:24 DEBUG 05:21:24 DEBUG 05:21:24 DEBUG "pod": "ds-idrepo-1" 05:21:24 DEBUG }, 05:21:24 DEBUG "value": [ 05:21:24 DEBUG 1712722884, 05:21:24 DEBUG "0" 05:21:24 DEBUG ] 05:21:24 DEBUG } 05:21:24 DEBUG ] 05:21:24 DEBUG } 05:21:24 DEBUG } 05:21:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722884" 05:21:24 INFO [http_cmd]: http status code OK 05:21:24 DEBUG --- status code --- 05:21:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:24 DEBUG --- http response --- 05:21:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:21:24 DEBUG ----- output ----- 05:21:24 DEBUG { 05:21:24 DEBUG "status": "success", 05:21:24 DEBUG "data": { 05:21:24 DEBUG "resultType": "vector", 05:21:24 DEBUG "result": [ 05:21:24 DEBUG { 05:21:24 DEBUG "metric": { 05:21:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:21:24 DEBUG }, 05:21:24 DEBUG "value": [ 05:21:24 DEBUG 05:21:24 DEBUG 05:21:24 DEBUG 05:21:24 DEBUG [...] 05:21:24 DEBUG 05:21:24 DEBUG 05:21:24 DEBUG 05:21:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:21:24 DEBUG }, 05:21:24 DEBUG "value": [ 05:21:24 DEBUG 1712722884, 05:21:24 DEBUG "1260.2333333333333" 05:21:24 DEBUG ] 05:21:24 DEBUG } 05:21:24 DEBUG ] 05:21:24 DEBUG } 05:21:24 DEBUG } 05:21:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722884" 05:21:24 INFO [http_cmd]: http status code OK 05:21:24 DEBUG --- status code --- 05:21:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:24 DEBUG --- http response --- 05:21:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:21:24 DEBUG ----- output ----- 05:21:24 DEBUG { 05:21:24 DEBUG "status": "success", 05:21:24 DEBUG "data": { 05:21:24 DEBUG "resultType": "vector", 05:21:24 DEBUG "result": [ 05:21:24 DEBUG { 05:21:24 DEBUG "metric": { 05:21:24 DEBUG "instance": "10.142.0.25:9100" 05:21:24 DEBUG }, 05:21:24 DEBUG "value": [ 05:21:24 DEBUG 05:21:24 DEBUG 05:21:24 DEBUG 05:21:24 DEBUG [...] 05:21:24 DEBUG 05:21:24 DEBUG 05:21:24 DEBUG 05:21:24 DEBUG "instance": "10.142.0.77:9100" 05:21:24 DEBUG }, 05:21:24 DEBUG "value": [ 05:21:24 DEBUG 1712722884, 05:21:24 DEBUG "0.15086666666666915" 05:21:24 DEBUG ] 05:21:24 DEBUG } 05:21:24 DEBUG ] 05:21:24 DEBUG } 05:21:24 DEBUG } 05:21:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722885" 05:21:25 INFO [http_cmd]: http status code OK 05:21:25 DEBUG --- status code --- 05:21:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:25 DEBUG --- http response --- 05:21:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:21:25 DEBUG ----- output ----- 05:21:25 DEBUG { 05:21:25 DEBUG "status": "success", 05:21:25 DEBUG "data": { 05:21:25 DEBUG "resultType": "vector", 05:21:25 DEBUG "result": [ 05:21:25 DEBUG { 05:21:25 DEBUG "metric": { 05:21:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:21:25 DEBUG }, 05:21:25 DEBUG "value": [ 05:21:25 DEBUG 05:21:25 DEBUG 05:21:25 DEBUG 05:21:25 DEBUG [...] 05:21:25 DEBUG 05:21:25 DEBUG 05:21:25 DEBUG 05:21:25 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 05:21:25 DEBUG }, 05:21:25 DEBUG "value": [ 05:21:25 DEBUG 1712722885, 05:21:25 DEBUG "0" 05:21:25 DEBUG ] 05:21:25 DEBUG } 05:21:25 DEBUG ] 05:21:25 DEBUG } 05:21:25 DEBUG } 05:21:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712722885" 05:21:25 INFO [http_cmd]: http status code OK 05:21:25 DEBUG --- status code --- 05:21:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:25 DEBUG --- http response --- 05:21:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:21:25 DEBUG ----- output ----- 05:21:25 DEBUG { 05:21:25 DEBUG "status": "success", 05:21:25 DEBUG "data": { 05:21:25 DEBUG "resultType": "vector", 05:21:25 DEBUG "result": [ 05:21:25 DEBUG { 05:21:25 DEBUG "metric": { 05:21:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:21:25 DEBUG }, 05:21:25 DEBUG "value": [ 05:21:25 DEBUG 05:21:25 DEBUG 05:21:25 DEBUG 05:21:25 DEBUG [...] 05:21:25 DEBUG 05:21:25 DEBUG 05:21:25 DEBUG 05:21:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:21:25 DEBUG }, 05:21:25 DEBUG "value": [ 05:21:25 DEBUG 1712722885, 05:21:25 DEBUG "3.600958269171264" 05:21:25 DEBUG ] 05:21:25 DEBUG } 05:21:25 DEBUG ] 05:21:25 DEBUG } 05:21:25 DEBUG } 05:21:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712722885" 05:21:25 INFO [http_cmd]: http status code OK 05:21:25 DEBUG --- status code --- 05:21:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:25 DEBUG --- http response --- 05:21:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:21:25 DEBUG ----- output ----- 05:21:25 DEBUG { 05:21:25 DEBUG "status": "success", 05:21:25 DEBUG "data": { 05:21:25 DEBUG "resultType": "vector", 05:21:25 DEBUG "result": [ 05:21:25 DEBUG { 05:21:25 DEBUG "metric": { 05:21:25 DEBUG "pod": "ds-idrepo-0" 05:21:25 DEBUG }, 05:21:25 DEBUG "value": [ 05:21:25 DEBUG 05:21:25 DEBUG 05:21:25 DEBUG 05:21:25 DEBUG [...] 05:21:25 DEBUG 05:21:25 DEBUG 05:21:25 DEBUG 05:21:25 DEBUG "pod": "ds-cts-2" 05:21:25 DEBUG }, 05:21:25 DEBUG "value": [ 05:21:25 DEBUG 1712722885, 05:21:25 DEBUG "0" 05:21:25 DEBUG ] 05:21:25 DEBUG } 05:21:25 DEBUG ] 05:21:25 DEBUG } 05:21:25 DEBUG } 05:21:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722885" 05:21:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722885" 05:21:25 INFO [http_cmd]: http status code OK 05:21:25 DEBUG --- status code --- 05:21:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:25 DEBUG --- http response --- 05:21:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:21:25 DEBUG ----- output ----- 05:21:25 DEBUG { 05:21:25 DEBUG "status": "success", 05:21:25 DEBUG "data": { 05:21:25 DEBUG "resultType": "vector", 05:21:25 DEBUG "result": [ 05:21:25 DEBUG { 05:21:25 DEBUG "metric": { 05:21:25 DEBUG "pod": "ds-idrepo-2" 05:21:25 DEBUG }, 05:21:25 DEBUG "value": [ 05:21:25 DEBUG 05:21:25 DEBUG 05:21:25 DEBUG 05:21:25 DEBUG [...] 05:21:25 DEBUG 05:21:25 DEBUG 05:21:25 DEBUG 05:21:25 DEBUG "pod": "ds-cts-2" 05:21:25 DEBUG }, 05:21:25 DEBUG "value": [ 05:21:25 DEBUG 1712722885, 05:21:25 DEBUG "0.0001" 05:21:25 DEBUG ] 05:21:25 DEBUG } 05:21:25 DEBUG ] 05:21:25 DEBUG } 05:21:25 DEBUG } 05:21:25 INFO [http_cmd]: http status code OK 05:21:25 DEBUG --- status code --- 05:21:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:25 DEBUG --- http response --- 05:21:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:21:25 DEBUG ----- output ----- 05:21:25 DEBUG { 05:21:25 DEBUG "status": "success", 05:21:25 DEBUG "data": { 05:21:25 DEBUG "resultType": "vector", 05:21:25 DEBUG "result": [ 05:21:25 DEBUG { 05:21:25 DEBUG "metric": { 05:21:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:21:25 DEBUG }, 05:21:25 DEBUG "value": [ 05:21:25 DEBUG 05:21:25 DEBUG 05:21:25 DEBUG 05:21:25 DEBUG [...] 05:21:25 DEBUG 05:21:25 DEBUG 05:21:25 DEBUG 05:21:25 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 05:21:25 DEBUG }, 05:21:25 DEBUG "value": [ 05:21:25 DEBUG 1712722885, 05:21:25 DEBUG "0" 05:21:25 DEBUG ] 05:21:25 DEBUG } 05:21:25 DEBUG ] 05:21:25 DEBUG } 05:21:25 DEBUG } 05:21:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722886" 05:21:26 INFO [http_cmd]: http status code OK 05:21:26 DEBUG --- status code --- 05:21:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:26 DEBUG --- http response --- 05:21:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:21:26 DEBUG ----- output ----- 05:21:26 DEBUG { 05:21:26 DEBUG "status": "success", 05:21:26 DEBUG "data": { 05:21:26 DEBUG "resultType": "vector", 05:21:26 DEBUG "result": [ 05:21:26 DEBUG { 05:21:26 DEBUG "metric": { 05:21:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:21:26 DEBUG }, 05:21:26 DEBUG "value": [ 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG [...] 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:21:26 DEBUG }, 05:21:26 DEBUG "value": [ 05:21:26 DEBUG 1712722886, 05:21:26 DEBUG "1259.9666666666667" 05:21:26 DEBUG ] 05:21:26 DEBUG } 05:21:26 DEBUG ] 05:21:26 DEBUG } 05:21:26 DEBUG } 05:21:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712722886" 05:21:26 INFO [http_cmd]: http status code OK 05:21:26 DEBUG --- status code --- 05:21:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:26 DEBUG --- http response --- 05:21:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:21:26 DEBUG ----- output ----- 05:21:26 DEBUG { 05:21:26 DEBUG "status": "success", 05:21:26 DEBUG "data": { 05:21:26 DEBUG "resultType": "vector", 05:21:26 DEBUG "result": [ 05:21:26 DEBUG { 05:21:26 DEBUG "metric": { 05:21:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:21:26 DEBUG }, 05:21:26 DEBUG "value": [ 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG [...] 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:21:26 DEBUG }, 05:21:26 DEBUG "value": [ 05:21:26 DEBUG 1712722886, 05:21:26 DEBUG "0.041" 05:21:26 DEBUG ] 05:21:26 DEBUG } 05:21:26 DEBUG ] 05:21:26 DEBUG } 05:21:26 DEBUG } 05:21:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722886" 05:21:26 INFO [http_cmd]: http status code OK 05:21:26 DEBUG --- status code --- 05:21:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:26 DEBUG --- http response --- 05:21:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:21:26 DEBUG ----- output ----- 05:21:26 DEBUG { 05:21:26 DEBUG "status": "success", 05:21:26 DEBUG "data": { 05:21:26 DEBUG "resultType": "vector", 05:21:26 DEBUG "result": [ 05:21:26 DEBUG { 05:21:26 DEBUG "metric": { 05:21:26 DEBUG "pod": "ds-idrepo-0" 05:21:26 DEBUG }, 05:21:26 DEBUG "value": [ 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG [...] 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG "pod": "ds-idrepo-1" 05:21:26 DEBUG }, 05:21:26 DEBUG "value": [ 05:21:26 DEBUG 1712722886, 05:21:26 DEBUG "0" 05:21:26 DEBUG ] 05:21:26 DEBUG } 05:21:26 DEBUG ] 05:21:26 DEBUG } 05:21:26 DEBUG } 05:21:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722886" 05:21:26 INFO [http_cmd]: http status code OK 05:21:26 DEBUG --- status code --- 05:21:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:26 DEBUG --- http response --- 05:21:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:21:26 DEBUG ----- output ----- 05:21:26 DEBUG { 05:21:26 DEBUG "status": "success", 05:21:26 DEBUG "data": { 05:21:26 DEBUG "resultType": "vector", 05:21:26 DEBUG "result": [ 05:21:26 DEBUG { 05:21:26 DEBUG "metric": { 05:21:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:21:26 DEBUG }, 05:21:26 DEBUG "value": [ 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG [...] 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:21:26 DEBUG }, 05:21:26 DEBUG "value": [ 05:21:26 DEBUG 1712722886, 05:21:26 DEBUG "1246.1333333333332" 05:21:26 DEBUG ] 05:21:26 DEBUG } 05:21:26 DEBUG ] 05:21:26 DEBUG } 05:21:26 DEBUG } 05:21:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722886" 05:21:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712722886" 05:21:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722886" 05:21:26 INFO [http_cmd]: http status code OK 05:21:26 DEBUG --- status code --- 05:21:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:26 DEBUG --- http response --- 05:21:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:21:26 DEBUG ----- output ----- 05:21:26 DEBUG { 05:21:26 DEBUG "status": "success", 05:21:26 DEBUG "data": { 05:21:26 DEBUG "resultType": "vector", 05:21:26 DEBUG "result": [ 05:21:26 DEBUG { 05:21:26 DEBUG "metric": { 05:21:26 DEBUG "pod": "ds-idrepo-0" 05:21:26 DEBUG }, 05:21:26 DEBUG "value": [ 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG [...] 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG "pod": "ds-idrepo-1" 05:21:26 DEBUG }, 05:21:26 DEBUG "value": [ 05:21:26 DEBUG 1712722886, 05:21:26 DEBUG "0" 05:21:26 DEBUG ] 05:21:26 DEBUG } 05:21:26 DEBUG ] 05:21:26 DEBUG } 05:21:26 DEBUG } 05:21:26 INFO [http_cmd]: http status code OK 05:21:26 DEBUG --- status code --- 05:21:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:26 DEBUG --- http response --- 05:21:26 INFO [http_cmd]: http status code OK 05:21:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:21:26 DEBUG --- status code --- 05:21:26 DEBUG ----- output ----- 05:21:26 DEBUG { 05:21:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:26 DEBUG "status": "success", 05:21:26 DEBUG --- http response --- 05:21:26 DEBUG "data": { 05:21:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 05:21:26 DEBUG "resultType": "vector", 05:21:26 DEBUG ----- output ----- 05:21:26 DEBUG "result": [ 05:21:26 DEBUG { 05:21:26 DEBUG { 05:21:26 DEBUG "status": "success", 05:21:26 DEBUG "data": { 05:21:26 DEBUG "resultType": "vector", 05:21:26 DEBUG "metric": { 05:21:26 DEBUG "result": [ 05:21:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:21:26 DEBUG { 05:21:26 DEBUG }, 05:21:26 DEBUG "metric": { 05:21:26 DEBUG "value": [ 05:21:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:21:26 DEBUG 05:21:26 DEBUG }, 05:21:26 DEBUG 05:21:26 DEBUG "value": [ 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG [...] 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG [...] 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:21:26 DEBUG 05:21:26 DEBUG }, 05:21:26 DEBUG 05:21:26 DEBUG "value": [ 05:21:26 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:21:26 DEBUG 1712722886, 05:21:26 DEBUG }, 05:21:26 DEBUG "2485587968" 05:21:26 DEBUG "value": [ 05:21:26 DEBUG ] 05:21:26 DEBUG 1712722886, 05:21:26 DEBUG } 05:21:26 DEBUG "0" 05:21:26 DEBUG ] 05:21:26 DEBUG ] 05:21:26 DEBUG } 05:21:26 DEBUG } 05:21:26 DEBUG } 05:21:26 DEBUG ] 05:21:26 DEBUG } 05:21:26 DEBUG } 05:21:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722886" 05:21:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722886" 05:21:26 INFO [http_cmd]: http status code OK 05:21:26 DEBUG --- status code --- 05:21:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:26 DEBUG --- http response --- 05:21:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:21:26 DEBUG ----- output ----- 05:21:26 DEBUG { 05:21:26 DEBUG "status": "success", 05:21:26 DEBUG "data": { 05:21:26 DEBUG "resultType": "vector", 05:21:26 DEBUG "result": [ 05:21:26 DEBUG { 05:21:26 DEBUG "metric": { 05:21:26 DEBUG "instance": "10.142.0.25:9100" 05:21:26 DEBUG }, 05:21:26 DEBUG "value": [ 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG [...] 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG "instance": "10.142.0.77:9100" 05:21:26 DEBUG }, 05:21:26 DEBUG "value": [ 05:21:26 DEBUG 1712722886, 05:21:26 DEBUG "0.05899999999999951" 05:21:26 DEBUG ] 05:21:26 DEBUG } 05:21:26 DEBUG ] 05:21:26 DEBUG } 05:21:26 DEBUG } 05:21:26 INFO [http_cmd]: http status code OK 05:21:26 DEBUG --- status code --- 05:21:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:21:26 DEBUG --- http response --- 05:21:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:21:26 DEBUG ----- output ----- 05:21:26 DEBUG { 05:21:26 DEBUG "status": "success", 05:21:26 DEBUG "data": { 05:21:26 DEBUG "resultType": "vector", 05:21:26 DEBUG "result": [ 05:21:26 DEBUG { 05:21:26 DEBUG "metric": { 05:21:26 DEBUG "instance": "10.142.0.25:9100" 05:21:26 DEBUG }, 05:21:26 DEBUG "value": [ 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG [...] 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG 05:21:26 DEBUG "instance": "10.142.0.77:9100" 05:21:26 DEBUG }, 05:21:26 DEBUG "value": [ 05:21:26 DEBUG 1712722886, 05:21:26 DEBUG "37608992.2" 05:21:26 DEBUG ] 05:21:26 DEBUG } 05:21:26 DEBUG ] 05:21:26 DEBUG } 05:21:26 DEBUG } 05:21:41 INFO 05:21:41 INFO [loop_until]: kubectl --namespace=xlou top pods 05:21:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:21:41 INFO [loop_until]: OK (rc = 0) 05:21:41 DEBUG --- stdout --- 05:21:41 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 8852m 8807Mi am-654c4fcbcf-9scrj 8513m 8421Mi am-654c4fcbcf-v5qb5 7373m 8406Mi ds-cts-0 7335m 12716Mi ds-cts-1 7262m 12836Mi ds-cts-2 6354m 12380Mi ds-idrepo-0 9779m 5621Mi ds-idrepo-1 308m 5668Mi ds-idrepo-2 329m 5698Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2212Mi lodemon-b776b7d88-thrqf 8m 70Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3526m 1656Mi 05:21:41 DEBUG --- stderr --- 05:21:41 DEBUG 05:21:47 INFO 05:21:47 INFO [loop_until]: kubectl --namespace=xlou top node 05:21:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:21:47 INFO [loop_until]: OK (rc = 0) 05:21:47 DEBUG --- stdout --- 05:21:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 95m 0% 2311Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8199m 51% 9284Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 78m 0% 3254Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10854m 68% 6505Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9601m 60% 9309Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9943m 62% 9646Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 74m 0% 851Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9767m 61% 6279Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 7329m 46% 13321Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 6740m 42% 13719Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 403m 2% 6363Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7337m 46% 13365Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 370m 2% 6327Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3708m 23% 5340Mi 9% 05:21:47 DEBUG --- stderr --- 05:21:47 DEBUG 05:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722943" 05:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712722943" 05:22:23 INFO [http_cmd]: http status code OK 05:22:23 DEBUG --- status code --- 05:22:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:23 DEBUG --- http response --- 05:22:23 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:22:23 ERROR [http_cmd]: ERROR 05:22:23 ERROR --- status code --- 05:22:23 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:22:23 ERROR --- http response --- 05:22:23 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:22:23 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722943" 05:22:23 INFO [http_cmd]: http status code OK 05:22:23 DEBUG --- status code --- 05:22:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:23 DEBUG --- http response --- 05:22:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:22:23 DEBUG ----- output ----- 05:22:23 DEBUG { 05:22:23 DEBUG "status": "success", 05:22:23 DEBUG "data": { 05:22:23 DEBUG "resultType": "vector", 05:22:23 DEBUG "result": [ 05:22:23 DEBUG { 05:22:23 DEBUG "metric": { 05:22:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:22:23 DEBUG }, 05:22:23 DEBUG "value": [ 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG [...] 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:22:23 DEBUG }, 05:22:23 DEBUG "value": [ 05:22:23 DEBUG 1712722943, 05:22:23 DEBUG "0" 05:22:23 DEBUG ] 05:22:23 DEBUG } 05:22:23 DEBUG ] 05:22:23 DEBUG } 05:22:23 DEBUG } 05:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722943" 05:22:23 INFO [http_cmd]: http status code OK 05:22:23 DEBUG --- status code --- 05:22:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:23 DEBUG --- http response --- 05:22:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:22:23 DEBUG ----- output ----- 05:22:23 DEBUG { 05:22:23 DEBUG "status": "success", 05:22:23 DEBUG "data": { 05:22:23 DEBUG "resultType": "vector", 05:22:23 DEBUG "result": [ 05:22:23 DEBUG { 05:22:23 DEBUG "metric": { 05:22:23 DEBUG "pod": "ds-idrepo-0" 05:22:23 DEBUG }, 05:22:23 DEBUG "value": [ 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG [...] 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG "pod": "ds-cts-2" 05:22:23 DEBUG }, 05:22:23 DEBUG "value": [ 05:22:23 DEBUG 1712722943, 05:22:23 DEBUG "30" 05:22:23 DEBUG ] 05:22:23 DEBUG } 05:22:23 DEBUG ] 05:22:23 DEBUG } 05:22:23 DEBUG } 05:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722943" 05:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722943" 05:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722943" 05:22:23 INFO [http_cmd]: http status code OK 05:22:23 DEBUG --- status code --- 05:22:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:23 DEBUG --- http response --- 05:22:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:22:23 DEBUG ----- output ----- 05:22:23 DEBUG { 05:22:23 DEBUG "status": "success", 05:22:23 DEBUG "data": { 05:22:23 DEBUG "resultType": "vector", 05:22:23 DEBUG "result": [ 05:22:23 DEBUG { 05:22:23 DEBUG "metric": { 05:22:23 DEBUG "instance": "10.142.0.25:9100" 05:22:23 DEBUG }, 05:22:23 DEBUG "value": [ 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG [...] 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG "instance": "10.142.0.77:9100" 05:22:23 DEBUG }, 05:22:23 DEBUG "value": [ 05:22:23 DEBUG 1712722943, 05:22:23 DEBUG "273984963584" 05:22:23 DEBUG ] 05:22:23 DEBUG } 05:22:23 DEBUG ] 05:22:23 DEBUG } 05:22:23 DEBUG } 05:22:23 INFO [http_cmd]: http status code OK 05:22:23 DEBUG --- status code --- 05:22:23 INFO [http_cmd]: http status code OK 05:22:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:23 DEBUG --- status code --- 05:22:23 DEBUG --- http response --- 05:22:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:22:23 DEBUG --- http response --- 05:22:23 DEBUG ----- output ----- 05:22:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:22:23 DEBUG { 05:22:23 DEBUG ----- output ----- 05:22:23 DEBUG "status": "success", 05:22:23 DEBUG { 05:22:23 DEBUG "data": { 05:22:23 DEBUG "status": "success", 05:22:23 DEBUG "resultType": "vector", 05:22:23 DEBUG "data": { 05:22:23 DEBUG "result": [ 05:22:23 DEBUG "resultType": "vector", 05:22:23 DEBUG { 05:22:23 DEBUG "result": [ 05:22:23 DEBUG "metric": { 05:22:23 DEBUG { 05:22:23 DEBUG "pod": "ds-idrepo-0" 05:22:23 DEBUG "metric": { 05:22:23 DEBUG }, 05:22:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:22:23 DEBUG }, 05:22:23 DEBUG "value": [ 05:22:23 DEBUG "value": [ 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG [...] 05:22:23 DEBUG [...] 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:22:23 DEBUG "pod": "ds-cts-2" 05:22:23 DEBUG }, 05:22:23 DEBUG }, 05:22:23 DEBUG "value": [ 05:22:23 DEBUG "value": [ 05:22:23 DEBUG 1712722943, 05:22:23 DEBUG 1712722943, 05:22:23 DEBUG "1249.2" 05:22:23 DEBUG "0" 05:22:23 DEBUG ] 05:22:23 DEBUG ] 05:22:23 DEBUG } 05:22:23 DEBUG } 05:22:23 DEBUG ] 05:22:23 DEBUG ] 05:22:23 DEBUG } 05:22:23 DEBUG } 05:22:23 DEBUG } 05:22:23 DEBUG } 05:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722943" 05:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722943" 05:22:23 INFO [http_cmd]: http status code OK 05:22:23 DEBUG --- status code --- 05:22:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:23 DEBUG --- http response --- 05:22:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:22:23 DEBUG ----- output ----- 05:22:23 DEBUG { 05:22:23 DEBUG "status": "success", 05:22:23 DEBUG "data": { 05:22:23 DEBUG "resultType": "vector", 05:22:23 DEBUG "result": [ 05:22:23 DEBUG { 05:22:23 DEBUG "metric": { 05:22:23 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:22:23 DEBUG }, 05:22:23 DEBUG "value": [ 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG [...] 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:22:23 DEBUG }, 05:22:23 DEBUG "value": [ 05:22:23 DEBUG 1712722943, 05:22:23 DEBUG "0" 05:22:23 DEBUG ] 05:22:23 DEBUG } 05:22:23 DEBUG ] 05:22:23 DEBUG } 05:22:23 DEBUG } 05:22:23 INFO [http_cmd]: http status code OK 05:22:23 DEBUG --- status code --- 05:22:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:23 DEBUG --- http response --- 05:22:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:22:23 DEBUG ----- output ----- 05:22:23 DEBUG { 05:22:23 DEBUG "status": "success", 05:22:23 DEBUG "data": { 05:22:23 DEBUG "resultType": "vector", 05:22:23 DEBUG "result": [ 05:22:23 DEBUG { 05:22:23 DEBUG "metric": { 05:22:23 DEBUG "instance": "10.142.0.25:9100" 05:22:23 DEBUG }, 05:22:23 DEBUG "value": [ 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG [...] 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG "instance": "10.142.0.77:9100" 05:22:23 DEBUG }, 05:22:23 DEBUG "value": [ 05:22:23 DEBUG 1712722943, 05:22:23 DEBUG "37561042.06666666" 05:22:23 DEBUG ] 05:22:23 DEBUG } 05:22:23 DEBUG ] 05:22:23 DEBUG } 05:22:23 DEBUG } 05:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722943" 05:22:23 INFO [http_cmd]: http status code OK 05:22:23 DEBUG --- status code --- 05:22:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:23 DEBUG --- http response --- 05:22:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:22:23 DEBUG ----- output ----- 05:22:23 DEBUG { 05:22:23 DEBUG "status": "success", 05:22:23 DEBUG "data": { 05:22:23 DEBUG "resultType": "vector", 05:22:23 DEBUG "result": [ 05:22:23 DEBUG { 05:22:23 DEBUG "metric": { 05:22:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:22:23 DEBUG }, 05:22:23 DEBUG "value": [ 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG [...] 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG 05:22:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:22:23 DEBUG }, 05:22:23 DEBUG "value": [ 05:22:23 DEBUG 1712722943, 05:22:23 DEBUG "227185.5" 05:22:23 DEBUG ] 05:22:23 DEBUG } 05:22:23 DEBUG ] 05:22:23 DEBUG } 05:22:23 DEBUG } 05:22:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722944" 05:22:24 INFO [http_cmd]: http status code OK 05:22:24 DEBUG --- status code --- 05:22:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:24 DEBUG --- http response --- 05:22:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:22:24 DEBUG ----- output ----- 05:22:24 DEBUG { 05:22:24 DEBUG "status": "success", 05:22:24 DEBUG "data": { 05:22:24 DEBUG "resultType": "vector", 05:22:24 DEBUG "result": [ 05:22:24 DEBUG { 05:22:24 DEBUG "metric": { 05:22:24 DEBUG "pod": "ds-idrepo-0" 05:22:24 DEBUG }, 05:22:24 DEBUG "value": [ 05:22:24 DEBUG 05:22:24 DEBUG 05:22:24 DEBUG 05:22:24 DEBUG [...] 05:22:24 DEBUG 05:22:24 DEBUG 05:22:24 DEBUG 05:22:24 DEBUG "pod": "ds-idrepo-1" 05:22:24 DEBUG }, 05:22:24 DEBUG "value": [ 05:22:24 DEBUG 1712722944, 05:22:24 DEBUG "0" 05:22:24 DEBUG ] 05:22:24 DEBUG } 05:22:24 DEBUG ] 05:22:24 DEBUG } 05:22:24 DEBUG } 05:22:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722944" 05:22:24 INFO [http_cmd]: http status code OK 05:22:24 DEBUG --- status code --- 05:22:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:24 DEBUG --- http response --- 05:22:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:22:24 DEBUG ----- output ----- 05:22:24 DEBUG { 05:22:24 DEBUG "status": "success", 05:22:24 DEBUG "data": { 05:22:24 DEBUG "resultType": "vector", 05:22:24 DEBUG "result": [ 05:22:24 DEBUG { 05:22:24 DEBUG "metric": { 05:22:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:22:24 DEBUG }, 05:22:24 DEBUG "value": [ 05:22:24 DEBUG 05:22:24 DEBUG 05:22:24 DEBUG 05:22:24 DEBUG [...] 05:22:24 DEBUG 05:22:24 DEBUG 05:22:24 DEBUG 05:22:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:22:24 DEBUG }, 05:22:24 DEBUG "value": [ 05:22:24 DEBUG 1712722944, 05:22:24 DEBUG "1249.3666666666666" 05:22:24 DEBUG ] 05:22:24 DEBUG } 05:22:24 DEBUG ] 05:22:24 DEBUG } 05:22:24 DEBUG } 05:22:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722944" 05:22:24 INFO [http_cmd]: http status code OK 05:22:24 DEBUG --- status code --- 05:22:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:24 DEBUG --- http response --- 05:22:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:22:24 DEBUG ----- output ----- 05:22:24 DEBUG { 05:22:24 DEBUG "status": "success", 05:22:24 DEBUG "data": { 05:22:24 DEBUG "resultType": "vector", 05:22:24 DEBUG "result": [ 05:22:24 DEBUG { 05:22:24 DEBUG "metric": { 05:22:24 DEBUG "instance": "10.142.0.25:9100" 05:22:24 DEBUG }, 05:22:24 DEBUG "value": [ 05:22:24 DEBUG 05:22:24 DEBUG 05:22:24 DEBUG 05:22:24 DEBUG [...] 05:22:24 DEBUG 05:22:24 DEBUG 05:22:24 DEBUG 05:22:24 DEBUG "instance": "10.142.0.77:9100" 05:22:24 DEBUG }, 05:22:24 DEBUG "value": [ 05:22:24 DEBUG 1712722944, 05:22:24 DEBUG "0.15616666666666626" 05:22:24 DEBUG ] 05:22:24 DEBUG } 05:22:24 DEBUG ] 05:22:24 DEBUG } 05:22:24 DEBUG } 05:22:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722945" 05:22:25 INFO [http_cmd]: http status code OK 05:22:25 DEBUG --- status code --- 05:22:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:25 DEBUG --- http response --- 05:22:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:22:25 DEBUG ----- output ----- 05:22:25 DEBUG { 05:22:25 DEBUG "status": "success", 05:22:25 DEBUG "data": { 05:22:25 DEBUG "resultType": "vector", 05:22:25 DEBUG "result": [ 05:22:25 DEBUG { 05:22:25 DEBUG "metric": { 05:22:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:22:25 DEBUG }, 05:22:25 DEBUG "value": [ 05:22:25 DEBUG 05:22:25 DEBUG 05:22:25 DEBUG 05:22:25 DEBUG [...] 05:22:25 DEBUG 05:22:25 DEBUG 05:22:25 DEBUG 05:22:25 DEBUG "pod": "ds-idrepo-2" 05:22:25 DEBUG }, 05:22:25 DEBUG "value": [ 05:22:25 DEBUG 1712722945, 05:22:25 DEBUG "0" 05:22:25 DEBUG ] 05:22:25 DEBUG } 05:22:25 DEBUG ] 05:22:25 DEBUG } 05:22:25 DEBUG } 05:22:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712722945" 05:22:25 INFO [http_cmd]: http status code OK 05:22:25 DEBUG --- status code --- 05:22:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:25 DEBUG --- http response --- 05:22:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:22:25 DEBUG ----- output ----- 05:22:25 DEBUG { 05:22:25 DEBUG "status": "success", 05:22:25 DEBUG "data": { 05:22:25 DEBUG "resultType": "vector", 05:22:25 DEBUG "result": [ 05:22:25 DEBUG { 05:22:25 DEBUG "metric": { 05:22:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:22:25 DEBUG }, 05:22:25 DEBUG "value": [ 05:22:25 DEBUG 05:22:25 DEBUG 05:22:25 DEBUG 05:22:25 DEBUG [...] 05:22:25 DEBUG 05:22:25 DEBUG 05:22:25 DEBUG 05:22:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:22:25 DEBUG }, 05:22:25 DEBUG "value": [ 05:22:25 DEBUG 1712722945, 05:22:25 DEBUG "3.595961763492956" 05:22:25 DEBUG ] 05:22:25 DEBUG } 05:22:25 DEBUG ] 05:22:25 DEBUG } 05:22:25 DEBUG } 05:22:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712722945" 05:22:25 INFO [http_cmd]: http status code OK 05:22:25 DEBUG --- status code --- 05:22:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:25 DEBUG --- http response --- 05:22:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:22:25 DEBUG ----- output ----- 05:22:25 DEBUG { 05:22:25 DEBUG "status": "success", 05:22:25 DEBUG "data": { 05:22:25 DEBUG "resultType": "vector", 05:22:25 DEBUG "result": [ 05:22:25 DEBUG { 05:22:25 DEBUG "metric": { 05:22:25 DEBUG "pod": "ds-idrepo-0" 05:22:25 DEBUG }, 05:22:25 DEBUG "value": [ 05:22:25 DEBUG 05:22:25 DEBUG 05:22:25 DEBUG 05:22:25 DEBUG [...] 05:22:25 DEBUG 05:22:25 DEBUG 05:22:25 DEBUG 05:22:25 DEBUG "pod": "ds-cts-2" 05:22:25 DEBUG }, 05:22:25 DEBUG "value": [ 05:22:25 DEBUG 1712722945, 05:22:25 DEBUG "0" 05:22:25 DEBUG ] 05:22:25 DEBUG } 05:22:25 DEBUG ] 05:22:25 DEBUG } 05:22:25 DEBUG } 05:22:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722945" 05:22:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722945" 05:22:25 INFO [http_cmd]: http status code OK 05:22:25 INFO [http_cmd]: http status code OK 05:22:25 DEBUG --- status code --- 05:22:25 DEBUG --- status code --- 05:22:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:25 DEBUG --- http response --- 05:22:25 DEBUG --- http response --- 05:22:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:22:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:22:25 DEBUG ----- output ----- 05:22:25 DEBUG ----- output ----- 05:22:25 DEBUG { 05:22:25 DEBUG { 05:22:25 DEBUG "status": "success", 05:22:25 DEBUG "status": "success", 05:22:25 DEBUG "data": { 05:22:25 DEBUG "data": { 05:22:25 DEBUG "resultType": "vector", 05:22:25 DEBUG "resultType": "vector", 05:22:25 DEBUG "result": [ 05:22:25 DEBUG "result": [ 05:22:25 DEBUG { 05:22:25 DEBUG { 05:22:25 DEBUG "metric": { 05:22:25 DEBUG "metric": { 05:22:25 DEBUG "pod": "ds-idrepo-2" 05:22:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:22:25 DEBUG }, 05:22:25 DEBUG }, 05:22:25 DEBUG "value": [ 05:22:25 DEBUG "value": [ 05:22:25 DEBUG 05:22:25 DEBUG 05:22:25 DEBUG 05:22:25 DEBUG 05:22:25 DEBUG 05:22:25 DEBUG 05:22:25 DEBUG [...] 05:22:25 DEBUG [...] 05:22:25 DEBUG 05:22:25 DEBUG 05:22:25 DEBUG 05:22:25 DEBUG 05:22:25 DEBUG 05:22:25 DEBUG 05:22:25 DEBUG "pod": "ds-cts-2" 05:22:25 DEBUG "pod": "ds-idrepo-2" 05:22:25 DEBUG }, 05:22:25 DEBUG }, 05:22:25 DEBUG "value": [ 05:22:25 DEBUG "value": [ 05:22:25 DEBUG 1712722945, 05:22:25 DEBUG 1712722945, 05:22:25 DEBUG "0.000056882222222222234" 05:22:25 DEBUG "0" 05:22:25 DEBUG ] 05:22:25 DEBUG ] 05:22:25 DEBUG } 05:22:25 DEBUG } 05:22:25 DEBUG ] 05:22:25 DEBUG ] 05:22:25 DEBUG } 05:22:25 DEBUG } 05:22:25 DEBUG } 05:22:25 DEBUG } 05:22:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722946" 05:22:26 INFO [http_cmd]: http status code OK 05:22:26 DEBUG --- status code --- 05:22:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:26 DEBUG --- http response --- 05:22:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:22:26 DEBUG ----- output ----- 05:22:26 DEBUG { 05:22:26 DEBUG "status": "success", 05:22:26 DEBUG "data": { 05:22:26 DEBUG "resultType": "vector", 05:22:26 DEBUG "result": [ 05:22:26 DEBUG { 05:22:26 DEBUG "metric": { 05:22:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:22:26 DEBUG }, 05:22:26 DEBUG "value": [ 05:22:26 DEBUG 05:22:26 DEBUG 05:22:26 DEBUG 05:22:26 DEBUG [...] 05:22:26 DEBUG 05:22:26 DEBUG 05:22:26 DEBUG 05:22:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:22:26 DEBUG }, 05:22:26 DEBUG "value": [ 05:22:26 DEBUG 1712722946, 05:22:26 DEBUG "1249.3333333333333" 05:22:26 DEBUG ] 05:22:26 DEBUG } 05:22:26 DEBUG ] 05:22:26 DEBUG } 05:22:26 DEBUG } 05:22:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712722946" 05:22:26 INFO [http_cmd]: http status code OK 05:22:26 DEBUG --- status code --- 05:22:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:26 DEBUG --- http response --- 05:22:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:22:26 DEBUG ----- output ----- 05:22:26 DEBUG { 05:22:26 DEBUG "status": "success", 05:22:26 DEBUG "data": { 05:22:26 DEBUG "resultType": "vector", 05:22:26 DEBUG "result": [ 05:22:26 DEBUG { 05:22:26 DEBUG "metric": { 05:22:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:22:26 DEBUG }, 05:22:26 DEBUG "value": [ 05:22:26 DEBUG 05:22:26 DEBUG 05:22:26 DEBUG 05:22:26 DEBUG [...] 05:22:26 DEBUG 05:22:26 DEBUG 05:22:26 DEBUG 05:22:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:22:26 DEBUG }, 05:22:26 DEBUG "value": [ 05:22:26 DEBUG 1712722946, 05:22:26 DEBUG "0.041" 05:22:26 DEBUG ] 05:22:26 DEBUG } 05:22:26 DEBUG ] 05:22:26 DEBUG } 05:22:26 DEBUG } 05:22:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722946" 05:22:26 INFO [http_cmd]: http status code OK 05:22:26 DEBUG --- status code --- 05:22:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:26 DEBUG --- http response --- 05:22:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:22:26 DEBUG ----- output ----- 05:22:26 DEBUG { 05:22:26 DEBUG "status": "success", 05:22:26 DEBUG "data": { 05:22:26 DEBUG "resultType": "vector", 05:22:26 DEBUG "result": [ 05:22:26 DEBUG { 05:22:26 DEBUG "metric": { 05:22:26 DEBUG "pod": "ds-idrepo-0" 05:22:26 DEBUG }, 05:22:26 DEBUG "value": [ 05:22:26 DEBUG 05:22:26 DEBUG 05:22:26 DEBUG 05:22:26 DEBUG [...] 05:22:26 DEBUG 05:22:26 DEBUG 05:22:26 DEBUG 05:22:26 DEBUG "pod": "ds-idrepo-1" 05:22:26 DEBUG }, 05:22:26 DEBUG "value": [ 05:22:26 DEBUG 1712722946, 05:22:26 DEBUG "0" 05:22:26 DEBUG ] 05:22:26 DEBUG } 05:22:26 DEBUG ] 05:22:26 DEBUG } 05:22:26 DEBUG } 05:22:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722946" 05:22:26 INFO [http_cmd]: http status code OK 05:22:26 DEBUG --- status code --- 05:22:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:26 DEBUG --- http response --- 05:22:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:22:26 DEBUG ----- output ----- 05:22:26 DEBUG { 05:22:26 DEBUG "status": "success", 05:22:26 DEBUG "data": { 05:22:26 DEBUG "resultType": "vector", 05:22:26 DEBUG "result": [ 05:22:26 DEBUG { 05:22:26 DEBUG "metric": { 05:22:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:22:26 DEBUG }, 05:22:26 DEBUG "value": [ 05:22:26 DEBUG 05:22:26 DEBUG 05:22:26 DEBUG 05:22:26 DEBUG [...] 05:22:26 DEBUG 05:22:26 DEBUG 05:22:26 DEBUG 05:22:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:22:26 DEBUG }, 05:22:26 DEBUG "value": [ 05:22:26 DEBUG 1712722946, 05:22:26 DEBUG "1252.4666666666667" 05:22:26 DEBUG ] 05:22:26 DEBUG } 05:22:26 DEBUG ] 05:22:26 DEBUG } 05:22:26 DEBUG } 05:22:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722946" 05:22:26 INFO [http_cmd]: http status code OK 05:22:26 DEBUG --- status code --- 05:22:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:26 DEBUG --- http response --- 05:22:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:22:26 DEBUG ----- output ----- 05:22:26 DEBUG { 05:22:26 DEBUG "status": "success", 05:22:26 DEBUG "data": { 05:22:26 DEBUG "resultType": "vector", 05:22:26 DEBUG "result": [ 05:22:26 DEBUG { 05:22:26 DEBUG "metric": { 05:22:26 DEBUG "pod": "ds-idrepo-0" 05:22:26 DEBUG }, 05:22:26 DEBUG "value": [ 05:22:26 DEBUG 05:22:26 DEBUG 05:22:26 DEBUG 05:22:26 DEBUG [...] 05:22:26 DEBUG 05:22:26 DEBUG 05:22:26 DEBUG 05:22:26 DEBUG "pod": "ds-idrepo-1" 05:22:26 DEBUG }, 05:22:26 DEBUG "value": [ 05:22:26 DEBUG 1712722946, 05:22:26 DEBUG "0" 05:22:26 DEBUG ] 05:22:26 DEBUG } 05:22:26 DEBUG ] 05:22:26 DEBUG } 05:22:26 DEBUG } 05:22:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712722946" 05:22:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712722946" 05:22:26 INFO [http_cmd]: http status code OK 05:22:26 INFO [http_cmd]: http status code OK 05:22:26 DEBUG --- status code --- 05:22:26 DEBUG --- status code --- 05:22:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:26 DEBUG --- http response --- 05:22:26 DEBUG --- http response --- 05:22:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 05:22:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:22:26 DEBUG ----- output ----- 05:22:26 DEBUG { 05:22:26 DEBUG "status": "success", 05:22:26 DEBUG "data": { 05:22:26 DEBUG "resultType": "vector", 05:22:26 DEBUG "result": [ 05:22:26 DEBUG { 05:22:26 DEBUG "metric": { 05:22:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:22:26 DEBUG }, 05:22:26 DEBUG "value": [ 05:22:26 DEBUG 05:22:26 DEBUG 05:22:26 DEBUG 05:22:26 DEBUG [...] 05:22:26 DEBUG 05:22:26 DEBUG 05:22:26 DEBUG 05:22:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:22:26 DEBUG }, 05:22:26 DEBUG "value": [ 05:22:26 DEBUG 1712722946, 05:22:26 DEBUG "2487889920" 05:22:26 DEBUG ] 05:22:26 DEBUG } 05:22:26 DEBUG ] 05:22:26 DEBUG } 05:22:26 DEBUG } 05:22:26 DEBUG ----- output ----- 05:22:26 DEBUG { 05:22:26 DEBUG "status": "success", 05:22:26 DEBUG "data": { 05:22:26 DEBUG "resultType": "vector", 05:22:26 DEBUG "result": [ 05:22:26 DEBUG { 05:22:26 DEBUG "metric": { 05:22:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:22:26 DEBUG }, 05:22:26 DEBUG "value": [ 05:22:26 DEBUG 05:22:26 DEBUG 05:22:26 DEBUG 05:22:26 DEBUG [...] 05:22:26 DEBUG 05:22:26 DEBUG 05:22:26 DEBUG 05:22:26 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:22:26 DEBUG }, 05:22:26 DEBUG "value": [ 05:22:26 DEBUG 1712722946, 05:22:26 DEBUG "0" 05:22:26 DEBUG ] 05:22:26 DEBUG } 05:22:26 DEBUG ] 05:22:26 DEBUG } 05:22:26 DEBUG } 05:22:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722947" 05:22:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712722947" 05:22:27 INFO [http_cmd]: http status code OK 05:22:27 DEBUG --- status code --- 05:22:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:27 DEBUG --- http response --- 05:22:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:22:27 DEBUG ----- output ----- 05:22:27 INFO [http_cmd]: http status code OK 05:22:27 DEBUG { 05:22:27 DEBUG --- status code --- 05:22:27 DEBUG "status": "success", 05:22:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:22:27 DEBUG "data": { 05:22:27 DEBUG --- http response --- 05:22:27 DEBUG "resultType": "vector", 05:22:27 DEBUG "result": [ 05:22:27 DEBUG { 05:22:27 DEBUG "metric": { 05:22:27 DEBUG "instance": "10.142.0.25:9100" 05:22:27 DEBUG }, 05:22:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:22:27 DEBUG "value": [ 05:22:27 DEBUG ----- output ----- 05:22:27 DEBUG 05:22:27 DEBUG { 05:22:27 DEBUG 05:22:27 DEBUG "status": "success", 05:22:27 DEBUG 05:22:27 DEBUG "data": { 05:22:27 DEBUG [...] 05:22:27 DEBUG "resultType": "vector", 05:22:27 DEBUG 05:22:27 DEBUG "result": [ 05:22:27 DEBUG 05:22:27 DEBUG { 05:22:27 DEBUG 05:22:27 DEBUG "metric": { 05:22:27 DEBUG "instance": "10.142.0.77:9100" 05:22:27 DEBUG "instance": "10.142.0.25:9100" 05:22:27 DEBUG }, 05:22:27 DEBUG }, 05:22:27 DEBUG "value": [ 05:22:27 DEBUG "value": [ 05:22:27 DEBUG 1712722947, 05:22:27 DEBUG 05:22:27 DEBUG "0.05799999999999959" 05:22:27 DEBUG 05:22:27 DEBUG ] 05:22:27 DEBUG 05:22:27 DEBUG } 05:22:27 DEBUG [...] 05:22:27 DEBUG ] 05:22:27 DEBUG 05:22:27 DEBUG } 05:22:27 DEBUG 05:22:27 DEBUG } 05:22:27 DEBUG 05:22:27 DEBUG "instance": "10.142.0.77:9100" 05:22:27 DEBUG }, 05:22:27 DEBUG "value": [ 05:22:27 DEBUG 1712722947, 05:22:27 DEBUG "37560307.43333334" 05:22:27 DEBUG ] 05:22:27 DEBUG } 05:22:27 DEBUG ] 05:22:27 DEBUG } 05:22:27 DEBUG } 05:22:41 INFO 05:22:41 INFO [loop_until]: kubectl --namespace=xlou top pods 05:22:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:22:41 INFO [loop_until]: OK (rc = 0) 05:22:41 DEBUG --- stdout --- 05:22:41 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 9114m 8817Mi am-654c4fcbcf-9scrj 8698m 8424Mi am-654c4fcbcf-v5qb5 7371m 8404Mi ds-cts-0 7277m 12389Mi ds-cts-1 7541m 13154Mi ds-cts-2 6767m 12498Mi ds-idrepo-0 9435m 5621Mi ds-idrepo-1 315m 5668Mi ds-idrepo-2 339m 5698Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 4m 2212Mi lodemon-b776b7d88-thrqf 7m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3533m 1657Mi 05:22:41 DEBUG --- stderr --- 05:22:41 DEBUG 05:22:47 INFO 05:22:47 INFO [loop_until]: kubectl --namespace=xlou top node 05:22:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:22:47 INFO [loop_until]: OK (rc = 0) 05:22:47 DEBUG --- stdout --- 05:22:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 93m 0% 2307Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8175m 51% 9280Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 83m 0% 3253Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10789m 67% 6501Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9570m 60% 9307Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9897m 62% 9646Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 77m 0% 855Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9729m 61% 6280Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 7172m 45% 13934Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 6764m 42% 13197Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 374m 2% 6364Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7433m 46% 13505Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 373m 2% 6330Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3682m 23% 5342Mi 9% 05:22:47 DEBUG --- stderr --- 05:22:47 DEBUG 05:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712723003" 05:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723003" 05:23:23 ERROR [http_cmd]: ERROR 05:23:23 ERROR --- status code --- 05:23:23 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:23:23 ERROR --- http response --- 05:23:23 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:23:23 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:23:23 INFO [http_cmd]: http status code OK 05:23:23 DEBUG --- status code --- 05:23:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:23 DEBUG --- http response --- 05:23:23 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723003" 05:23:23 INFO [http_cmd]: http status code OK 05:23:23 DEBUG --- status code --- 05:23:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:23 DEBUG --- http response --- 05:23:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:23:23 DEBUG ----- output ----- 05:23:23 DEBUG { 05:23:23 DEBUG "status": "success", 05:23:23 DEBUG "data": { 05:23:23 DEBUG "resultType": "vector", 05:23:23 DEBUG "result": [ 05:23:23 DEBUG { 05:23:23 DEBUG "metric": { 05:23:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:23:23 DEBUG }, 05:23:23 DEBUG "value": [ 05:23:23 DEBUG 05:23:23 DEBUG 05:23:23 DEBUG 05:23:23 DEBUG [...] 05:23:23 DEBUG 05:23:23 DEBUG 05:23:23 DEBUG 05:23:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:23:23 DEBUG }, 05:23:23 DEBUG "value": [ 05:23:23 DEBUG 1712723003, 05:23:23 DEBUG "0" 05:23:23 DEBUG ] 05:23:23 DEBUG } 05:23:23 DEBUG ] 05:23:23 DEBUG } 05:23:23 DEBUG } 05:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723003" 05:23:23 INFO [http_cmd]: http status code OK 05:23:23 DEBUG --- status code --- 05:23:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:23 DEBUG --- http response --- 05:23:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:23:23 DEBUG ----- output ----- 05:23:23 DEBUG { 05:23:23 DEBUG "status": "success", 05:23:23 DEBUG "data": { 05:23:23 DEBUG "resultType": "vector", 05:23:23 DEBUG "result": [ 05:23:23 DEBUG { 05:23:23 DEBUG "metric": { 05:23:23 DEBUG "pod": "ds-idrepo-0" 05:23:23 DEBUG }, 05:23:23 DEBUG "value": [ 05:23:23 DEBUG 05:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723003" 05:23:23 DEBUG 05:23:23 DEBUG 05:23:23 DEBUG [...] 05:23:23 DEBUG 05:23:23 DEBUG 05:23:23 DEBUG 05:23:23 DEBUG "pod": "ds-cts-2" 05:23:23 DEBUG }, 05:23:23 DEBUG "value": [ 05:23:23 DEBUG 1712723003, 05:23:23 DEBUG "29.666666666666668" 05:23:23 DEBUG ] 05:23:23 DEBUG } 05:23:23 DEBUG ] 05:23:23 DEBUG } 05:23:23 DEBUG } 05:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723003" 05:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723003" 05:23:23 INFO [http_cmd]: http status code OK 05:23:23 DEBUG --- status code --- 05:23:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:23 DEBUG --- http response --- 05:23:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:23:23 DEBUG ----- output ----- 05:23:23 DEBUG { 05:23:23 DEBUG "status": "success", 05:23:23 DEBUG "data": { 05:23:23 DEBUG "resultType": "vector", 05:23:23 DEBUG "result": [ 05:23:23 DEBUG { 05:23:23 DEBUG "metric": { 05:23:23 DEBUG "instance": "10.142.0.25:9100" 05:23:23 DEBUG }, 05:23:23 DEBUG "value": [ 05:23:23 DEBUG 05:23:23 DEBUG 05:23:23 DEBUG 05:23:23 DEBUG [...] 05:23:23 DEBUG 05:23:23 DEBUG 05:23:23 DEBUG 05:23:23 DEBUG "instance": "10.142.0.77:9100" 05:23:23 DEBUG }, 05:23:23 DEBUG "value": [ 05:23:23 DEBUG 1712723003, 05:23:23 DEBUG "275823848448" 05:23:23 DEBUG ] 05:23:23 DEBUG } 05:23:23 DEBUG ] 05:23:23 DEBUG } 05:23:23 DEBUG } 05:23:23 INFO [http_cmd]: http status code OK 05:23:23 INFO [http_cmd]: http status code OK 05:23:23 DEBUG --- status code --- 05:23:23 DEBUG --- status code --- 05:23:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:23 DEBUG --- http response --- 05:23:23 DEBUG --- http response --- 05:23:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:23:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:23:23 DEBUG ----- output ----- 05:23:23 DEBUG ----- output ----- 05:23:23 DEBUG { 05:23:23 DEBUG { 05:23:23 DEBUG "status": "success", 05:23:23 DEBUG "status": "success", 05:23:23 DEBUG "data": { 05:23:23 DEBUG "data": { 05:23:23 DEBUG "resultType": "vector", 05:23:23 DEBUG "resultType": "vector", 05:23:23 DEBUG "result": [ 05:23:23 DEBUG "result": [ 05:23:23 DEBUG { 05:23:23 DEBUG { 05:23:23 DEBUG "metric": { 05:23:23 DEBUG "metric": { 05:23:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:23:23 DEBUG "pod": "ds-idrepo-0" 05:23:23 DEBUG }, 05:23:23 DEBUG }, 05:23:23 DEBUG "value": [ 05:23:23 DEBUG "value": [ 05:23:23 DEBUG 05:23:23 DEBUG 05:23:23 DEBUG 05:23:23 DEBUG 05:23:23 DEBUG [...] 05:23:23 DEBUG 05:23:23 DEBUG 05:23:23 DEBUG 05:23:23 DEBUG "pod": "ds-cts-2" 05:23:23 DEBUG }, 05:23:23 DEBUG "value": [ 05:23:23 DEBUG 1712723003, 05:23:23 DEBUG "0" 05:23:23 DEBUG ] 05:23:23 DEBUG } 05:23:23 DEBUG ] 05:23:23 DEBUG } 05:23:23 DEBUG } 05:23:23 DEBUG 05:23:23 DEBUG 05:23:23 DEBUG [...] 05:23:23 DEBUG 05:23:23 DEBUG 05:23:23 DEBUG 05:23:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:23:23 DEBUG }, 05:23:23 DEBUG "value": [ 05:23:23 DEBUG 1712723003, 05:23:23 DEBUG "1261.2333333333333" 05:23:23 DEBUG ] 05:23:23 DEBUG } 05:23:23 DEBUG ] 05:23:23 DEBUG } 05:23:23 DEBUG } 05:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723003" 05:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723003" 05:23:23 INFO [http_cmd]: http status code OK 05:23:23 DEBUG --- status code --- 05:23:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:23 DEBUG --- http response --- 05:23:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:23:23 DEBUG ----- output ----- 05:23:23 DEBUG { 05:23:23 DEBUG "status": "success", 05:23:23 DEBUG "data": { 05:23:23 DEBUG "resultType": "vector", 05:23:23 DEBUG "result": [ 05:23:23 DEBUG { 05:23:23 DEBUG "metric": { 05:23:23 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:23:23 DEBUG }, 05:23:23 DEBUG "value": [ 05:23:23 DEBUG 05:23:23 DEBUG 05:23:23 DEBUG 05:23:23 DEBUG [...] 05:23:23 DEBUG 05:23:23 DEBUG 05:23:23 DEBUG 05:23:23 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:23:23 DEBUG }, 05:23:23 DEBUG "value": [ 05:23:23 DEBUG 1712723003, 05:23:23 INFO [http_cmd]: http status code OK 05:23:23 DEBUG "0" 05:23:23 DEBUG --- status code --- 05:23:23 DEBUG ] 05:23:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:23 DEBUG } 05:23:23 DEBUG --- http response --- 05:23:23 DEBUG ] 05:23:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:23:23 DEBUG } 05:23:23 DEBUG ----- output ----- 05:23:23 DEBUG } 05:23:23 DEBUG { 05:23:23 DEBUG "status": "success", 05:23:23 DEBUG "data": { 05:23:23 DEBUG "resultType": "vector", 05:23:23 DEBUG "result": [ 05:23:23 DEBUG { 05:23:23 DEBUG "metric": { 05:23:23 DEBUG "instance": "10.142.0.25:9100" 05:23:23 DEBUG }, 05:23:23 DEBUG "value": [ 05:23:23 DEBUG 05:23:23 DEBUG 05:23:23 DEBUG 05:23:23 DEBUG [...] 05:23:23 DEBUG 05:23:23 DEBUG 05:23:23 DEBUG 05:23:23 DEBUG "instance": "10.142.0.77:9100" 05:23:23 DEBUG }, 05:23:23 DEBUG "value": [ 05:23:23 DEBUG 1712723003, 05:23:23 DEBUG "37774186.03333333" 05:23:23 DEBUG ] 05:23:23 DEBUG } 05:23:23 DEBUG ] 05:23:23 DEBUG } 05:23:23 DEBUG } 05:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723003" 05:23:23 INFO [http_cmd]: http status code OK 05:23:23 DEBUG --- status code --- 05:23:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:23 DEBUG --- http response --- 05:23:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:23:23 DEBUG ----- output ----- 05:23:23 DEBUG { 05:23:23 DEBUG "status": "success", 05:23:23 DEBUG "data": { 05:23:23 DEBUG "resultType": "vector", 05:23:23 DEBUG "result": [ 05:23:23 DEBUG { 05:23:23 DEBUG "metric": { 05:23:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:23:23 DEBUG }, 05:23:23 DEBUG "value": [ 05:23:23 DEBUG 05:23:23 DEBUG 05:23:23 DEBUG 05:23:23 DEBUG [...] 05:23:23 DEBUG 05:23:23 DEBUG 05:23:23 DEBUG 05:23:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:23:23 DEBUG }, 05:23:23 DEBUG "value": [ 05:23:23 DEBUG 1712723003, 05:23:23 DEBUG "226933" 05:23:23 DEBUG ] 05:23:23 DEBUG } 05:23:23 DEBUG ] 05:23:23 DEBUG } 05:23:23 DEBUG } 05:23:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723004" 05:23:24 INFO [http_cmd]: http status code OK 05:23:24 DEBUG --- status code --- 05:23:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:24 DEBUG --- http response --- 05:23:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:23:24 DEBUG ----- output ----- 05:23:24 DEBUG { 05:23:24 DEBUG "status": "success", 05:23:24 DEBUG "data": { 05:23:24 DEBUG "resultType": "vector", 05:23:24 DEBUG "result": [ 05:23:24 DEBUG { 05:23:24 DEBUG "metric": { 05:23:24 DEBUG "pod": "ds-idrepo-0" 05:23:24 DEBUG }, 05:23:24 DEBUG "value": [ 05:23:24 DEBUG 05:23:24 DEBUG 05:23:24 DEBUG 05:23:24 DEBUG [...] 05:23:24 DEBUG 05:23:24 DEBUG 05:23:24 DEBUG 05:23:24 DEBUG "pod": "ds-idrepo-1" 05:23:24 DEBUG }, 05:23:24 DEBUG "value": [ 05:23:24 DEBUG 1712723004, 05:23:24 DEBUG "0" 05:23:24 DEBUG ] 05:23:24 DEBUG } 05:23:24 DEBUG ] 05:23:24 DEBUG } 05:23:24 DEBUG } 05:23:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723004" 05:23:24 INFO [http_cmd]: http status code OK 05:23:24 DEBUG --- status code --- 05:23:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:24 DEBUG --- http response --- 05:23:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:23:24 DEBUG ----- output ----- 05:23:24 DEBUG { 05:23:24 DEBUG "status": "success", 05:23:24 DEBUG "data": { 05:23:24 DEBUG "resultType": "vector", 05:23:24 DEBUG "result": [ 05:23:24 DEBUG { 05:23:24 DEBUG "metric": { 05:23:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:23:24 DEBUG }, 05:23:24 DEBUG "value": [ 05:23:24 DEBUG 05:23:24 DEBUG 05:23:24 DEBUG 05:23:24 DEBUG [...] 05:23:24 DEBUG 05:23:24 DEBUG 05:23:24 DEBUG 05:23:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:23:24 DEBUG }, 05:23:24 DEBUG "value": [ 05:23:24 DEBUG 1712723004, 05:23:24 DEBUG "1261.4" 05:23:24 DEBUG ] 05:23:24 DEBUG } 05:23:24 DEBUG ] 05:23:24 DEBUG } 05:23:24 DEBUG } 05:23:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723004" 05:23:24 INFO [http_cmd]: http status code OK 05:23:24 DEBUG --- status code --- 05:23:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:24 DEBUG --- http response --- 05:23:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:23:24 DEBUG ----- output ----- 05:23:24 DEBUG { 05:23:24 DEBUG "status": "success", 05:23:24 DEBUG "data": { 05:23:24 DEBUG "resultType": "vector", 05:23:24 DEBUG "result": [ 05:23:24 DEBUG { 05:23:24 DEBUG "metric": { 05:23:24 DEBUG "instance": "10.142.0.25:9100" 05:23:24 DEBUG }, 05:23:24 DEBUG "value": [ 05:23:24 DEBUG 05:23:24 DEBUG 05:23:24 DEBUG 05:23:24 DEBUG [...] 05:23:24 DEBUG 05:23:24 DEBUG 05:23:24 DEBUG 05:23:24 DEBUG "instance": "10.142.0.77:9100" 05:23:24 DEBUG }, 05:23:24 DEBUG "value": [ 05:23:24 DEBUG 1712723004, 05:23:24 DEBUG "0.08146666666666864" 05:23:24 DEBUG ] 05:23:24 DEBUG } 05:23:24 DEBUG ] 05:23:24 DEBUG } 05:23:24 DEBUG } 05:23:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723005" 05:23:25 INFO [http_cmd]: http status code OK 05:23:25 DEBUG --- status code --- 05:23:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:25 DEBUG --- http response --- 05:23:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:23:25 DEBUG ----- output ----- 05:23:25 DEBUG { 05:23:25 DEBUG "status": "success", 05:23:25 DEBUG "data": { 05:23:25 DEBUG "resultType": "vector", 05:23:25 DEBUG "result": [ 05:23:25 DEBUG { 05:23:25 DEBUG "metric": { 05:23:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:23:25 DEBUG }, 05:23:25 DEBUG "value": [ 05:23:25 DEBUG 05:23:25 DEBUG 05:23:25 DEBUG 05:23:25 DEBUG [...] 05:23:25 DEBUG 05:23:25 DEBUG 05:23:25 DEBUG 05:23:25 DEBUG "pod": "ds-idrepo-2" 05:23:25 DEBUG }, 05:23:25 DEBUG "value": [ 05:23:25 DEBUG 1712723005, 05:23:25 DEBUG "3650.038619880099" 05:23:25 DEBUG ] 05:23:25 DEBUG } 05:23:25 DEBUG ] 05:23:25 DEBUG } 05:23:25 DEBUG } 05:23:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712723005" 05:23:25 INFO [http_cmd]: http status code OK 05:23:25 DEBUG --- status code --- 05:23:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:25 DEBUG --- http response --- 05:23:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:23:25 DEBUG ----- output ----- 05:23:25 DEBUG { 05:23:25 DEBUG "status": "success", 05:23:25 DEBUG "data": { 05:23:25 DEBUG "resultType": "vector", 05:23:25 DEBUG "result": [ 05:23:25 DEBUG { 05:23:25 DEBUG "metric": { 05:23:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:23:25 DEBUG }, 05:23:25 DEBUG "value": [ 05:23:25 DEBUG 05:23:25 DEBUG 05:23:25 DEBUG 05:23:25 DEBUG [...] 05:23:25 DEBUG 05:23:25 DEBUG 05:23:25 DEBUG 05:23:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:23:25 DEBUG }, 05:23:25 DEBUG "value": [ 05:23:25 DEBUG 1712723005, 05:23:25 DEBUG "3.6047872538232197" 05:23:25 DEBUG ] 05:23:25 DEBUG } 05:23:25 DEBUG ] 05:23:25 DEBUG } 05:23:25 DEBUG } 05:23:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712723005" 05:23:25 INFO [http_cmd]: http status code OK 05:23:25 DEBUG --- status code --- 05:23:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:25 DEBUG --- http response --- 05:23:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:23:25 DEBUG ----- output ----- 05:23:25 DEBUG { 05:23:25 DEBUG "status": "success", 05:23:25 DEBUG "data": { 05:23:25 DEBUG "resultType": "vector", 05:23:25 DEBUG "result": [ 05:23:25 DEBUG { 05:23:25 DEBUG "metric": { 05:23:25 DEBUG "pod": "ds-idrepo-0" 05:23:25 DEBUG }, 05:23:25 DEBUG "value": [ 05:23:25 DEBUG 05:23:25 DEBUG 05:23:25 DEBUG 05:23:25 DEBUG [...] 05:23:25 DEBUG 05:23:25 DEBUG 05:23:25 DEBUG 05:23:25 DEBUG "pod": "ds-cts-2" 05:23:25 DEBUG }, 05:23:25 DEBUG "value": [ 05:23:25 DEBUG 1712723005, 05:23:25 DEBUG "0" 05:23:25 DEBUG ] 05:23:25 DEBUG } 05:23:25 DEBUG ] 05:23:25 DEBUG } 05:23:25 DEBUG } 05:23:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723006" 05:23:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723006" 05:23:26 INFO [http_cmd]: http status code OK 05:23:26 DEBUG --- status code --- 05:23:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:26 DEBUG --- http response --- 05:23:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:23:26 INFO [http_cmd]: http status code OK 05:23:26 DEBUG ----- output ----- 05:23:26 DEBUG --- status code --- 05:23:26 DEBUG { 05:23:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:26 DEBUG "status": "success", 05:23:26 DEBUG --- http response --- 05:23:26 DEBUG "data": { 05:23:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:23:26 DEBUG ----- output ----- 05:23:26 DEBUG "resultType": "vector", 05:23:26 DEBUG { 05:23:26 DEBUG "result": [ 05:23:26 DEBUG "status": "success", 05:23:26 DEBUG { 05:23:26 DEBUG "data": { 05:23:26 DEBUG "metric": { 05:23:26 DEBUG "resultType": "vector", 05:23:26 DEBUG "pod": "ds-idrepo-2" 05:23:26 DEBUG "result": [ 05:23:26 DEBUG }, 05:23:26 DEBUG { 05:23:26 DEBUG "value": [ 05:23:26 DEBUG "metric": { 05:23:26 DEBUG 05:23:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:23:26 DEBUG 05:23:26 DEBUG }, 05:23:26 DEBUG 05:23:26 DEBUG "value": [ 05:23:26 DEBUG [...] 05:23:26 DEBUG 05:23:26 DEBUG 05:23:26 DEBUG 05:23:26 DEBUG 05:23:26 DEBUG 05:23:26 DEBUG 05:23:26 DEBUG "pod": "ds-cts-2" 05:23:26 DEBUG [...] 05:23:26 DEBUG }, 05:23:26 DEBUG 05:23:26 DEBUG 05:23:26 DEBUG "value": [ 05:23:26 DEBUG 05:23:26 DEBUG 1712723006, 05:23:26 DEBUG "pod": "ds-idrepo-2" 05:23:26 DEBUG "0.000033333333333333335" 05:23:26 DEBUG }, 05:23:26 DEBUG ] 05:23:26 DEBUG "value": [ 05:23:26 DEBUG } 05:23:26 DEBUG 1712723006, 05:23:26 DEBUG ] 05:23:26 DEBUG } 05:23:26 DEBUG "0.12925046291170528" 05:23:26 DEBUG } 05:23:26 DEBUG ] 05:23:26 DEBUG } 05:23:26 DEBUG ] 05:23:26 DEBUG } 05:23:26 DEBUG } 05:23:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723006" 05:23:26 INFO [http_cmd]: http status code OK 05:23:26 DEBUG --- status code --- 05:23:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:26 DEBUG --- http response --- 05:23:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:23:26 DEBUG ----- output ----- 05:23:26 DEBUG { 05:23:26 DEBUG "status": "success", 05:23:26 DEBUG "data": { 05:23:26 DEBUG "resultType": "vector", 05:23:26 DEBUG "result": [ 05:23:26 DEBUG { 05:23:26 DEBUG "metric": { 05:23:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:23:26 DEBUG }, 05:23:26 DEBUG "value": [ 05:23:26 DEBUG 05:23:26 DEBUG 05:23:26 DEBUG 05:23:26 DEBUG [...] 05:23:26 DEBUG 05:23:26 DEBUG 05:23:26 DEBUG 05:23:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:23:26 DEBUG }, 05:23:26 DEBUG "value": [ 05:23:26 DEBUG 1712723006, 05:23:26 DEBUG "1261.2" 05:23:26 DEBUG ] 05:23:26 DEBUG } 05:23:26 DEBUG ] 05:23:26 DEBUG } 05:23:26 DEBUG } 05:23:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712723006" 05:23:26 INFO [http_cmd]: http status code OK 05:23:26 DEBUG --- status code --- 05:23:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:26 DEBUG --- http response --- 05:23:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:23:26 DEBUG ----- output ----- 05:23:26 DEBUG { 05:23:26 DEBUG "status": "success", 05:23:26 DEBUG "data": { 05:23:26 DEBUG "resultType": "vector", 05:23:26 DEBUG "result": [ 05:23:26 DEBUG { 05:23:26 DEBUG "metric": { 05:23:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:23:26 DEBUG }, 05:23:26 DEBUG "value": [ 05:23:26 DEBUG 05:23:26 DEBUG 05:23:26 DEBUG 05:23:26 DEBUG [...] 05:23:26 DEBUG 05:23:26 DEBUG 05:23:26 DEBUG 05:23:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:23:26 DEBUG }, 05:23:26 DEBUG "value": [ 05:23:26 DEBUG 1712723006, 05:23:26 DEBUG "0.041" 05:23:26 DEBUG ] 05:23:26 DEBUG } 05:23:26 DEBUG ] 05:23:26 DEBUG } 05:23:26 DEBUG } 05:23:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723006" 05:23:26 INFO [http_cmd]: http status code OK 05:23:26 DEBUG --- status code --- 05:23:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:26 DEBUG --- http response --- 05:23:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:23:26 DEBUG ----- output ----- 05:23:26 DEBUG { 05:23:26 DEBUG "status": "success", 05:23:26 DEBUG "data": { 05:23:26 DEBUG "resultType": "vector", 05:23:26 DEBUG "result": [ 05:23:26 DEBUG { 05:23:26 DEBUG "metric": { 05:23:26 DEBUG "pod": "ds-idrepo-0" 05:23:26 DEBUG }, 05:23:26 DEBUG "value": [ 05:23:26 DEBUG 05:23:26 DEBUG 05:23:26 DEBUG 05:23:26 DEBUG [...] 05:23:26 DEBUG 05:23:26 DEBUG 05:23:26 DEBUG 05:23:26 DEBUG "pod": "ds-idrepo-1" 05:23:26 DEBUG }, 05:23:26 DEBUG "value": [ 05:23:26 DEBUG 1712723006, 05:23:26 DEBUG "0" 05:23:26 DEBUG ] 05:23:26 DEBUG } 05:23:26 DEBUG ] 05:23:26 DEBUG } 05:23:26 DEBUG } 05:23:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723006" 05:23:26 INFO [http_cmd]: http status code OK 05:23:26 DEBUG --- status code --- 05:23:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:26 DEBUG --- http response --- 05:23:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:23:26 DEBUG ----- output ----- 05:23:26 DEBUG { 05:23:26 DEBUG "status": "success", 05:23:26 DEBUG "data": { 05:23:26 DEBUG "resultType": "vector", 05:23:26 DEBUG "result": [ 05:23:26 DEBUG { 05:23:26 DEBUG "metric": { 05:23:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:23:26 DEBUG }, 05:23:26 DEBUG "value": [ 05:23:26 DEBUG 05:23:26 DEBUG 05:23:26 DEBUG 05:23:26 DEBUG [...] 05:23:26 DEBUG 05:23:26 DEBUG 05:23:26 DEBUG 05:23:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:23:26 DEBUG }, 05:23:26 DEBUG "value": [ 05:23:26 DEBUG 1712723006, 05:23:26 DEBUG "1272.2" 05:23:26 DEBUG ] 05:23:26 DEBUG } 05:23:26 DEBUG ] 05:23:26 DEBUG } 05:23:26 DEBUG } 05:23:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723006" 05:23:26 INFO [http_cmd]: http status code OK 05:23:26 DEBUG --- status code --- 05:23:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:26 DEBUG --- http response --- 05:23:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:23:26 DEBUG ----- output ----- 05:23:26 DEBUG { 05:23:26 DEBUG "status": "success", 05:23:26 DEBUG "data": { 05:23:26 DEBUG "resultType": "vector", 05:23:26 DEBUG "result": [ 05:23:26 DEBUG { 05:23:26 DEBUG "metric": { 05:23:26 DEBUG "pod": "ds-idrepo-0" 05:23:26 DEBUG }, 05:23:26 DEBUG "value": [ 05:23:26 DEBUG 05:23:26 DEBUG 05:23:26 DEBUG 05:23:26 DEBUG [...] 05:23:26 DEBUG 05:23:26 DEBUG 05:23:26 DEBUG 05:23:26 DEBUG "pod": "ds-idrepo-1" 05:23:26 DEBUG }, 05:23:26 DEBUG "value": [ 05:23:26 DEBUG 1712723006, 05:23:26 DEBUG "0" 05:23:26 DEBUG ] 05:23:26 DEBUG } 05:23:26 DEBUG ] 05:23:26 DEBUG } 05:23:26 DEBUG } 05:23:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712723006" 05:23:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723006" 05:23:27 INFO [http_cmd]: http status code OK 05:23:27 DEBUG --- status code --- 05:23:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:27 DEBUG --- http response --- 05:23:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:23:27 INFO [http_cmd]: http status code OK 05:23:27 DEBUG ----- output ----- 05:23:27 DEBUG --- status code --- 05:23:27 DEBUG { 05:23:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:27 DEBUG "status": "success", 05:23:27 DEBUG --- http response --- 05:23:27 DEBUG "data": { 05:23:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:23:27 DEBUG "resultType": "vector", 05:23:27 DEBUG ----- output ----- 05:23:27 DEBUG "result": [ 05:23:27 DEBUG { 05:23:27 DEBUG "status": "success", 05:23:27 DEBUG "data": { 05:23:27 DEBUG "resultType": "vector", 05:23:27 DEBUG { 05:23:27 DEBUG "result": [ 05:23:27 DEBUG "metric": { 05:23:27 DEBUG { 05:23:27 DEBUG "metric": { 05:23:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:23:27 DEBUG }, 05:23:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:23:27 DEBUG "value": [ 05:23:27 DEBUG }, 05:23:27 DEBUG 05:23:27 DEBUG "value": [ 05:23:27 DEBUG 05:23:27 DEBUG 05:23:27 DEBUG 05:23:27 DEBUG 05:23:27 DEBUG [...] 05:23:27 DEBUG 05:23:27 DEBUG 05:23:27 DEBUG [...] 05:23:27 DEBUG 05:23:27 DEBUG 05:23:27 DEBUG 05:23:27 DEBUG 05:23:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:23:27 DEBUG 05:23:27 DEBUG }, 05:23:27 DEBUG "pod": "ds-idrepo-2" 05:23:27 DEBUG }, 05:23:27 DEBUG "value": [ 05:23:27 DEBUG "value": [ 05:23:27 DEBUG 1712723006, 05:23:27 DEBUG 1712723006, 05:23:27 DEBUG "1442.8556240305206" 05:23:27 DEBUG "2483027968" 05:23:27 DEBUG ] 05:23:27 DEBUG ] 05:23:27 DEBUG } 05:23:27 DEBUG } 05:23:27 DEBUG ] 05:23:27 DEBUG ] 05:23:27 DEBUG } 05:23:27 DEBUG } 05:23:27 DEBUG } 05:23:27 DEBUG } 05:23:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723007" 05:23:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723007" 05:23:27 INFO [http_cmd]: http status code OK 05:23:27 DEBUG --- status code --- 05:23:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:27 DEBUG --- http response --- 05:23:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:23:27 DEBUG ----- output ----- 05:23:27 DEBUG { 05:23:27 DEBUG "status": "success", 05:23:27 DEBUG "data": { 05:23:27 INFO [http_cmd]: http status code OK 05:23:27 DEBUG "resultType": "vector", 05:23:27 DEBUG --- status code --- 05:23:27 DEBUG "result": [ 05:23:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:23:27 DEBUG { 05:23:27 DEBUG --- http response --- 05:23:27 DEBUG "metric": { 05:23:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:23:27 DEBUG "instance": "10.142.0.25:9100" 05:23:27 DEBUG ----- output ----- 05:23:27 DEBUG }, 05:23:27 DEBUG { 05:23:27 DEBUG "value": [ 05:23:27 DEBUG "status": "success", 05:23:27 DEBUG 05:23:27 DEBUG "data": { 05:23:27 DEBUG 05:23:27 DEBUG "resultType": "vector", 05:23:27 DEBUG 05:23:27 DEBUG "result": [ 05:23:27 DEBUG [...] 05:23:27 DEBUG { 05:23:27 DEBUG 05:23:27 DEBUG "metric": { 05:23:27 DEBUG 05:23:27 DEBUG "instance": "10.142.0.25:9100" 05:23:27 DEBUG 05:23:27 DEBUG }, 05:23:27 DEBUG "instance": "10.142.0.77:9100" 05:23:27 DEBUG "value": [ 05:23:27 DEBUG }, 05:23:27 DEBUG 05:23:27 DEBUG "value": [ 05:23:27 DEBUG 05:23:27 DEBUG 1712723007, 05:23:27 DEBUG 05:23:27 DEBUG "37775790.46666667" 05:23:27 DEBUG [...] 05:23:27 DEBUG ] 05:23:27 DEBUG 05:23:27 DEBUG } 05:23:27 DEBUG ] 05:23:27 DEBUG 05:23:27 DEBUG } 05:23:27 DEBUG 05:23:27 DEBUG } 05:23:27 DEBUG "instance": "10.142.0.77:9100" 05:23:27 DEBUG }, 05:23:27 DEBUG "value": [ 05:23:27 DEBUG 1712723007, 05:23:27 DEBUG "0.032333333333333415" 05:23:27 DEBUG ] 05:23:27 DEBUG } 05:23:27 DEBUG ] 05:23:27 DEBUG } 05:23:27 DEBUG } 05:23:41 INFO 05:23:41 INFO [loop_until]: kubectl --namespace=xlou top pods 05:23:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:23:41 INFO [loop_until]: OK (rc = 0) 05:23:41 DEBUG --- stdout --- 05:23:41 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 8965m 8804Mi am-654c4fcbcf-9scrj 8710m 8420Mi am-654c4fcbcf-v5qb5 7401m 8400Mi ds-cts-0 7140m 12408Mi ds-cts-1 7545m 13210Mi ds-cts-2 6691m 13174Mi ds-idrepo-0 9544m 5622Mi ds-idrepo-1 314m 5668Mi ds-idrepo-2 322m 5698Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 5m 2213Mi lodemon-b776b7d88-thrqf 7m 70Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3699m 1658Mi 05:23:41 DEBUG --- stderr --- 05:23:41 DEBUG 05:23:47 INFO 05:23:47 INFO [loop_until]: kubectl --namespace=xlou top node 05:23:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:23:47 INFO [loop_until]: OK (rc = 0) 05:23:47 DEBUG --- stdout --- 05:23:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 92m 0% 2306Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8117m 51% 9277Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 86m 0% 3251Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10500m 66% 6518Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9599m 60% 9304Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9852m 62% 9648Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 76m 0% 856Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9685m 60% 6288Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 7256m 45% 13254Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 6785m 42% 13819Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 386m 2% 6363Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7617m 47% 13787Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 372m 2% 6331Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3708m 23% 5341Mi 9% 05:23:47 DEBUG --- stderr --- 05:23:47 DEBUG 05:24:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712723063" 05:24:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723063" 05:24:23 ERROR [http_cmd]: ERROR 05:24:23 INFO [http_cmd]: http status code OK 05:24:23 ERROR --- status code --- 05:24:23 DEBUG --- status code --- 05:24:23 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:24:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:23 ERROR --- http response --- 05:24:23 DEBUG --- http response --- 05:24:23 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:24:23 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:24:23 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:24:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723063" 05:24:23 INFO [http_cmd]: http status code OK 05:24:23 DEBUG --- status code --- 05:24:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:23 DEBUG --- http response --- 05:24:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:24:23 DEBUG ----- output ----- 05:24:23 DEBUG { 05:24:23 DEBUG "status": "success", 05:24:23 DEBUG "data": { 05:24:23 DEBUG "resultType": "vector", 05:24:23 DEBUG "result": [ 05:24:23 DEBUG { 05:24:23 DEBUG "metric": { 05:24:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:24:23 DEBUG }, 05:24:23 DEBUG "value": [ 05:24:23 DEBUG 05:24:23 DEBUG 05:24:23 DEBUG 05:24:23 DEBUG [...] 05:24:23 DEBUG 05:24:23 DEBUG 05:24:23 DEBUG 05:24:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:24:23 DEBUG }, 05:24:23 DEBUG "value": [ 05:24:23 DEBUG 1712723063, 05:24:23 DEBUG "0" 05:24:23 DEBUG ] 05:24:23 DEBUG } 05:24:23 DEBUG ] 05:24:23 DEBUG } 05:24:23 DEBUG } 05:24:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723063" 05:24:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723063" 05:24:23 INFO [http_cmd]: http status code OK 05:24:23 DEBUG --- status code --- 05:24:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:23 DEBUG --- http response --- 05:24:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723063" 05:24:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:24:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723063" 05:24:23 DEBUG ----- output ----- 05:24:23 INFO [http_cmd]: http status code OK 05:24:23 DEBUG { 05:24:23 DEBUG --- status code --- 05:24:23 DEBUG "status": "success", 05:24:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:23 DEBUG "data": { 05:24:23 DEBUG --- http response --- 05:24:23 DEBUG "resultType": "vector", 05:24:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:24:23 DEBUG "result": [ 05:24:23 DEBUG ----- output ----- 05:24:23 DEBUG { 05:24:23 DEBUG { 05:24:23 DEBUG "metric": { 05:24:23 DEBUG "status": "success", 05:24:23 DEBUG "pod": "ds-idrepo-0" 05:24:23 DEBUG "data": { 05:24:23 DEBUG }, 05:24:23 DEBUG "resultType": "vector", 05:24:23 DEBUG "value": [ 05:24:23 DEBUG "result": [ 05:24:23 DEBUG 05:24:23 DEBUG { 05:24:23 DEBUG 05:24:23 DEBUG 05:24:23 DEBUG [...] 05:24:23 DEBUG 05:24:23 DEBUG 05:24:23 DEBUG 05:24:23 DEBUG "pod": "ds-cts-2" 05:24:23 DEBUG }, 05:24:23 DEBUG "value": [ 05:24:23 DEBUG 1712723063, 05:24:23 DEBUG "29.966666666666665" 05:24:23 DEBUG ] 05:24:23 DEBUG } 05:24:23 DEBUG ] 05:24:23 DEBUG } 05:24:23 DEBUG } 05:24:23 DEBUG "metric": { 05:24:23 DEBUG "instance": "10.142.0.25:9100" 05:24:23 DEBUG }, 05:24:23 DEBUG "value": [ 05:24:23 DEBUG 05:24:23 DEBUG 05:24:23 DEBUG 05:24:23 DEBUG [...] 05:24:23 DEBUG 05:24:23 DEBUG 05:24:23 DEBUG 05:24:23 DEBUG "instance": "10.142.0.77:9100" 05:24:23 DEBUG }, 05:24:23 DEBUG "value": [ 05:24:23 DEBUG 1712723063, 05:24:23 DEBUG "277741003776" 05:24:23 DEBUG ] 05:24:23 DEBUG } 05:24:23 DEBUG ] 05:24:23 DEBUG } 05:24:23 DEBUG } 05:24:23 INFO [http_cmd]: http status code OK 05:24:23 DEBUG --- status code --- 05:24:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:23 DEBUG --- http response --- 05:24:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:24:23 DEBUG ----- output ----- 05:24:23 DEBUG { 05:24:23 DEBUG "status": "success", 05:24:23 DEBUG "data": { 05:24:23 DEBUG "resultType": "vector", 05:24:23 DEBUG "result": [ 05:24:23 DEBUG { 05:24:23 DEBUG "metric": { 05:24:23 DEBUG "pod": "ds-idrepo-0" 05:24:23 DEBUG }, 05:24:23 DEBUG "value": [ 05:24:23 DEBUG 05:24:23 DEBUG 05:24:23 DEBUG 05:24:23 DEBUG [...] 05:24:23 DEBUG 05:24:23 DEBUG 05:24:23 DEBUG 05:24:23 DEBUG "pod": "ds-cts-2" 05:24:23 DEBUG }, 05:24:23 DEBUG "value": [ 05:24:23 DEBUG 1712723063, 05:24:23 DEBUG "0" 05:24:23 DEBUG ] 05:24:23 DEBUG } 05:24:23 DEBUG ] 05:24:23 DEBUG } 05:24:23 DEBUG } 05:24:23 INFO [http_cmd]: http status code OK 05:24:23 DEBUG --- status code --- 05:24:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:23 DEBUG --- http response --- 05:24:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:24:23 DEBUG ----- output ----- 05:24:23 DEBUG { 05:24:23 DEBUG "status": "success", 05:24:23 DEBUG "data": { 05:24:23 DEBUG "resultType": "vector", 05:24:23 DEBUG "result": [ 05:24:23 DEBUG { 05:24:23 DEBUG "metric": { 05:24:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:24:23 DEBUG }, 05:24:23 DEBUG "value": [ 05:24:23 DEBUG 05:24:23 DEBUG 05:24:23 DEBUG 05:24:23 DEBUG [...] 05:24:23 DEBUG 05:24:23 DEBUG 05:24:23 DEBUG 05:24:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:24:23 DEBUG }, 05:24:23 DEBUG "value": [ 05:24:23 DEBUG 1712723063, 05:24:23 DEBUG "1259.6" 05:24:23 DEBUG ] 05:24:23 DEBUG } 05:24:23 DEBUG ] 05:24:23 DEBUG } 05:24:23 DEBUG } 05:24:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723063" 05:24:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723063" 05:24:23 INFO [http_cmd]: http status code OK 05:24:23 DEBUG --- status code --- 05:24:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:23 DEBUG --- http response --- 05:24:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:24:23 DEBUG ----- output ----- 05:24:23 DEBUG { 05:24:23 DEBUG "status": "success", 05:24:23 DEBUG "data": { 05:24:23 DEBUG "resultType": "vector", 05:24:23 DEBUG "result": [ 05:24:23 DEBUG { 05:24:23 DEBUG "metric": { 05:24:23 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:24:23 DEBUG }, 05:24:23 DEBUG "value": [ 05:24:23 DEBUG 05:24:23 DEBUG 05:24:23 DEBUG 05:24:23 DEBUG [...] 05:24:23 DEBUG 05:24:23 DEBUG 05:24:23 DEBUG 05:24:23 DEBUG "pod": "ds-idrepo-2" 05:24:23 DEBUG }, 05:24:23 DEBUG "value": [ 05:24:23 DEBUG 1712723063, 05:24:23 DEBUG "0" 05:24:23 DEBUG ] 05:24:23 DEBUG } 05:24:23 DEBUG ] 05:24:23 DEBUG } 05:24:23 DEBUG } 05:24:23 INFO [http_cmd]: http status code OK 05:24:23 DEBUG --- status code --- 05:24:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:23 DEBUG --- http response --- 05:24:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:24:23 DEBUG ----- output ----- 05:24:23 DEBUG { 05:24:23 DEBUG "status": "success", 05:24:23 DEBUG "data": { 05:24:23 DEBUG "resultType": "vector", 05:24:23 DEBUG "result": [ 05:24:23 DEBUG { 05:24:23 DEBUG "metric": { 05:24:23 DEBUG "instance": "10.142.0.25:9100" 05:24:23 DEBUG }, 05:24:23 DEBUG "value": [ 05:24:23 DEBUG 05:24:23 DEBUG 05:24:23 DEBUG 05:24:23 DEBUG [...] 05:24:23 DEBUG 05:24:23 DEBUG 05:24:23 DEBUG 05:24:23 DEBUG "instance": "10.142.0.77:9100" 05:24:23 DEBUG }, 05:24:23 DEBUG "value": [ 05:24:23 DEBUG 1712723063, 05:24:23 DEBUG "37623105.83333333" 05:24:23 DEBUG ] 05:24:23 DEBUG } 05:24:23 DEBUG ] 05:24:23 DEBUG } 05:24:23 DEBUG } 05:24:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723064" 05:24:24 INFO [http_cmd]: http status code OK 05:24:24 DEBUG --- status code --- 05:24:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:24 DEBUG --- http response --- 05:24:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:24:24 DEBUG ----- output ----- 05:24:24 DEBUG { 05:24:24 DEBUG "status": "success", 05:24:24 DEBUG "data": { 05:24:24 DEBUG "resultType": "vector", 05:24:24 DEBUG "result": [ 05:24:24 DEBUG { 05:24:24 DEBUG "metric": { 05:24:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:24:24 DEBUG }, 05:24:24 DEBUG "value": [ 05:24:24 DEBUG 05:24:24 DEBUG 05:24:24 DEBUG 05:24:24 DEBUG [...] 05:24:24 DEBUG 05:24:24 DEBUG 05:24:24 DEBUG 05:24:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:24:24 DEBUG }, 05:24:24 DEBUG "value": [ 05:24:24 DEBUG 1712723064, 05:24:24 DEBUG "226389.5" 05:24:24 DEBUG ] 05:24:24 DEBUG } 05:24:24 DEBUG ] 05:24:24 DEBUG } 05:24:24 DEBUG } 05:24:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723064" 05:24:24 INFO [http_cmd]: http status code OK 05:24:24 DEBUG --- status code --- 05:24:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:24 DEBUG --- http response --- 05:24:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:24:24 DEBUG ----- output ----- 05:24:24 DEBUG { 05:24:24 DEBUG "status": "success", 05:24:24 DEBUG "data": { 05:24:24 DEBUG "resultType": "vector", 05:24:24 DEBUG "result": [ 05:24:24 DEBUG { 05:24:24 DEBUG "metric": { 05:24:24 DEBUG "pod": "ds-idrepo-0" 05:24:24 DEBUG }, 05:24:24 DEBUG "value": [ 05:24:24 DEBUG 05:24:24 DEBUG 05:24:24 DEBUG 05:24:24 DEBUG [...] 05:24:24 DEBUG 05:24:24 DEBUG 05:24:24 DEBUG 05:24:24 DEBUG "pod": "ds-idrepo-1" 05:24:24 DEBUG }, 05:24:24 DEBUG "value": [ 05:24:24 DEBUG 1712723064, 05:24:24 DEBUG "0" 05:24:24 DEBUG ] 05:24:24 DEBUG } 05:24:24 DEBUG ] 05:24:24 DEBUG } 05:24:24 DEBUG } 05:24:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723064" 05:24:24 INFO [http_cmd]: http status code OK 05:24:24 DEBUG --- status code --- 05:24:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:24 DEBUG --- http response --- 05:24:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:24:24 DEBUG ----- output ----- 05:24:24 DEBUG { 05:24:24 DEBUG "status": "success", 05:24:24 DEBUG "data": { 05:24:24 DEBUG "resultType": "vector", 05:24:24 DEBUG "result": [ 05:24:24 DEBUG { 05:24:24 DEBUG "metric": { 05:24:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:24:24 DEBUG }, 05:24:24 DEBUG "value": [ 05:24:24 DEBUG 05:24:24 DEBUG 05:24:24 DEBUG 05:24:24 DEBUG [...] 05:24:24 DEBUG 05:24:24 DEBUG 05:24:24 DEBUG 05:24:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:24:24 DEBUG }, 05:24:24 DEBUG "value": [ 05:24:24 DEBUG 1712723064, 05:24:24 DEBUG "1259.4333333333334" 05:24:24 DEBUG ] 05:24:24 DEBUG } 05:24:24 DEBUG ] 05:24:24 DEBUG } 05:24:24 DEBUG } 05:24:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723064" 05:24:24 INFO [http_cmd]: http status code OK 05:24:24 DEBUG --- status code --- 05:24:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:24 DEBUG --- http response --- 05:24:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:24:24 DEBUG ----- output ----- 05:24:24 DEBUG { 05:24:24 DEBUG "status": "success", 05:24:24 DEBUG "data": { 05:24:24 DEBUG "resultType": "vector", 05:24:24 DEBUG "result": [ 05:24:24 DEBUG { 05:24:24 DEBUG "metric": { 05:24:24 DEBUG "instance": "10.142.0.25:9100" 05:24:24 DEBUG }, 05:24:24 DEBUG "value": [ 05:24:24 DEBUG 05:24:24 DEBUG 05:24:24 DEBUG 05:24:24 DEBUG [...] 05:24:24 DEBUG 05:24:24 DEBUG 05:24:24 DEBUG 05:24:24 DEBUG "instance": "10.142.0.77:9100" 05:24:24 DEBUG }, 05:24:24 DEBUG "value": [ 05:24:24 DEBUG 1712723064, 05:24:24 DEBUG "0.08626666666666646" 05:24:24 DEBUG ] 05:24:24 DEBUG } 05:24:24 DEBUG ] 05:24:24 DEBUG } 05:24:24 DEBUG } 05:24:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723065" 05:24:25 INFO [http_cmd]: http status code OK 05:24:25 DEBUG --- status code --- 05:24:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:25 DEBUG --- http response --- 05:24:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:24:25 DEBUG ----- output ----- 05:24:25 DEBUG { 05:24:25 DEBUG "status": "success", 05:24:25 DEBUG "data": { 05:24:25 DEBUG "resultType": "vector", 05:24:25 DEBUG "result": [ 05:24:25 DEBUG { 05:24:25 DEBUG "metric": { 05:24:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:24:25 DEBUG }, 05:24:25 DEBUG "value": [ 05:24:25 DEBUG 05:24:25 DEBUG 05:24:25 DEBUG 05:24:25 DEBUG [...] 05:24:25 DEBUG 05:24:25 DEBUG 05:24:25 DEBUG 05:24:25 DEBUG "pod": "ds-idrepo-2" 05:24:25 DEBUG }, 05:24:25 DEBUG "value": [ 05:24:25 DEBUG 1712723065, 05:24:25 DEBUG "0" 05:24:25 DEBUG ] 05:24:25 DEBUG } 05:24:25 DEBUG ] 05:24:25 DEBUG } 05:24:25 DEBUG } 05:24:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712723065" 05:24:25 INFO [http_cmd]: http status code OK 05:24:25 DEBUG --- status code --- 05:24:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:25 DEBUG --- http response --- 05:24:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:24:25 DEBUG ----- output ----- 05:24:25 DEBUG { 05:24:25 DEBUG "status": "success", 05:24:25 DEBUG "data": { 05:24:25 DEBUG "resultType": "vector", 05:24:25 DEBUG "result": [ 05:24:25 DEBUG { 05:24:25 DEBUG "metric": { 05:24:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:24:25 DEBUG }, 05:24:25 DEBUG "value": [ 05:24:25 DEBUG 05:24:25 DEBUG 05:24:25 DEBUG 05:24:25 DEBUG [...] 05:24:25 DEBUG 05:24:25 DEBUG 05:24:25 DEBUG 05:24:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:24:25 DEBUG }, 05:24:25 DEBUG "value": [ 05:24:25 DEBUG 1712723065, 05:24:25 DEBUG "3.682503112925883" 05:24:25 DEBUG ] 05:24:25 DEBUG } 05:24:25 DEBUG ] 05:24:25 DEBUG } 05:24:25 DEBUG } 05:24:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712723065" 05:24:25 INFO [http_cmd]: http status code OK 05:24:25 DEBUG --- status code --- 05:24:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:25 DEBUG --- http response --- 05:24:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:24:25 DEBUG ----- output ----- 05:24:25 DEBUG { 05:24:25 DEBUG "status": "success", 05:24:25 DEBUG "data": { 05:24:25 DEBUG "resultType": "vector", 05:24:25 DEBUG "result": [ 05:24:25 DEBUG { 05:24:25 DEBUG "metric": { 05:24:25 DEBUG "pod": "ds-idrepo-0" 05:24:25 DEBUG }, 05:24:25 DEBUG "value": [ 05:24:25 DEBUG 05:24:25 DEBUG 05:24:25 DEBUG 05:24:25 DEBUG [...] 05:24:25 DEBUG 05:24:25 DEBUG 05:24:25 DEBUG 05:24:25 DEBUG "pod": "ds-cts-2" 05:24:25 DEBUG }, 05:24:25 DEBUG "value": [ 05:24:25 DEBUG 1712723065, 05:24:25 DEBUG "0" 05:24:25 DEBUG ] 05:24:25 DEBUG } 05:24:25 DEBUG ] 05:24:25 DEBUG } 05:24:25 DEBUG } 05:24:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723066" 05:24:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723066" 05:24:26 INFO [http_cmd]: http status code OK 05:24:26 DEBUG --- status code --- 05:24:26 INFO [http_cmd]: http status code OK 05:24:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:26 DEBUG --- status code --- 05:24:26 DEBUG --- http response --- 05:24:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:24:26 DEBUG --- http response --- 05:24:26 DEBUG ----- output ----- 05:24:26 DEBUG { 05:24:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:24:26 DEBUG "status": "success", 05:24:26 DEBUG ----- output ----- 05:24:26 DEBUG "data": { 05:24:26 DEBUG { 05:24:26 DEBUG "resultType": "vector", 05:24:26 DEBUG "status": "success", 05:24:26 DEBUG "result": [ 05:24:26 DEBUG "data": { 05:24:26 DEBUG { 05:24:26 DEBUG "resultType": "vector", 05:24:26 DEBUG "metric": { 05:24:26 DEBUG "result": [ 05:24:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:24:26 DEBUG { 05:24:26 DEBUG }, 05:24:26 DEBUG "metric": { 05:24:26 DEBUG "value": [ 05:24:26 DEBUG "pod": "ds-idrepo-2" 05:24:26 DEBUG 05:24:26 DEBUG }, 05:24:26 DEBUG 05:24:26 DEBUG "value": [ 05:24:26 DEBUG 05:24:26 DEBUG 05:24:26 DEBUG 05:24:26 DEBUG [...] 05:24:26 DEBUG 05:24:26 DEBUG 05:24:26 DEBUG [...] 05:24:26 DEBUG 05:24:26 DEBUG 05:24:26 DEBUG 05:24:26 DEBUG 05:24:26 DEBUG "pod": "ds-idrepo-2" 05:24:26 DEBUG 05:24:26 DEBUG }, 05:24:26 DEBUG "pod": "ds-cts-2" 05:24:26 DEBUG "value": [ 05:24:26 DEBUG }, 05:24:26 DEBUG 1712723066, 05:24:26 DEBUG "value": [ 05:24:26 DEBUG "0" 05:24:26 DEBUG 1712723066, 05:24:26 DEBUG ] 05:24:26 DEBUG "0.00015399666666666666" 05:24:26 DEBUG } 05:24:26 DEBUG ] 05:24:26 DEBUG ] 05:24:26 DEBUG } 05:24:26 DEBUG } 05:24:26 DEBUG ] 05:24:26 DEBUG } 05:24:26 DEBUG } 05:24:26 DEBUG } 05:24:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723066" 05:24:26 INFO [http_cmd]: http status code OK 05:24:26 DEBUG --- status code --- 05:24:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:26 DEBUG --- http response --- 05:24:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:24:26 DEBUG ----- output ----- 05:24:26 DEBUG { 05:24:26 DEBUG "status": "success", 05:24:26 DEBUG "data": { 05:24:26 DEBUG "resultType": "vector", 05:24:26 DEBUG "result": [ 05:24:26 DEBUG { 05:24:26 DEBUG "metric": { 05:24:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:24:26 DEBUG }, 05:24:26 DEBUG "value": [ 05:24:26 DEBUG 05:24:26 DEBUG 05:24:26 DEBUG 05:24:26 DEBUG [...] 05:24:26 DEBUG 05:24:26 DEBUG 05:24:26 DEBUG 05:24:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:24:26 DEBUG }, 05:24:26 DEBUG "value": [ 05:24:26 DEBUG 1712723066, 05:24:26 DEBUG "1259.8666666666666" 05:24:26 DEBUG ] 05:24:26 DEBUG } 05:24:26 DEBUG ] 05:24:26 DEBUG } 05:24:26 DEBUG } 05:24:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712723066" 05:24:26 INFO [http_cmd]: http status code OK 05:24:26 DEBUG --- status code --- 05:24:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:26 DEBUG --- http response --- 05:24:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:24:26 DEBUG ----- output ----- 05:24:26 DEBUG { 05:24:26 DEBUG "status": "success", 05:24:26 DEBUG "data": { 05:24:26 DEBUG "resultType": "vector", 05:24:26 DEBUG "result": [ 05:24:26 DEBUG { 05:24:26 DEBUG "metric": { 05:24:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:24:26 DEBUG }, 05:24:26 DEBUG "value": [ 05:24:26 DEBUG 05:24:26 DEBUG 05:24:26 DEBUG 05:24:26 DEBUG [...] 05:24:26 DEBUG 05:24:26 DEBUG 05:24:26 DEBUG 05:24:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:24:26 DEBUG }, 05:24:26 DEBUG "value": [ 05:24:26 DEBUG 1712723066, 05:24:26 DEBUG "0.041" 05:24:26 DEBUG ] 05:24:26 DEBUG } 05:24:26 DEBUG ] 05:24:26 DEBUG } 05:24:26 DEBUG } 05:24:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723066" 05:24:26 INFO [http_cmd]: http status code OK 05:24:26 DEBUG --- status code --- 05:24:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:26 DEBUG --- http response --- 05:24:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:24:26 DEBUG ----- output ----- 05:24:26 DEBUG { 05:24:26 DEBUG "status": "success", 05:24:26 DEBUG "data": { 05:24:26 DEBUG "resultType": "vector", 05:24:26 DEBUG "result": [ 05:24:26 DEBUG { 05:24:26 DEBUG "metric": { 05:24:26 DEBUG "pod": "ds-idrepo-0" 05:24:26 DEBUG }, 05:24:26 DEBUG "value": [ 05:24:26 DEBUG 05:24:26 DEBUG 05:24:26 DEBUG 05:24:26 DEBUG [...] 05:24:26 DEBUG 05:24:26 DEBUG 05:24:26 DEBUG 05:24:26 DEBUG "pod": "ds-idrepo-1" 05:24:26 DEBUG }, 05:24:26 DEBUG "value": [ 05:24:26 DEBUG 1712723066, 05:24:26 DEBUG "0" 05:24:26 DEBUG ] 05:24:26 DEBUG } 05:24:26 DEBUG ] 05:24:26 DEBUG } 05:24:26 DEBUG } 05:24:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723066" 05:24:26 INFO [http_cmd]: http status code OK 05:24:26 DEBUG --- status code --- 05:24:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:26 DEBUG --- http response --- 05:24:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:24:26 DEBUG ----- output ----- 05:24:26 DEBUG { 05:24:26 DEBUG "status": "success", 05:24:26 DEBUG "data": { 05:24:26 DEBUG "resultType": "vector", 05:24:26 DEBUG "result": [ 05:24:26 DEBUG { 05:24:26 DEBUG "metric": { 05:24:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:24:26 DEBUG }, 05:24:26 DEBUG "value": [ 05:24:26 DEBUG 05:24:26 DEBUG 05:24:26 DEBUG 05:24:26 DEBUG [...] 05:24:26 DEBUG 05:24:26 DEBUG 05:24:26 DEBUG 05:24:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:24:26 DEBUG }, 05:24:26 DEBUG "value": [ 05:24:26 DEBUG 1712723066, 05:24:26 DEBUG "1258.8" 05:24:26 DEBUG ] 05:24:26 DEBUG } 05:24:26 DEBUG ] 05:24:26 DEBUG } 05:24:26 DEBUG } 05:24:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723067" 05:24:27 INFO [http_cmd]: http status code OK 05:24:27 DEBUG --- status code --- 05:24:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:27 DEBUG --- http response --- 05:24:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:24:27 DEBUG ----- output ----- 05:24:27 DEBUG { 05:24:27 DEBUG "status": "success", 05:24:27 DEBUG "data": { 05:24:27 DEBUG "resultType": "vector", 05:24:27 DEBUG "result": [ 05:24:27 DEBUG { 05:24:27 DEBUG "metric": { 05:24:27 DEBUG "pod": "ds-idrepo-0" 05:24:27 DEBUG }, 05:24:27 DEBUG "value": [ 05:24:27 DEBUG 05:24:27 DEBUG 05:24:27 DEBUG 05:24:27 DEBUG [...] 05:24:27 DEBUG 05:24:27 DEBUG 05:24:27 DEBUG 05:24:27 DEBUG "pod": "ds-idrepo-1" 05:24:27 DEBUG }, 05:24:27 DEBUG "value": [ 05:24:27 DEBUG 1712723067, 05:24:27 DEBUG "0" 05:24:27 DEBUG ] 05:24:27 DEBUG } 05:24:27 DEBUG ] 05:24:27 DEBUG } 05:24:27 DEBUG } 05:24:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712723067" 05:24:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723067" 05:24:27 INFO [http_cmd]: http status code OK 05:24:27 DEBUG --- status code --- 05:24:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:27 DEBUG --- http response --- 05:24:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:24:27 DEBUG ----- output ----- 05:24:27 DEBUG { 05:24:27 DEBUG "status": "success", 05:24:27 DEBUG "data": { 05:24:27 DEBUG "resultType": "vector", 05:24:27 DEBUG "result": [ 05:24:27 DEBUG { 05:24:27 DEBUG "metric": { 05:24:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:24:27 DEBUG }, 05:24:27 DEBUG "value": [ 05:24:27 DEBUG 05:24:27 DEBUG 05:24:27 DEBUG 05:24:27 DEBUG [...] 05:24:27 DEBUG 05:24:27 DEBUG 05:24:27 DEBUG 05:24:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:24:27 DEBUG }, 05:24:27 DEBUG "value": [ 05:24:27 DEBUG 1712723067, 05:24:27 DEBUG "2484199424" 05:24:27 DEBUG ] 05:24:27 DEBUG } 05:24:27 DEBUG ] 05:24:27 DEBUG } 05:24:27 DEBUG } 05:24:27 INFO [http_cmd]: http status code OK 05:24:27 DEBUG --- status code --- 05:24:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:27 DEBUG --- http response --- 05:24:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:24:27 DEBUG ----- output ----- 05:24:27 DEBUG { 05:24:27 DEBUG "status": "success", 05:24:27 DEBUG "data": { 05:24:27 DEBUG "resultType": "vector", 05:24:27 DEBUG "result": [ 05:24:27 DEBUG { 05:24:27 DEBUG "metric": { 05:24:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:24:27 DEBUG }, 05:24:27 DEBUG "value": [ 05:24:27 DEBUG 05:24:27 DEBUG 05:24:27 DEBUG 05:24:27 DEBUG [...] 05:24:27 DEBUG 05:24:27 DEBUG 05:24:27 DEBUG 05:24:27 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 05:24:27 DEBUG }, 05:24:27 DEBUG "value": [ 05:24:27 DEBUG 1712723067, 05:24:27 DEBUG "0" 05:24:27 DEBUG ] 05:24:27 DEBUG } 05:24:27 DEBUG ] 05:24:27 DEBUG } 05:24:27 DEBUG } 05:24:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723067" 05:24:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723067" 05:24:27 INFO [http_cmd]: http status code OK 05:24:27 DEBUG --- status code --- 05:24:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:27 DEBUG --- http response --- 05:24:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:24:27 DEBUG ----- output ----- 05:24:27 DEBUG { 05:24:27 DEBUG "status": "success", 05:24:27 DEBUG "data": { 05:24:27 DEBUG "resultType": "vector", 05:24:27 DEBUG "result": [ 05:24:27 DEBUG { 05:24:27 DEBUG "metric": { 05:24:27 DEBUG "instance": "10.142.0.25:9100" 05:24:27 DEBUG }, 05:24:27 DEBUG "value": [ 05:24:27 DEBUG 05:24:27 DEBUG 05:24:27 DEBUG 05:24:27 DEBUG [...] 05:24:27 DEBUG 05:24:27 DEBUG 05:24:27 DEBUG 05:24:27 DEBUG "instance": "10.142.0.77:9100" 05:24:27 DEBUG }, 05:24:27 DEBUG "value": [ 05:24:27 DEBUG 1712723067, 05:24:27 DEBUG "37622278.13333333" 05:24:27 DEBUG ] 05:24:27 DEBUG } 05:24:27 DEBUG ] 05:24:27 DEBUG } 05:24:27 DEBUG } 05:24:27 INFO [http_cmd]: http status code OK 05:24:27 DEBUG --- status code --- 05:24:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:24:27 DEBUG --- http response --- 05:24:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:24:27 DEBUG ----- output ----- 05:24:27 DEBUG { 05:24:27 DEBUG "status": "success", 05:24:27 DEBUG "data": { 05:24:27 DEBUG "resultType": "vector", 05:24:27 DEBUG "result": [ 05:24:27 DEBUG { 05:24:27 DEBUG "metric": { 05:24:27 DEBUG "instance": "10.142.0.25:9100" 05:24:27 DEBUG }, 05:24:27 DEBUG "value": [ 05:24:27 DEBUG 05:24:27 DEBUG 05:24:27 DEBUG 05:24:27 DEBUG [...] 05:24:27 DEBUG 05:24:27 DEBUG 05:24:27 DEBUG 05:24:27 DEBUG "instance": "10.142.0.77:9100" 05:24:27 DEBUG }, 05:24:27 DEBUG "value": [ 05:24:27 DEBUG 1712723067, 05:24:27 DEBUG "0.0303333333333334" 05:24:27 DEBUG ] 05:24:27 DEBUG } 05:24:27 DEBUG ] 05:24:27 DEBUG } 05:24:27 DEBUG } 05:24:41 INFO 05:24:41 INFO [loop_until]: kubectl --namespace=xlou top pods 05:24:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:24:41 INFO [loop_until]: OK (rc = 0) 05:24:41 DEBUG --- stdout --- 05:24:41 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 8971m 8807Mi am-654c4fcbcf-9scrj 8695m 8423Mi am-654c4fcbcf-v5qb5 7363m 8402Mi ds-cts-0 7377m 12610Mi ds-cts-1 7204m 12576Mi ds-cts-2 6700m 13259Mi ds-idrepo-0 9692m 5621Mi ds-idrepo-1 316m 5668Mi ds-idrepo-2 335m 5698Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 4m 70Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3594m 1659Mi 05:24:41 DEBUG --- stderr --- 05:24:41 DEBUG 05:24:47 INFO 05:24:47 INFO [loop_until]: kubectl --namespace=xlou top node 05:24:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:24:48 INFO [loop_until]: OK (rc = 0) 05:24:48 DEBUG --- stdout --- 05:24:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 92m 0% 2302Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8189m 51% 9279Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 85m 0% 3260Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10786m 67% 6512Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9550m 60% 9307Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9950m 62% 9659Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 75m 0% 858Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9776m 61% 6289Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 7468m 46% 13202Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 6320m 39% 13932Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 385m 2% 6365Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7493m 47% 13278Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 371m 2% 6332Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3582m 22% 5342Mi 9% 05:24:48 DEBUG --- stderr --- 05:24:48 DEBUG 05:25:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712723123" 05:25:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723123" 05:25:23 ERROR [http_cmd]: ERROR 05:25:23 INFO [http_cmd]: http status code OK 05:25:23 ERROR --- status code --- 05:25:23 DEBUG --- status code --- 05:25:23 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:25:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:23 ERROR --- http response --- 05:25:23 DEBUG --- http response --- 05:25:23 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:25:23 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:25:23 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:25:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723123" 05:25:23 INFO [http_cmd]: http status code OK 05:25:23 DEBUG --- status code --- 05:25:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:23 DEBUG --- http response --- 05:25:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:25:23 DEBUG ----- output ----- 05:25:23 DEBUG { 05:25:23 DEBUG "status": "success", 05:25:23 DEBUG "data": { 05:25:23 DEBUG "resultType": "vector", 05:25:23 DEBUG "result": [ 05:25:23 DEBUG { 05:25:23 DEBUG "metric": { 05:25:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:25:23 DEBUG }, 05:25:23 DEBUG "value": [ 05:25:23 DEBUG 05:25:23 DEBUG 05:25:23 DEBUG 05:25:23 DEBUG [...] 05:25:23 DEBUG 05:25:23 DEBUG 05:25:23 DEBUG 05:25:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:25:23 DEBUG }, 05:25:23 DEBUG "value": [ 05:25:23 DEBUG 1712723123, 05:25:23 DEBUG "0" 05:25:23 DEBUG ] 05:25:23 DEBUG } 05:25:23 DEBUG ] 05:25:23 DEBUG } 05:25:23 DEBUG } 05:25:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723123" 05:25:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723123" 05:25:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723123" 05:25:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723123" 05:25:23 INFO [http_cmd]: http status code OK 05:25:23 INFO [http_cmd]: http status code OK 05:25:23 DEBUG --- status code --- 05:25:23 DEBUG --- status code --- 05:25:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:23 DEBUG --- http response --- 05:25:23 DEBUG --- http response --- 05:25:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:25:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:25:23 DEBUG ----- output ----- 05:25:23 DEBUG { 05:25:23 DEBUG "status": "success", 05:25:23 DEBUG "data": { 05:25:23 DEBUG "resultType": "vector", 05:25:23 DEBUG "result": [ 05:25:23 DEBUG { 05:25:23 DEBUG "metric": { 05:25:23 DEBUG "pod": "ds-idrepo-0" 05:25:23 DEBUG }, 05:25:23 DEBUG "value": [ 05:25:23 DEBUG 05:25:23 DEBUG 05:25:23 DEBUG 05:25:23 DEBUG [...] 05:25:23 DEBUG 05:25:23 DEBUG 05:25:23 DEBUG 05:25:23 DEBUG "pod": "ds-cts-2" 05:25:23 DEBUG }, 05:25:23 DEBUG "value": [ 05:25:23 DEBUG 1712723123, 05:25:23 DEBUG "0" 05:25:23 DEBUG ] 05:25:23 DEBUG } 05:25:23 DEBUG ] 05:25:23 DEBUG } 05:25:23 DEBUG } 05:25:23 DEBUG ----- output ----- 05:25:23 INFO [http_cmd]: http status code OK 05:25:23 DEBUG { 05:25:23 DEBUG "status": "success", 05:25:23 DEBUG "data": { 05:25:23 DEBUG "resultType": "vector", 05:25:23 DEBUG "result": [ 05:25:23 DEBUG { 05:25:23 DEBUG "metric": { 05:25:23 DEBUG "pod": "ds-idrepo-0" 05:25:23 DEBUG }, 05:25:23 DEBUG "value": [ 05:25:23 DEBUG 05:25:23 DEBUG 05:25:23 DEBUG 05:25:23 DEBUG [...] 05:25:23 DEBUG 05:25:23 DEBUG 05:25:23 DEBUG 05:25:23 DEBUG "pod": "ds-cts-2" 05:25:23 DEBUG }, 05:25:23 DEBUG "value": [ 05:25:23 DEBUG 1712723123, 05:25:23 DEBUG "29.633333333333333" 05:25:23 DEBUG ] 05:25:23 DEBUG } 05:25:23 DEBUG ] 05:25:23 DEBUG } 05:25:23 DEBUG } 05:25:23 DEBUG --- status code --- 05:25:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:23 DEBUG --- http response --- 05:25:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:25:23 DEBUG ----- output ----- 05:25:23 DEBUG { 05:25:23 DEBUG "status": "success", 05:25:23 DEBUG "data": { 05:25:23 DEBUG "resultType": "vector", 05:25:23 DEBUG "result": [ 05:25:23 DEBUG { 05:25:23 DEBUG "metric": { 05:25:23 DEBUG "instance": "10.142.0.25:9100" 05:25:23 DEBUG }, 05:25:23 DEBUG "value": [ 05:25:23 DEBUG 05:25:23 DEBUG 05:25:23 DEBUG 05:25:23 DEBUG [...] 05:25:23 DEBUG 05:25:23 DEBUG 05:25:23 DEBUG 05:25:23 INFO [http_cmd]: http status code OK 05:25:23 DEBUG "instance": "10.142.0.77:9100" 05:25:23 DEBUG --- status code --- 05:25:23 DEBUG }, 05:25:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:23 DEBUG "value": [ 05:25:23 DEBUG --- http response --- 05:25:23 DEBUG 1712723123, 05:25:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:25:23 DEBUG "279448228864" 05:25:23 DEBUG ----- output ----- 05:25:23 DEBUG ] 05:25:23 DEBUG { 05:25:23 DEBUG } 05:25:23 DEBUG "status": "success", 05:25:23 DEBUG ] 05:25:23 DEBUG "data": { 05:25:23 DEBUG } 05:25:23 DEBUG "resultType": "vector", 05:25:23 DEBUG } 05:25:23 DEBUG "result": [ 05:25:23 DEBUG { 05:25:23 DEBUG "metric": { 05:25:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:25:23 DEBUG }, 05:25:23 DEBUG "value": [ 05:25:23 DEBUG 05:25:23 DEBUG 05:25:23 DEBUG 05:25:23 DEBUG [...] 05:25:23 DEBUG 05:25:23 DEBUG 05:25:23 DEBUG 05:25:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:25:23 DEBUG }, 05:25:23 DEBUG "value": [ 05:25:23 DEBUG 1712723123, 05:25:23 DEBUG "1264.7" 05:25:23 DEBUG ] 05:25:23 DEBUG } 05:25:23 DEBUG ] 05:25:23 DEBUG } 05:25:23 DEBUG } 05:25:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723123" 05:25:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723123" 05:25:23 INFO [http_cmd]: http status code OK 05:25:23 DEBUG --- status code --- 05:25:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:23 DEBUG --- http response --- 05:25:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:25:23 DEBUG ----- output ----- 05:25:23 DEBUG { 05:25:23 DEBUG "status": "success", 05:25:23 DEBUG "data": { 05:25:23 DEBUG "resultType": "vector", 05:25:23 DEBUG "result": [ 05:25:23 DEBUG { 05:25:23 DEBUG "metric": { 05:25:23 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:25:23 DEBUG }, 05:25:23 DEBUG "value": [ 05:25:23 DEBUG 05:25:23 DEBUG 05:25:23 DEBUG 05:25:23 DEBUG [...] 05:25:23 DEBUG 05:25:23 DEBUG 05:25:23 DEBUG 05:25:23 DEBUG "pod": "ds-idrepo-2" 05:25:23 DEBUG }, 05:25:23 DEBUG "value": [ 05:25:23 DEBUG 1712723123, 05:25:23 DEBUG "0" 05:25:23 DEBUG ] 05:25:23 DEBUG } 05:25:23 INFO [http_cmd]: http status code OK 05:25:23 DEBUG ] 05:25:23 DEBUG --- status code --- 05:25:23 DEBUG } 05:25:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:23 DEBUG } 05:25:23 DEBUG --- http response --- 05:25:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:25:23 DEBUG ----- output ----- 05:25:23 DEBUG { 05:25:23 DEBUG "status": "success", 05:25:23 DEBUG "data": { 05:25:23 DEBUG "resultType": "vector", 05:25:23 DEBUG "result": [ 05:25:23 DEBUG { 05:25:23 DEBUG "metric": { 05:25:23 DEBUG "instance": "10.142.0.25:9100" 05:25:23 DEBUG }, 05:25:23 DEBUG "value": [ 05:25:23 DEBUG 05:25:23 DEBUG 05:25:23 DEBUG 05:25:23 DEBUG [...] 05:25:23 DEBUG 05:25:23 DEBUG 05:25:23 DEBUG 05:25:23 DEBUG "instance": "10.142.0.77:9100" 05:25:23 DEBUG }, 05:25:23 DEBUG "value": [ 05:25:23 DEBUG 1712723123, 05:25:23 DEBUG "37685669.5" 05:25:23 DEBUG ] 05:25:23 DEBUG } 05:25:23 DEBUG ] 05:25:23 DEBUG } 05:25:23 DEBUG } 05:25:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723124" 05:25:24 INFO [http_cmd]: http status code OK 05:25:24 DEBUG --- status code --- 05:25:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:24 DEBUG --- http response --- 05:25:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:25:24 DEBUG ----- output ----- 05:25:24 DEBUG { 05:25:24 DEBUG "status": "success", 05:25:24 DEBUG "data": { 05:25:24 DEBUG "resultType": "vector", 05:25:24 DEBUG "result": [ 05:25:24 DEBUG { 05:25:24 DEBUG "metric": { 05:25:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:25:24 DEBUG }, 05:25:24 DEBUG "value": [ 05:25:24 DEBUG 05:25:24 DEBUG 05:25:24 DEBUG 05:25:24 DEBUG [...] 05:25:24 DEBUG 05:25:24 DEBUG 05:25:24 DEBUG 05:25:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:25:24 DEBUG }, 05:25:24 DEBUG "value": [ 05:25:24 DEBUG 1712723124, 05:25:24 DEBUG "226447" 05:25:24 DEBUG ] 05:25:24 DEBUG } 05:25:24 DEBUG ] 05:25:24 DEBUG } 05:25:24 DEBUG } 05:25:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723124" 05:25:24 INFO [http_cmd]: http status code OK 05:25:24 DEBUG --- status code --- 05:25:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:24 DEBUG --- http response --- 05:25:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:25:24 DEBUG ----- output ----- 05:25:24 DEBUG { 05:25:24 DEBUG "status": "success", 05:25:24 DEBUG "data": { 05:25:24 DEBUG "resultType": "vector", 05:25:24 DEBUG "result": [ 05:25:24 DEBUG { 05:25:24 DEBUG "metric": { 05:25:24 DEBUG "pod": "ds-idrepo-0" 05:25:24 DEBUG }, 05:25:24 DEBUG "value": [ 05:25:24 DEBUG 05:25:24 DEBUG 05:25:24 DEBUG 05:25:24 DEBUG [...] 05:25:24 DEBUG 05:25:24 DEBUG 05:25:24 DEBUG 05:25:24 DEBUG "pod": "ds-idrepo-1" 05:25:24 DEBUG }, 05:25:24 DEBUG "value": [ 05:25:24 DEBUG 1712723124, 05:25:24 DEBUG "0" 05:25:24 DEBUG ] 05:25:24 DEBUG } 05:25:24 DEBUG ] 05:25:24 DEBUG } 05:25:24 DEBUG } 05:25:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723124" 05:25:24 INFO [http_cmd]: http status code OK 05:25:24 DEBUG --- status code --- 05:25:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:24 DEBUG --- http response --- 05:25:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:25:24 DEBUG ----- output ----- 05:25:24 DEBUG { 05:25:24 DEBUG "status": "success", 05:25:24 DEBUG "data": { 05:25:24 DEBUG "resultType": "vector", 05:25:24 DEBUG "result": [ 05:25:24 DEBUG { 05:25:24 DEBUG "metric": { 05:25:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:25:24 DEBUG }, 05:25:24 DEBUG "value": [ 05:25:24 DEBUG 05:25:24 DEBUG 05:25:24 DEBUG 05:25:24 DEBUG [...] 05:25:24 DEBUG 05:25:24 DEBUG 05:25:24 DEBUG 05:25:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:25:24 DEBUG }, 05:25:24 DEBUG "value": [ 05:25:24 DEBUG 1712723124, 05:25:24 DEBUG "1265.7333333333333" 05:25:24 DEBUG ] 05:25:24 DEBUG } 05:25:24 DEBUG ] 05:25:24 DEBUG } 05:25:24 DEBUG } 05:25:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723124" 05:25:24 INFO [http_cmd]: http status code OK 05:25:24 DEBUG --- status code --- 05:25:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:24 DEBUG --- http response --- 05:25:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:25:24 DEBUG ----- output ----- 05:25:24 DEBUG { 05:25:24 DEBUG "status": "success", 05:25:24 DEBUG "data": { 05:25:24 DEBUG "resultType": "vector", 05:25:24 DEBUG "result": [ 05:25:24 DEBUG { 05:25:24 DEBUG "metric": { 05:25:24 DEBUG "instance": "10.142.0.25:9100" 05:25:24 DEBUG }, 05:25:24 DEBUG "value": [ 05:25:24 DEBUG 05:25:24 DEBUG 05:25:24 DEBUG 05:25:24 DEBUG [...] 05:25:24 DEBUG 05:25:24 DEBUG 05:25:24 DEBUG 05:25:24 DEBUG "instance": "10.142.0.77:9100" 05:25:24 DEBUG }, 05:25:24 DEBUG "value": [ 05:25:24 DEBUG 1712723124, 05:25:24 DEBUG "0.10266666666666922" 05:25:24 DEBUG ] 05:25:24 DEBUG } 05:25:24 DEBUG ] 05:25:24 DEBUG } 05:25:24 DEBUG } 05:25:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723125" 05:25:25 INFO [http_cmd]: http status code OK 05:25:25 DEBUG --- status code --- 05:25:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:25 DEBUG --- http response --- 05:25:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:25:25 DEBUG ----- output ----- 05:25:25 DEBUG { 05:25:25 DEBUG "status": "success", 05:25:25 DEBUG "data": { 05:25:25 DEBUG "resultType": "vector", 05:25:25 DEBUG "result": [ 05:25:25 DEBUG { 05:25:25 DEBUG "metric": { 05:25:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:25:25 DEBUG }, 05:25:25 DEBUG "value": [ 05:25:25 DEBUG 05:25:25 DEBUG 05:25:25 DEBUG 05:25:25 DEBUG [...] 05:25:25 DEBUG 05:25:25 DEBUG 05:25:25 DEBUG 05:25:25 DEBUG "pod": "ds-idrepo-2" 05:25:25 DEBUG }, 05:25:25 DEBUG "value": [ 05:25:25 DEBUG 1712723125, 05:25:25 DEBUG "162734724.0171474" 05:25:25 DEBUG ] 05:25:25 DEBUG } 05:25:25 DEBUG ] 05:25:25 DEBUG } 05:25:25 DEBUG } 05:25:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712723125" 05:25:25 INFO [http_cmd]: http status code OK 05:25:25 DEBUG --- status code --- 05:25:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:25 DEBUG --- http response --- 05:25:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:25:25 DEBUG ----- output ----- 05:25:25 DEBUG { 05:25:25 DEBUG "status": "success", 05:25:25 DEBUG "data": { 05:25:25 DEBUG "resultType": "vector", 05:25:25 DEBUG "result": [ 05:25:25 DEBUG { 05:25:25 DEBUG "metric": { 05:25:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:25:25 DEBUG }, 05:25:25 DEBUG "value": [ 05:25:25 DEBUG 05:25:25 DEBUG 05:25:25 DEBUG 05:25:25 DEBUG [...] 05:25:25 DEBUG 05:25:25 DEBUG 05:25:25 DEBUG 05:25:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:25:25 DEBUG }, 05:25:25 DEBUG "value": [ 05:25:25 DEBUG 1712723125, 05:25:25 DEBUG "3.6202105549355053" 05:25:25 DEBUG ] 05:25:25 DEBUG } 05:25:25 DEBUG ] 05:25:25 DEBUG } 05:25:25 DEBUG } 05:25:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712723126" 05:25:26 INFO [http_cmd]: http status code OK 05:25:26 DEBUG --- status code --- 05:25:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:26 DEBUG --- http response --- 05:25:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:25:26 DEBUG ----- output ----- 05:25:26 DEBUG { 05:25:26 DEBUG "status": "success", 05:25:26 DEBUG "data": { 05:25:26 DEBUG "resultType": "vector", 05:25:26 DEBUG "result": [ 05:25:26 DEBUG { 05:25:26 DEBUG "metric": { 05:25:26 DEBUG "pod": "ds-idrepo-0" 05:25:26 DEBUG }, 05:25:26 DEBUG "value": [ 05:25:26 DEBUG 05:25:26 DEBUG 05:25:26 DEBUG 05:25:26 DEBUG [...] 05:25:26 DEBUG 05:25:26 DEBUG 05:25:26 DEBUG 05:25:26 DEBUG "pod": "ds-cts-2" 05:25:26 DEBUG }, 05:25:26 DEBUG "value": [ 05:25:26 DEBUG 1712723126, 05:25:26 DEBUG "0" 05:25:26 DEBUG ] 05:25:26 DEBUG } 05:25:26 DEBUG ] 05:25:26 DEBUG } 05:25:26 DEBUG } 05:25:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723126" 05:25:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723126" 05:25:26 INFO [http_cmd]: http status code OK 05:25:26 DEBUG --- status code --- 05:25:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:26 DEBUG --- http response --- 05:25:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:25:26 DEBUG ----- output ----- 05:25:26 DEBUG { 05:25:26 DEBUG "status": "success", 05:25:26 DEBUG "data": { 05:25:26 DEBUG "resultType": "vector", 05:25:26 DEBUG "result": [ 05:25:26 DEBUG { 05:25:26 DEBUG "metric": { 05:25:26 DEBUG "pod": "ds-idrepo-2" 05:25:26 DEBUG }, 05:25:26 DEBUG "value": [ 05:25:26 DEBUG 05:25:26 DEBUG 05:25:26 DEBUG 05:25:26 DEBUG [...] 05:25:26 DEBUG 05:25:26 DEBUG 05:25:26 DEBUG 05:25:26 DEBUG "pod": "ds-cts-2" 05:25:26 DEBUG }, 05:25:26 DEBUG "value": [ 05:25:26 DEBUG 1712723126, 05:25:26 DEBUG "0" 05:25:26 DEBUG ] 05:25:26 DEBUG } 05:25:26 DEBUG ] 05:25:26 DEBUG } 05:25:26 DEBUG } 05:25:26 INFO [http_cmd]: http status code OK 05:25:26 DEBUG --- status code --- 05:25:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:26 DEBUG --- http response --- 05:25:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:25:26 DEBUG ----- output ----- 05:25:26 DEBUG { 05:25:26 DEBUG "status": "success", 05:25:26 DEBUG "data": { 05:25:26 DEBUG "resultType": "vector", 05:25:26 DEBUG "result": [ 05:25:26 DEBUG { 05:25:26 DEBUG "metric": { 05:25:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:25:26 DEBUG }, 05:25:26 DEBUG "value": [ 05:25:26 DEBUG 05:25:26 DEBUG 05:25:26 DEBUG 05:25:26 DEBUG [...] 05:25:26 DEBUG 05:25:26 DEBUG 05:25:26 DEBUG 05:25:26 DEBUG "pod": "ds-idrepo-2" 05:25:26 DEBUG }, 05:25:26 DEBUG "value": [ 05:25:26 DEBUG 1712723126, 05:25:26 DEBUG "0.24277812116861283" 05:25:26 DEBUG ] 05:25:26 DEBUG } 05:25:26 DEBUG ] 05:25:26 DEBUG } 05:25:26 DEBUG } 05:25:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723126" 05:25:26 INFO [http_cmd]: http status code OK 05:25:26 DEBUG --- status code --- 05:25:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:26 DEBUG --- http response --- 05:25:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:25:26 DEBUG ----- output ----- 05:25:26 DEBUG { 05:25:26 DEBUG "status": "success", 05:25:26 DEBUG "data": { 05:25:26 DEBUG "resultType": "vector", 05:25:26 DEBUG "result": [ 05:25:26 DEBUG { 05:25:26 DEBUG "metric": { 05:25:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:25:26 DEBUG }, 05:25:26 DEBUG "value": [ 05:25:26 DEBUG 05:25:26 DEBUG 05:25:26 DEBUG 05:25:26 DEBUG [...] 05:25:26 DEBUG 05:25:26 DEBUG 05:25:26 DEBUG 05:25:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:25:26 DEBUG }, 05:25:26 DEBUG "value": [ 05:25:26 DEBUG 1712723126, 05:25:26 DEBUG "1265.8666666666666" 05:25:26 DEBUG ] 05:25:26 DEBUG } 05:25:26 DEBUG ] 05:25:26 DEBUG } 05:25:26 DEBUG } 05:25:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712723126" 05:25:26 INFO [http_cmd]: http status code OK 05:25:26 DEBUG --- status code --- 05:25:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:26 DEBUG --- http response --- 05:25:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:25:26 DEBUG ----- output ----- 05:25:26 DEBUG { 05:25:26 DEBUG "status": "success", 05:25:26 DEBUG "data": { 05:25:26 DEBUG "resultType": "vector", 05:25:26 DEBUG "result": [ 05:25:26 DEBUG { 05:25:26 DEBUG "metric": { 05:25:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:25:26 DEBUG }, 05:25:26 DEBUG "value": [ 05:25:26 DEBUG 05:25:26 DEBUG 05:25:26 DEBUG 05:25:26 DEBUG [...] 05:25:26 DEBUG 05:25:26 DEBUG 05:25:26 DEBUG 05:25:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:25:26 DEBUG }, 05:25:26 DEBUG "value": [ 05:25:26 DEBUG 1712723126, 05:25:26 DEBUG "0.041" 05:25:26 DEBUG ] 05:25:26 DEBUG } 05:25:26 DEBUG ] 05:25:26 DEBUG } 05:25:26 DEBUG } 05:25:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723126" 05:25:26 INFO [http_cmd]: http status code OK 05:25:26 DEBUG --- status code --- 05:25:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:26 DEBUG --- http response --- 05:25:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:25:26 DEBUG ----- output ----- 05:25:26 DEBUG { 05:25:26 DEBUG "status": "success", 05:25:26 DEBUG "data": { 05:25:26 DEBUG "resultType": "vector", 05:25:26 DEBUG "result": [ 05:25:26 DEBUG { 05:25:26 DEBUG "metric": { 05:25:26 DEBUG "pod": "ds-idrepo-0" 05:25:26 DEBUG }, 05:25:26 DEBUG "value": [ 05:25:26 DEBUG 05:25:26 DEBUG 05:25:26 DEBUG 05:25:26 DEBUG [...] 05:25:26 DEBUG 05:25:26 DEBUG 05:25:26 DEBUG 05:25:26 DEBUG "pod": "ds-idrepo-1" 05:25:26 DEBUG }, 05:25:26 DEBUG "value": [ 05:25:26 DEBUG 1712723126, 05:25:26 DEBUG "0" 05:25:26 DEBUG ] 05:25:26 DEBUG } 05:25:26 DEBUG ] 05:25:26 DEBUG } 05:25:26 DEBUG } 05:25:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723127" 05:25:27 INFO [http_cmd]: http status code OK 05:25:27 DEBUG --- status code --- 05:25:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:27 DEBUG --- http response --- 05:25:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:25:27 DEBUG ----- output ----- 05:25:27 DEBUG { 05:25:27 DEBUG "status": "success", 05:25:27 DEBUG "data": { 05:25:27 DEBUG "resultType": "vector", 05:25:27 DEBUG "result": [ 05:25:27 DEBUG { 05:25:27 DEBUG "metric": { 05:25:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:25:27 DEBUG }, 05:25:27 DEBUG "value": [ 05:25:27 DEBUG 05:25:27 DEBUG 05:25:27 DEBUG 05:25:27 DEBUG [...] 05:25:27 DEBUG 05:25:27 DEBUG 05:25:27 DEBUG 05:25:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:25:27 DEBUG }, 05:25:27 DEBUG "value": [ 05:25:27 DEBUG 1712723127, 05:25:27 DEBUG "1250.2" 05:25:27 DEBUG ] 05:25:27 DEBUG } 05:25:27 DEBUG ] 05:25:27 DEBUG } 05:25:27 DEBUG } 05:25:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723127" 05:25:27 INFO [http_cmd]: http status code OK 05:25:27 DEBUG --- status code --- 05:25:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:27 DEBUG --- http response --- 05:25:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:25:27 DEBUG ----- output ----- 05:25:27 DEBUG { 05:25:27 DEBUG "status": "success", 05:25:27 DEBUG "data": { 05:25:27 DEBUG "resultType": "vector", 05:25:27 DEBUG "result": [ 05:25:27 DEBUG { 05:25:27 DEBUG "metric": { 05:25:27 DEBUG "pod": "ds-idrepo-0" 05:25:27 DEBUG }, 05:25:27 DEBUG "value": [ 05:25:27 DEBUG 05:25:27 DEBUG 05:25:27 DEBUG 05:25:27 DEBUG [...] 05:25:27 DEBUG 05:25:27 DEBUG 05:25:27 DEBUG 05:25:27 DEBUG "pod": "ds-idrepo-1" 05:25:27 DEBUG }, 05:25:27 DEBUG "value": [ 05:25:27 DEBUG 1712723127, 05:25:27 DEBUG "0" 05:25:27 DEBUG ] 05:25:27 DEBUG } 05:25:27 DEBUG ] 05:25:27 DEBUG } 05:25:27 DEBUG } 05:25:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712723127" 05:25:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723127" 05:25:27 INFO [http_cmd]: http status code OK 05:25:27 DEBUG --- status code --- 05:25:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:27 DEBUG --- http response --- 05:25:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:25:27 INFO [http_cmd]: http status code OK 05:25:27 DEBUG ----- output ----- 05:25:27 DEBUG --- status code --- 05:25:27 DEBUG { 05:25:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:27 DEBUG "status": "success", 05:25:27 DEBUG --- http response --- 05:25:27 DEBUG "data": { 05:25:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:25:27 DEBUG "resultType": "vector", 05:25:27 DEBUG ----- output ----- 05:25:27 DEBUG "result": [ 05:25:27 DEBUG { 05:25:27 DEBUG { 05:25:27 DEBUG "status": "success", 05:25:27 DEBUG "metric": { 05:25:27 DEBUG "data": { 05:25:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:25:27 DEBUG "resultType": "vector", 05:25:27 DEBUG }, 05:25:27 DEBUG "result": [ 05:25:27 DEBUG "value": [ 05:25:27 DEBUG { 05:25:27 DEBUG 05:25:27 DEBUG "metric": { 05:25:27 DEBUG 05:25:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:25:27 DEBUG 05:25:27 DEBUG }, 05:25:27 DEBUG [...] 05:25:27 DEBUG "value": [ 05:25:27 DEBUG 05:25:27 DEBUG 05:25:27 DEBUG 05:25:27 DEBUG 05:25:27 DEBUG 05:25:27 DEBUG 05:25:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:25:27 DEBUG [...] 05:25:27 DEBUG }, 05:25:27 DEBUG 05:25:27 DEBUG "value": [ 05:25:27 DEBUG 05:25:27 DEBUG 1712723127, 05:25:27 DEBUG 05:25:27 DEBUG "2486632448" 05:25:27 DEBUG "pod": "ds-idrepo-2" 05:25:27 DEBUG ] 05:25:27 DEBUG }, 05:25:27 DEBUG } 05:25:27 DEBUG "value": [ 05:25:27 DEBUG ] 05:25:27 DEBUG 1712723127, 05:25:27 DEBUG } 05:25:27 DEBUG "2765.3277485855415" 05:25:27 DEBUG } 05:25:27 DEBUG ] 05:25:27 DEBUG } 05:25:27 DEBUG ] 05:25:27 DEBUG } 05:25:27 DEBUG } 05:25:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723127" 05:25:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723127" 05:25:27 INFO [http_cmd]: http status code OK 05:25:27 DEBUG --- status code --- 05:25:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:27 DEBUG --- http response --- 05:25:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:25:27 DEBUG ----- output ----- 05:25:27 DEBUG { 05:25:27 DEBUG "status": "success", 05:25:27 DEBUG "data": { 05:25:27 DEBUG "resultType": "vector", 05:25:27 DEBUG "result": [ 05:25:27 DEBUG { 05:25:27 DEBUG "metric": { 05:25:27 DEBUG "instance": "10.142.0.25:9100" 05:25:27 DEBUG }, 05:25:27 DEBUG "value": [ 05:25:27 DEBUG 05:25:27 DEBUG 05:25:27 DEBUG 05:25:27 DEBUG [...] 05:25:27 DEBUG 05:25:27 DEBUG 05:25:27 DEBUG 05:25:27 DEBUG "instance": "10.142.0.77:9100" 05:25:27 DEBUG }, 05:25:27 DEBUG "value": [ 05:25:27 DEBUG 1712723127, 05:25:27 DEBUG "37685215.8" 05:25:27 DEBUG ] 05:25:27 DEBUG } 05:25:27 DEBUG ] 05:25:27 DEBUG } 05:25:27 DEBUG } 05:25:27 INFO [http_cmd]: http status code OK 05:25:27 DEBUG --- status code --- 05:25:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:25:27 DEBUG --- http response --- 05:25:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:25:27 DEBUG ----- output ----- 05:25:27 DEBUG { 05:25:27 DEBUG "status": "success", 05:25:27 DEBUG "data": { 05:25:27 DEBUG "resultType": "vector", 05:25:27 DEBUG "result": [ 05:25:27 DEBUG { 05:25:27 DEBUG "metric": { 05:25:27 DEBUG "instance": "10.142.0.25:9100" 05:25:27 DEBUG }, 05:25:27 DEBUG "value": [ 05:25:27 DEBUG 05:25:27 DEBUG 05:25:27 DEBUG 05:25:27 DEBUG [...] 05:25:27 DEBUG 05:25:27 DEBUG 05:25:27 DEBUG 05:25:27 DEBUG "instance": "10.142.0.77:9100" 05:25:27 DEBUG }, 05:25:27 DEBUG "value": [ 05:25:27 DEBUG 1712723127, 05:25:27 DEBUG "0.0386666666666662" 05:25:27 DEBUG ] 05:25:27 DEBUG } 05:25:27 DEBUG ] 05:25:27 DEBUG } 05:25:27 DEBUG } 05:25:41 INFO 05:25:41 INFO [loop_until]: kubectl --namespace=xlou top pods 05:25:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:25:41 INFO [loop_until]: OK (rc = 0) 05:25:41 DEBUG --- stdout --- 05:25:41 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 9057m 8804Mi am-654c4fcbcf-9scrj 8851m 8421Mi am-654c4fcbcf-v5qb5 7391m 8401Mi ds-cts-0 7454m 13201Mi ds-cts-1 7266m 13319Mi ds-cts-2 6312m 12380Mi ds-idrepo-0 9437m 5621Mi ds-idrepo-1 323m 5668Mi ds-idrepo-2 338m 5698Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 5m 2213Mi lodemon-b776b7d88-thrqf 2m 70Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3561m 1659Mi 05:25:41 DEBUG --- stderr --- 05:25:41 DEBUG 05:25:48 INFO 05:25:48 INFO [loop_until]: kubectl --namespace=xlou top node 05:25:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:25:48 INFO [loop_until]: OK (rc = 0) 05:25:48 DEBUG --- stdout --- 05:25:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 94m 0% 2303Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8083m 50% 9281Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 82m 0% 3258Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10342m 65% 6498Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9523m 59% 9313Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9887m 62% 9648Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 75m 0% 857Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9682m 60% 6281Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 7194m 45% 13828Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 6657m 41% 13102Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 390m 2% 6362Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7474m 47% 13901Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 373m 2% 6327Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3685m 23% 5342Mi 9% 05:25:48 DEBUG --- stderr --- 05:25:48 DEBUG 05:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723183" 05:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712723183" 05:26:23 INFO [http_cmd]: http status code OK 05:26:23 DEBUG --- status code --- 05:26:23 ERROR [http_cmd]: ERROR 05:26:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:23 ERROR --- status code --- 05:26:23 DEBUG --- http response --- 05:26:23 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:26:23 ERROR --- http response --- 05:26:23 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:26:23 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:26:23 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723183" 05:26:23 INFO [http_cmd]: http status code OK 05:26:23 DEBUG --- status code --- 05:26:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:23 DEBUG --- http response --- 05:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:26:23 DEBUG ----- output ----- 05:26:23 DEBUG { 05:26:23 DEBUG "status": "success", 05:26:23 DEBUG "data": { 05:26:23 DEBUG "resultType": "vector", 05:26:23 DEBUG "result": [ 05:26:23 DEBUG { 05:26:23 DEBUG "metric": { 05:26:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:26:23 DEBUG }, 05:26:23 DEBUG "value": [ 05:26:23 DEBUG 05:26:23 DEBUG 05:26:23 DEBUG 05:26:23 DEBUG [...] 05:26:23 DEBUG 05:26:23 DEBUG 05:26:23 DEBUG 05:26:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:26:23 DEBUG }, 05:26:23 DEBUG "value": [ 05:26:23 DEBUG 1712723183, 05:26:23 DEBUG "0" 05:26:23 DEBUG ] 05:26:23 DEBUG } 05:26:23 DEBUG ] 05:26:23 DEBUG } 05:26:23 DEBUG } 05:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723183" 05:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723183" 05:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723183" 05:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723183" 05:26:23 INFO [http_cmd]: http status code OK 05:26:23 DEBUG --- status code --- 05:26:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:23 DEBUG --- http response --- 05:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:26:23 DEBUG ----- output ----- 05:26:23 DEBUG { 05:26:23 DEBUG "status": "success", 05:26:23 DEBUG "data": { 05:26:23 DEBUG "resultType": "vector", 05:26:23 DEBUG "result": [ 05:26:23 DEBUG { 05:26:23 DEBUG "metric": { 05:26:23 DEBUG "pod": "ds-idrepo-0" 05:26:23 DEBUG }, 05:26:23 DEBUG "value": [ 05:26:23 DEBUG 05:26:23 DEBUG 05:26:23 DEBUG 05:26:23 DEBUG [...] 05:26:23 DEBUG 05:26:23 DEBUG 05:26:23 DEBUG 05:26:23 DEBUG "pod": "ds-cts-2" 05:26:23 DEBUG }, 05:26:23 DEBUG "value": [ 05:26:23 DEBUG 1712723183, 05:26:23 DEBUG "29.933333333333334" 05:26:23 DEBUG ] 05:26:23 DEBUG } 05:26:23 DEBUG ] 05:26:23 DEBUG } 05:26:23 DEBUG } 05:26:23 INFO [http_cmd]: http status code OK 05:26:23 DEBUG --- status code --- 05:26:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:23 DEBUG --- http response --- 05:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:26:23 DEBUG ----- output ----- 05:26:23 DEBUG { 05:26:23 DEBUG "status": "success", 05:26:23 DEBUG "data": { 05:26:23 DEBUG "resultType": "vector", 05:26:23 DEBUG "result": [ 05:26:23 DEBUG { 05:26:23 DEBUG "metric": { 05:26:23 DEBUG "pod": "ds-idrepo-0" 05:26:23 DEBUG }, 05:26:23 DEBUG "value": [ 05:26:23 DEBUG 05:26:23 DEBUG 05:26:23 DEBUG 05:26:23 DEBUG [...] 05:26:23 DEBUG 05:26:23 DEBUG 05:26:23 DEBUG 05:26:23 DEBUG "pod": "ds-cts-2" 05:26:23 DEBUG }, 05:26:23 DEBUG "value": [ 05:26:23 DEBUG 1712723183, 05:26:23 DEBUG "0" 05:26:23 DEBUG ] 05:26:23 DEBUG } 05:26:23 DEBUG ] 05:26:23 DEBUG } 05:26:23 DEBUG } 05:26:23 INFO [http_cmd]: http status code OK 05:26:23 DEBUG --- status code --- 05:26:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:23 DEBUG --- http response --- 05:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:26:23 DEBUG ----- output ----- 05:26:23 DEBUG { 05:26:23 DEBUG "status": "success", 05:26:23 DEBUG "data": { 05:26:23 DEBUG "resultType": "vector", 05:26:23 DEBUG "result": [ 05:26:23 DEBUG { 05:26:23 DEBUG "metric": { 05:26:23 DEBUG "instance": "10.142.0.25:9100" 05:26:23 DEBUG }, 05:26:23 DEBUG "value": [ 05:26:23 DEBUG 05:26:23 DEBUG 05:26:23 DEBUG 05:26:23 DEBUG [...] 05:26:23 DEBUG 05:26:23 DEBUG 05:26:23 DEBUG 05:26:23 INFO [http_cmd]: http status code OK 05:26:23 DEBUG "instance": "10.142.0.77:9100" 05:26:23 DEBUG --- status code --- 05:26:23 DEBUG }, 05:26:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:23 DEBUG "value": [ 05:26:23 DEBUG --- http response --- 05:26:23 DEBUG 1712723183, 05:26:23 DEBUG "281323801600" 05:26:23 DEBUG ] 05:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:26:23 DEBUG } 05:26:23 DEBUG ----- output ----- 05:26:23 DEBUG ] 05:26:23 DEBUG { 05:26:23 DEBUG } 05:26:23 DEBUG "status": "success", 05:26:23 DEBUG } 05:26:23 DEBUG "data": { 05:26:23 DEBUG "resultType": "vector", 05:26:23 DEBUG "result": [ 05:26:23 DEBUG { 05:26:23 DEBUG "metric": { 05:26:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:26:23 DEBUG }, 05:26:23 DEBUG "value": [ 05:26:23 DEBUG 05:26:23 DEBUG 05:26:23 DEBUG 05:26:23 DEBUG [...] 05:26:23 DEBUG 05:26:23 DEBUG 05:26:23 DEBUG 05:26:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:26:23 DEBUG }, 05:26:23 DEBUG "value": [ 05:26:23 DEBUG 1712723183, 05:26:23 DEBUG "1248.0333333333333" 05:26:23 DEBUG ] 05:26:23 DEBUG } 05:26:23 DEBUG ] 05:26:23 DEBUG } 05:26:23 DEBUG } 05:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723183" 05:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723183" 05:26:23 INFO [http_cmd]: http status code OK 05:26:23 DEBUG --- status code --- 05:26:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:23 DEBUG --- http response --- 05:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:26:23 INFO [http_cmd]: http status code OK 05:26:23 DEBUG ----- output ----- 05:26:23 DEBUG --- status code --- 05:26:23 DEBUG { 05:26:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:23 DEBUG "status": "success", 05:26:23 DEBUG --- http response --- 05:26:23 DEBUG "data": { 05:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:26:23 DEBUG "resultType": "vector", 05:26:23 DEBUG ----- output ----- 05:26:23 DEBUG "result": [ 05:26:23 DEBUG { 05:26:23 DEBUG { 05:26:23 DEBUG "status": "success", 05:26:23 DEBUG "data": { 05:26:23 DEBUG "metric": { 05:26:23 DEBUG "resultType": "vector", 05:26:23 DEBUG "instance": "10.142.0.25:9100" 05:26:23 DEBUG "result": [ 05:26:23 DEBUG }, 05:26:23 DEBUG { 05:26:23 DEBUG "value": [ 05:26:23 DEBUG "metric": { 05:26:23 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:26:23 DEBUG 05:26:23 DEBUG }, 05:26:23 DEBUG 05:26:23 DEBUG "value": [ 05:26:23 DEBUG 05:26:23 DEBUG 05:26:23 DEBUG [...] 05:26:23 DEBUG 05:26:23 DEBUG 05:26:23 DEBUG 05:26:23 DEBUG 05:26:23 DEBUG 05:26:23 DEBUG [...] 05:26:23 DEBUG 05:26:23 DEBUG 05:26:23 DEBUG "instance": "10.142.0.77:9100" 05:26:23 DEBUG 05:26:23 DEBUG }, 05:26:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:26:23 DEBUG "value": [ 05:26:23 DEBUG 1712723183, 05:26:23 DEBUG "37504590.63333333" 05:26:23 DEBUG }, 05:26:23 DEBUG ] 05:26:23 DEBUG "value": [ 05:26:23 DEBUG } 05:26:23 DEBUG 1712723183, 05:26:23 DEBUG ] 05:26:23 DEBUG "0" 05:26:23 DEBUG } 05:26:23 DEBUG ] 05:26:23 DEBUG } 05:26:23 DEBUG } 05:26:23 DEBUG ] 05:26:23 DEBUG } 05:26:23 DEBUG } 05:26:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723184" 05:26:24 INFO [http_cmd]: http status code OK 05:26:24 DEBUG --- status code --- 05:26:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:24 DEBUG --- http response --- 05:26:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:26:24 DEBUG ----- output ----- 05:26:24 DEBUG { 05:26:24 DEBUG "status": "success", 05:26:24 DEBUG "data": { 05:26:24 DEBUG "resultType": "vector", 05:26:24 DEBUG "result": [ 05:26:24 DEBUG { 05:26:24 DEBUG "metric": { 05:26:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:26:24 DEBUG }, 05:26:24 DEBUG "value": [ 05:26:24 DEBUG 05:26:24 DEBUG 05:26:24 DEBUG 05:26:24 DEBUG [...] 05:26:24 DEBUG 05:26:24 DEBUG 05:26:24 DEBUG 05:26:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:26:24 DEBUG }, 05:26:24 DEBUG "value": [ 05:26:24 DEBUG 1712723184, 05:26:24 DEBUG "226244" 05:26:24 DEBUG ] 05:26:24 DEBUG } 05:26:24 DEBUG ] 05:26:24 DEBUG } 05:26:24 DEBUG } 05:26:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723184" 05:26:24 INFO [http_cmd]: http status code OK 05:26:24 DEBUG --- status code --- 05:26:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:24 DEBUG --- http response --- 05:26:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:26:24 DEBUG ----- output ----- 05:26:24 DEBUG { 05:26:24 DEBUG "status": "success", 05:26:24 DEBUG "data": { 05:26:24 DEBUG "resultType": "vector", 05:26:24 DEBUG "result": [ 05:26:24 DEBUG { 05:26:24 DEBUG "metric": { 05:26:24 DEBUG "pod": "ds-idrepo-0" 05:26:24 DEBUG }, 05:26:24 DEBUG "value": [ 05:26:24 DEBUG 05:26:24 DEBUG 05:26:24 DEBUG 05:26:24 DEBUG [...] 05:26:24 DEBUG 05:26:24 DEBUG 05:26:24 DEBUG 05:26:24 DEBUG "pod": "ds-idrepo-1" 05:26:24 DEBUG }, 05:26:24 DEBUG "value": [ 05:26:24 DEBUG 1712723184, 05:26:24 DEBUG "0" 05:26:24 DEBUG ] 05:26:24 DEBUG } 05:26:24 DEBUG ] 05:26:24 DEBUG } 05:26:24 DEBUG } 05:26:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723184" 05:26:24 INFO [http_cmd]: http status code OK 05:26:24 DEBUG --- status code --- 05:26:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:24 DEBUG --- http response --- 05:26:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:26:24 DEBUG ----- output ----- 05:26:24 DEBUG { 05:26:24 DEBUG "status": "success", 05:26:24 DEBUG "data": { 05:26:24 DEBUG "resultType": "vector", 05:26:24 DEBUG "result": [ 05:26:24 DEBUG { 05:26:24 DEBUG "metric": { 05:26:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:26:24 DEBUG }, 05:26:24 DEBUG "value": [ 05:26:24 DEBUG 05:26:24 DEBUG 05:26:24 DEBUG 05:26:24 DEBUG [...] 05:26:24 DEBUG 05:26:24 DEBUG 05:26:24 DEBUG 05:26:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:26:24 DEBUG }, 05:26:24 DEBUG "value": [ 05:26:24 DEBUG 1712723184, 05:26:24 DEBUG "1244.1" 05:26:24 DEBUG ] 05:26:24 DEBUG } 05:26:24 DEBUG ] 05:26:24 DEBUG } 05:26:24 DEBUG } 05:26:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723185" 05:26:25 INFO [http_cmd]: http status code OK 05:26:25 DEBUG --- status code --- 05:26:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:25 DEBUG --- http response --- 05:26:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:26:25 DEBUG ----- output ----- 05:26:25 DEBUG { 05:26:25 DEBUG "status": "success", 05:26:25 DEBUG "data": { 05:26:25 DEBUG "resultType": "vector", 05:26:25 DEBUG "result": [ 05:26:25 DEBUG { 05:26:25 DEBUG "metric": { 05:26:25 DEBUG "instance": "10.142.0.25:9100" 05:26:25 DEBUG }, 05:26:25 DEBUG "value": [ 05:26:25 DEBUG 05:26:25 DEBUG 05:26:25 DEBUG 05:26:25 DEBUG [...] 05:26:25 DEBUG 05:26:25 DEBUG 05:26:25 DEBUG 05:26:25 DEBUG "instance": "10.142.0.77:9100" 05:26:25 DEBUG }, 05:26:25 DEBUG "value": [ 05:26:25 DEBUG 1712723185, 05:26:25 DEBUG "0.14783333333333293" 05:26:25 DEBUG ] 05:26:25 DEBUG } 05:26:25 DEBUG ] 05:26:25 DEBUG } 05:26:25 DEBUG } 05:26:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723185" 05:26:25 INFO [http_cmd]: http status code OK 05:26:25 DEBUG --- status code --- 05:26:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:25 DEBUG --- http response --- 05:26:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:26:25 DEBUG ----- output ----- 05:26:25 DEBUG { 05:26:25 DEBUG "status": "success", 05:26:25 DEBUG "data": { 05:26:25 DEBUG "resultType": "vector", 05:26:25 DEBUG "result": [ 05:26:25 DEBUG { 05:26:25 DEBUG "metric": { 05:26:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:26:25 DEBUG }, 05:26:25 DEBUG "value": [ 05:26:25 DEBUG 05:26:25 DEBUG 05:26:25 DEBUG 05:26:25 DEBUG [...] 05:26:25 DEBUG 05:26:25 DEBUG 05:26:25 DEBUG 05:26:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:26:25 DEBUG }, 05:26:25 DEBUG "value": [ 05:26:25 DEBUG 1712723185, 05:26:25 DEBUG "74180.55157957778" 05:26:25 DEBUG ] 05:26:25 DEBUG } 05:26:25 DEBUG ] 05:26:25 DEBUG } 05:26:25 DEBUG } 05:26:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712723185" 05:26:25 INFO [http_cmd]: http status code OK 05:26:25 DEBUG --- status code --- 05:26:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:25 DEBUG --- http response --- 05:26:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:26:25 DEBUG ----- output ----- 05:26:25 DEBUG { 05:26:25 DEBUG "status": "success", 05:26:25 DEBUG "data": { 05:26:25 DEBUG "resultType": "vector", 05:26:25 DEBUG "result": [ 05:26:25 DEBUG { 05:26:25 DEBUG "metric": { 05:26:25 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:26:25 DEBUG }, 05:26:25 DEBUG "value": [ 05:26:25 DEBUG 05:26:25 DEBUG 05:26:25 DEBUG 05:26:25 DEBUG [...] 05:26:25 DEBUG 05:26:25 DEBUG 05:26:25 DEBUG 05:26:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:26:25 DEBUG }, 05:26:25 DEBUG "value": [ 05:26:25 DEBUG 1712723185, 05:26:25 DEBUG "3.600654505395647" 05:26:25 DEBUG ] 05:26:25 DEBUG } 05:26:25 DEBUG ] 05:26:25 DEBUG } 05:26:25 DEBUG } 05:26:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712723186" 05:26:26 INFO [http_cmd]: http status code OK 05:26:26 DEBUG --- status code --- 05:26:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:26 DEBUG --- http response --- 05:26:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:26:26 DEBUG ----- output ----- 05:26:26 DEBUG { 05:26:26 DEBUG "status": "success", 05:26:26 DEBUG "data": { 05:26:26 DEBUG "resultType": "vector", 05:26:26 DEBUG "result": [ 05:26:26 DEBUG { 05:26:26 DEBUG "metric": { 05:26:26 DEBUG "pod": "ds-idrepo-0" 05:26:26 DEBUG }, 05:26:26 DEBUG "value": [ 05:26:26 DEBUG 05:26:26 DEBUG 05:26:26 DEBUG 05:26:26 DEBUG [...] 05:26:26 DEBUG 05:26:26 DEBUG 05:26:26 DEBUG 05:26:26 DEBUG "pod": "ds-cts-2" 05:26:26 DEBUG }, 05:26:26 DEBUG "value": [ 05:26:26 DEBUG 1712723186, 05:26:26 DEBUG "0" 05:26:26 DEBUG ] 05:26:26 DEBUG } 05:26:26 DEBUG ] 05:26:26 DEBUG } 05:26:26 DEBUG } 05:26:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723186" 05:26:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723186" 05:26:26 INFO [http_cmd]: http status code OK 05:26:26 DEBUG --- status code --- 05:26:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:26 DEBUG --- http response --- 05:26:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:26:26 DEBUG ----- output ----- 05:26:26 DEBUG { 05:26:26 DEBUG "status": "success", 05:26:26 DEBUG "data": { 05:26:26 DEBUG "resultType": "vector", 05:26:26 DEBUG "result": [ 05:26:26 DEBUG { 05:26:26 DEBUG "metric": { 05:26:26 DEBUG "pod": "ds-idrepo-2" 05:26:26 DEBUG }, 05:26:26 DEBUG "value": [ 05:26:26 DEBUG 05:26:26 DEBUG 05:26:26 DEBUG 05:26:26 DEBUG [...] 05:26:26 DEBUG 05:26:26 DEBUG 05:26:26 DEBUG 05:26:26 DEBUG "pod": "ds-cts-2" 05:26:26 DEBUG }, 05:26:26 DEBUG "value": [ 05:26:26 DEBUG 1712723186, 05:26:26 DEBUG "0.000033333333333333335" 05:26:26 DEBUG ] 05:26:26 DEBUG } 05:26:26 DEBUG ] 05:26:26 DEBUG } 05:26:26 DEBUG } 05:26:26 INFO [http_cmd]: http status code OK 05:26:26 DEBUG --- status code --- 05:26:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:26 DEBUG --- http response --- 05:26:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:26:26 DEBUG ----- output ----- 05:26:26 DEBUG { 05:26:26 DEBUG "status": "success", 05:26:26 DEBUG "data": { 05:26:26 DEBUG "resultType": "vector", 05:26:26 DEBUG "result": [ 05:26:26 DEBUG { 05:26:26 DEBUG "metric": { 05:26:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:26:26 DEBUG }, 05:26:26 DEBUG "value": [ 05:26:26 DEBUG 05:26:26 DEBUG 05:26:26 DEBUG 05:26:26 DEBUG [...] 05:26:26 DEBUG 05:26:26 DEBUG 05:26:26 DEBUG 05:26:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:26:26 DEBUG }, 05:26:26 DEBUG "value": [ 05:26:26 DEBUG 1712723186, 05:26:26 DEBUG "7.396352300751252" 05:26:26 DEBUG ] 05:26:26 DEBUG } 05:26:26 DEBUG ] 05:26:26 DEBUG } 05:26:26 DEBUG } 05:26:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723186" 05:26:26 INFO [http_cmd]: http status code OK 05:26:26 DEBUG --- status code --- 05:26:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:26 DEBUG --- http response --- 05:26:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:26:26 DEBUG ----- output ----- 05:26:26 DEBUG { 05:26:26 DEBUG "status": "success", 05:26:26 DEBUG "data": { 05:26:26 DEBUG "resultType": "vector", 05:26:26 DEBUG "result": [ 05:26:26 DEBUG { 05:26:26 DEBUG "metric": { 05:26:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:26:26 DEBUG }, 05:26:26 DEBUG "value": [ 05:26:26 DEBUG 05:26:26 DEBUG 05:26:26 DEBUG 05:26:26 DEBUG [...] 05:26:26 DEBUG 05:26:26 DEBUG 05:26:26 DEBUG 05:26:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:26:26 DEBUG }, 05:26:26 DEBUG "value": [ 05:26:26 DEBUG 1712723186, 05:26:26 DEBUG "1244.1666666666667" 05:26:26 DEBUG ] 05:26:26 DEBUG } 05:26:26 DEBUG ] 05:26:26 DEBUG } 05:26:26 DEBUG } 05:26:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712723186" 05:26:26 INFO [http_cmd]: http status code OK 05:26:26 DEBUG --- status code --- 05:26:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:26 DEBUG --- http response --- 05:26:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:26:26 DEBUG ----- output ----- 05:26:26 DEBUG { 05:26:26 DEBUG "status": "success", 05:26:26 DEBUG "data": { 05:26:26 DEBUG "resultType": "vector", 05:26:26 DEBUG "result": [ 05:26:26 DEBUG { 05:26:26 DEBUG "metric": { 05:26:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:26:26 DEBUG }, 05:26:26 DEBUG "value": [ 05:26:26 DEBUG 05:26:26 DEBUG 05:26:26 DEBUG 05:26:26 DEBUG [...] 05:26:26 DEBUG 05:26:26 DEBUG 05:26:26 DEBUG 05:26:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:26:26 DEBUG }, 05:26:26 DEBUG "value": [ 05:26:26 DEBUG 1712723186, 05:26:26 DEBUG "0.042" 05:26:26 DEBUG ] 05:26:26 DEBUG } 05:26:26 DEBUG ] 05:26:26 DEBUG } 05:26:26 DEBUG } 05:26:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723186" 05:26:26 INFO [http_cmd]: http status code OK 05:26:26 DEBUG --- status code --- 05:26:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:26 DEBUG --- http response --- 05:26:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:26:26 DEBUG ----- output ----- 05:26:26 DEBUG { 05:26:26 DEBUG "status": "success", 05:26:26 DEBUG "data": { 05:26:26 DEBUG "resultType": "vector", 05:26:26 DEBUG "result": [ 05:26:26 DEBUG { 05:26:26 DEBUG "metric": { 05:26:26 DEBUG "pod": "ds-idrepo-0" 05:26:26 DEBUG }, 05:26:26 DEBUG "value": [ 05:26:26 DEBUG 05:26:26 DEBUG 05:26:26 DEBUG 05:26:26 DEBUG [...] 05:26:26 DEBUG 05:26:26 DEBUG 05:26:26 DEBUG 05:26:26 DEBUG "pod": "ds-idrepo-1" 05:26:26 DEBUG }, 05:26:26 DEBUG "value": [ 05:26:26 DEBUG 1712723186, 05:26:26 DEBUG "0" 05:26:26 DEBUG ] 05:26:26 DEBUG } 05:26:26 DEBUG ] 05:26:26 DEBUG } 05:26:26 DEBUG } 05:26:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723187" 05:26:27 INFO [http_cmd]: http status code OK 05:26:27 DEBUG --- status code --- 05:26:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:27 DEBUG --- http response --- 05:26:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:26:27 DEBUG ----- output ----- 05:26:27 DEBUG { 05:26:27 DEBUG "status": "success", 05:26:27 DEBUG "data": { 05:26:27 DEBUG "resultType": "vector", 05:26:27 DEBUG "result": [ 05:26:27 DEBUG { 05:26:27 DEBUG "metric": { 05:26:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:26:27 DEBUG }, 05:26:27 DEBUG "value": [ 05:26:27 DEBUG 05:26:27 DEBUG 05:26:27 DEBUG 05:26:27 DEBUG [...] 05:26:27 DEBUG 05:26:27 DEBUG 05:26:27 DEBUG 05:26:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:26:27 DEBUG }, 05:26:27 DEBUG "value": [ 05:26:27 DEBUG 1712723187, 05:26:27 DEBUG "1261.5" 05:26:27 DEBUG ] 05:26:27 DEBUG } 05:26:27 DEBUG ] 05:26:27 DEBUG } 05:26:27 DEBUG } 05:26:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723187" 05:26:27 INFO [http_cmd]: http status code OK 05:26:27 DEBUG --- status code --- 05:26:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:27 DEBUG --- http response --- 05:26:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:26:27 DEBUG ----- output ----- 05:26:27 DEBUG { 05:26:27 DEBUG "status": "success", 05:26:27 DEBUG "data": { 05:26:27 DEBUG "resultType": "vector", 05:26:27 DEBUG "result": [ 05:26:27 DEBUG { 05:26:27 DEBUG "metric": { 05:26:27 DEBUG "pod": "ds-idrepo-0" 05:26:27 DEBUG }, 05:26:27 DEBUG "value": [ 05:26:27 DEBUG 05:26:27 DEBUG 05:26:27 DEBUG 05:26:27 DEBUG [...] 05:26:27 DEBUG 05:26:27 DEBUG 05:26:27 DEBUG 05:26:27 DEBUG "pod": "ds-idrepo-1" 05:26:27 DEBUG }, 05:26:27 DEBUG "value": [ 05:26:27 DEBUG 1712723187, 05:26:27 DEBUG "0" 05:26:27 DEBUG ] 05:26:27 DEBUG } 05:26:27 DEBUG ] 05:26:27 DEBUG } 05:26:27 DEBUG } 05:26:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723187" 05:26:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712723187" 05:26:27 INFO [http_cmd]: http status code OK 05:26:27 DEBUG --- status code --- 05:26:27 INFO [http_cmd]: http status code OK 05:26:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:27 DEBUG --- status code --- 05:26:27 DEBUG --- http response --- 05:26:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:27 DEBUG --- http response --- 05:26:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:26:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:26:27 DEBUG ----- output ----- 05:26:27 DEBUG ----- output ----- 05:26:27 DEBUG { 05:26:27 DEBUG { 05:26:27 DEBUG "status": "success", 05:26:27 DEBUG "status": "success", 05:26:27 DEBUG "data": { 05:26:27 DEBUG "data": { 05:26:27 DEBUG "resultType": "vector", 05:26:27 DEBUG "result": [ 05:26:27 DEBUG "resultType": "vector", 05:26:27 DEBUG { 05:26:27 DEBUG "result": [ 05:26:27 DEBUG "metric": { 05:26:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:26:27 DEBUG { 05:26:27 DEBUG }, 05:26:27 DEBUG "metric": { 05:26:27 DEBUG "value": [ 05:26:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:26:27 DEBUG 05:26:27 DEBUG }, 05:26:27 DEBUG 05:26:27 DEBUG "value": [ 05:26:27 DEBUG 05:26:27 DEBUG 05:26:27 DEBUG [...] 05:26:27 DEBUG 05:26:27 DEBUG 05:26:27 DEBUG 05:26:27 DEBUG 05:26:27 DEBUG [...] 05:26:27 DEBUG 05:26:27 DEBUG 05:26:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:26:27 DEBUG 05:26:27 DEBUG }, 05:26:27 DEBUG 05:26:27 DEBUG "value": [ 05:26:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:26:27 DEBUG 1712723187, 05:26:27 DEBUG }, 05:26:27 DEBUG "0" 05:26:27 DEBUG "value": [ 05:26:27 DEBUG ] 05:26:27 DEBUG 1712723187, 05:26:27 DEBUG } 05:26:27 DEBUG "2487619584" 05:26:27 DEBUG ] 05:26:27 DEBUG ] 05:26:27 DEBUG } 05:26:27 DEBUG } 05:26:27 DEBUG } 05:26:27 DEBUG ] 05:26:27 DEBUG } 05:26:27 DEBUG } 05:26:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723187" 05:26:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723187" 05:26:27 INFO [http_cmd]: http status code OK 05:26:27 DEBUG --- status code --- 05:26:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:27 DEBUG --- http response --- 05:26:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:26:27 DEBUG ----- output ----- 05:26:27 DEBUG { 05:26:27 DEBUG "status": "success", 05:26:27 DEBUG "data": { 05:26:27 DEBUG "resultType": "vector", 05:26:27 DEBUG "result": [ 05:26:27 DEBUG { 05:26:27 DEBUG "metric": { 05:26:27 DEBUG "instance": "10.142.0.25:9100" 05:26:27 DEBUG }, 05:26:27 DEBUG "value": [ 05:26:27 DEBUG 05:26:27 DEBUG 05:26:27 DEBUG 05:26:27 DEBUG [...] 05:26:27 DEBUG 05:26:27 DEBUG 05:26:27 DEBUG 05:26:27 DEBUG "instance": "10.142.0.77:9100" 05:26:27 DEBUG }, 05:26:27 DEBUG "value": [ 05:26:27 DEBUG 1712723187, 05:26:27 DEBUG "37504063.7" 05:26:27 DEBUG ] 05:26:27 DEBUG } 05:26:27 DEBUG ] 05:26:27 DEBUG } 05:26:27 DEBUG } 05:26:27 INFO [http_cmd]: http status code OK 05:26:27 DEBUG --- status code --- 05:26:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:26:27 DEBUG --- http response --- 05:26:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:26:27 DEBUG ----- output ----- 05:26:27 DEBUG { 05:26:27 DEBUG "status": "success", 05:26:27 DEBUG "data": { 05:26:27 DEBUG "resultType": "vector", 05:26:27 DEBUG "result": [ 05:26:27 DEBUG { 05:26:27 DEBUG "metric": { 05:26:27 DEBUG "instance": "10.142.0.25:9100" 05:26:27 DEBUG }, 05:26:27 DEBUG "value": [ 05:26:27 DEBUG 05:26:27 DEBUG 05:26:27 DEBUG 05:26:27 DEBUG [...] 05:26:27 DEBUG 05:26:27 DEBUG 05:26:27 DEBUG 05:26:27 DEBUG "instance": "10.142.0.77:9100" 05:26:27 DEBUG }, 05:26:27 DEBUG "value": [ 05:26:27 DEBUG 1712723187, 05:26:27 DEBUG "0.05433333333333295" 05:26:27 DEBUG ] 05:26:27 DEBUG } 05:26:27 DEBUG ] 05:26:27 DEBUG } 05:26:27 DEBUG } 05:26:41 INFO 05:26:41 INFO [loop_until]: kubectl --namespace=xlou top pods 05:26:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:26:41 INFO [loop_until]: OK (rc = 0) 05:26:41 DEBUG --- stdout --- 05:26:41 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 8815m 8804Mi am-654c4fcbcf-9scrj 8473m 8421Mi am-654c4fcbcf-v5qb5 7360m 8398Mi ds-cts-0 7384m 12378Mi ds-cts-1 7232m 12398Mi ds-cts-2 6615m 13090Mi ds-idrepo-0 9582m 5622Mi ds-idrepo-1 315m 5669Mi ds-idrepo-2 327m 5698Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 5m 2213Mi lodemon-b776b7d88-thrqf 6m 70Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3521m 1660Mi 05:26:41 DEBUG --- stderr --- 05:26:41 DEBUG 05:26:48 INFO 05:26:48 INFO [loop_until]: kubectl --namespace=xlou top node 05:26:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:26:48 INFO [loop_until]: OK (rc = 0) 05:26:48 DEBUG --- stdout --- 05:26:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 93m 0% 2304Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8139m 51% 9279Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 81m 0% 3258Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10761m 67% 6525Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9177m 57% 9313Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9874m 62% 9647Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 80m 0% 857Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9750m 61% 6282Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 7234m 45% 13937Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 6711m 42% 13734Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 380m 2% 6365Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7225m 45% 13424Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 357m 2% 6330Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3692m 23% 5341Mi 9% 05:26:48 DEBUG --- stderr --- 05:26:48 DEBUG 05:27:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712723243" 05:27:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723243" 05:27:23 ERROR [http_cmd]: ERROR 05:27:23 INFO [http_cmd]: http status code OK 05:27:23 ERROR --- status code --- 05:27:23 DEBUG --- status code --- 05:27:23 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:27:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:23 ERROR --- http response --- 05:27:23 DEBUG --- http response --- 05:27:23 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:27:23 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:27:23 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:27:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723243" 05:27:23 INFO [http_cmd]: http status code OK 05:27:23 DEBUG --- status code --- 05:27:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:23 DEBUG --- http response --- 05:27:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:27:23 DEBUG ----- output ----- 05:27:23 DEBUG { 05:27:23 DEBUG "status": "success", 05:27:23 DEBUG "data": { 05:27:23 DEBUG "resultType": "vector", 05:27:23 DEBUG "result": [ 05:27:23 DEBUG { 05:27:23 DEBUG "metric": { 05:27:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:27:23 DEBUG }, 05:27:23 DEBUG "value": [ 05:27:23 DEBUG 05:27:23 DEBUG 05:27:23 DEBUG 05:27:23 DEBUG [...] 05:27:23 DEBUG 05:27:23 DEBUG 05:27:23 DEBUG 05:27:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:27:23 DEBUG }, 05:27:23 DEBUG "value": [ 05:27:23 DEBUG 1712723243, 05:27:23 DEBUG "0" 05:27:23 DEBUG ] 05:27:23 DEBUG } 05:27:23 DEBUG ] 05:27:23 DEBUG } 05:27:23 DEBUG } 05:27:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723243" 05:27:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723243" 05:27:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723243" 05:27:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723243" 05:27:23 INFO [http_cmd]: http status code OK 05:27:23 INFO [http_cmd]: http status code OK 05:27:23 DEBUG --- status code --- 05:27:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:23 DEBUG --- status code --- 05:27:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:23 DEBUG --- http response --- 05:27:23 DEBUG --- http response --- 05:27:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:27:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:27:23 DEBUG ----- output ----- 05:27:23 DEBUG ----- output ----- 05:27:23 DEBUG { 05:27:23 DEBUG { 05:27:23 DEBUG "status": "success", 05:27:23 DEBUG "status": "success", 05:27:23 DEBUG "data": { 05:27:23 DEBUG "data": { 05:27:23 DEBUG "resultType": "vector", 05:27:23 DEBUG "resultType": "vector", 05:27:23 DEBUG "result": [ 05:27:23 DEBUG "result": [ 05:27:23 DEBUG { 05:27:23 DEBUG { 05:27:23 DEBUG "metric": { 05:27:23 DEBUG "metric": { 05:27:23 DEBUG "pod": "ds-idrepo-0" 05:27:23 DEBUG "pod": "ds-idrepo-0" 05:27:23 DEBUG }, 05:27:23 DEBUG }, 05:27:23 DEBUG "value": [ 05:27:23 DEBUG 05:27:23 DEBUG 05:27:23 DEBUG 05:27:23 DEBUG [...] 05:27:23 DEBUG 05:27:23 DEBUG 05:27:23 DEBUG 05:27:23 DEBUG "pod": "ds-cts-2" 05:27:23 DEBUG }, 05:27:23 DEBUG "value": [ 05:27:23 DEBUG 1712723243, 05:27:23 DEBUG "0" 05:27:23 DEBUG ] 05:27:23 DEBUG } 05:27:23 DEBUG ] 05:27:23 DEBUG } 05:27:23 DEBUG } 05:27:23 DEBUG "value": [ 05:27:23 DEBUG 05:27:23 DEBUG 05:27:23 DEBUG 05:27:23 DEBUG [...] 05:27:23 DEBUG 05:27:23 DEBUG 05:27:23 DEBUG 05:27:23 DEBUG "pod": "ds-cts-2" 05:27:23 DEBUG }, 05:27:23 DEBUG "value": [ 05:27:23 DEBUG 1712723243, 05:27:23 DEBUG "29.833333333333332" 05:27:23 DEBUG ] 05:27:23 DEBUG } 05:27:23 DEBUG ] 05:27:23 DEBUG } 05:27:23 DEBUG } 05:27:23 INFO [http_cmd]: http status code OK 05:27:23 DEBUG --- status code --- 05:27:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:23 DEBUG --- http response --- 05:27:23 INFO [http_cmd]: http status code OK 05:27:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:27:23 DEBUG --- status code --- 05:27:23 DEBUG ----- output ----- 05:27:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:23 DEBUG { 05:27:23 DEBUG --- http response --- 05:27:23 DEBUG "status": "success", 05:27:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:27:23 DEBUG "data": { 05:27:23 DEBUG ----- output ----- 05:27:23 DEBUG "resultType": "vector", 05:27:23 DEBUG { 05:27:23 DEBUG "result": [ 05:27:23 DEBUG "status": "success", 05:27:23 DEBUG { 05:27:23 DEBUG "data": { 05:27:23 DEBUG "metric": { 05:27:23 DEBUG "resultType": "vector", 05:27:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:27:23 DEBUG "result": [ 05:27:23 DEBUG }, 05:27:23 DEBUG { 05:27:23 DEBUG "value": [ 05:27:23 DEBUG "metric": { 05:27:23 DEBUG 05:27:23 DEBUG "instance": "10.142.0.25:9100" 05:27:23 DEBUG 05:27:23 DEBUG }, 05:27:23 DEBUG 05:27:23 DEBUG "value": [ 05:27:23 DEBUG [...] 05:27:23 DEBUG 05:27:23 DEBUG 05:27:23 DEBUG 05:27:23 DEBUG 05:27:23 DEBUG 05:27:23 DEBUG 05:27:23 DEBUG [...] 05:27:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:27:23 DEBUG 05:27:23 DEBUG }, 05:27:23 DEBUG 05:27:23 DEBUG "value": [ 05:27:23 DEBUG 05:27:23 DEBUG 1712723243, 05:27:23 DEBUG "instance": "10.142.0.77:9100" 05:27:23 DEBUG "1257.4666666666667" 05:27:23 DEBUG ] 05:27:23 DEBUG }, 05:27:23 DEBUG } 05:27:23 DEBUG ] 05:27:23 DEBUG "value": [ 05:27:23 DEBUG } 05:27:23 DEBUG 1712723243, 05:27:23 DEBUG "283124382720" 05:27:23 DEBUG ] 05:27:23 DEBUG } 05:27:23 DEBUG ] 05:27:23 DEBUG } 05:27:23 DEBUG } 05:27:23 DEBUG } 05:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723244" 05:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723244" 05:27:24 INFO [http_cmd]: http status code OK 05:27:24 DEBUG --- status code --- 05:27:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:24 DEBUG --- http response --- 05:27:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:27:24 DEBUG ----- output ----- 05:27:24 DEBUG { 05:27:24 DEBUG "status": "success", 05:27:24 DEBUG "data": { 05:27:24 DEBUG "resultType": "vector", 05:27:24 DEBUG "result": [ 05:27:24 DEBUG { 05:27:24 DEBUG "metric": { 05:27:24 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:27:24 DEBUG }, 05:27:24 DEBUG "value": [ 05:27:24 DEBUG 05:27:24 DEBUG 05:27:24 DEBUG 05:27:24 DEBUG [...] 05:27:24 DEBUG 05:27:24 DEBUG 05:27:24 DEBUG 05:27:24 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:27:24 DEBUG }, 05:27:24 DEBUG "value": [ 05:27:24 DEBUG 1712723244, 05:27:24 DEBUG "0" 05:27:24 DEBUG ] 05:27:24 DEBUG } 05:27:24 DEBUG ] 05:27:24 DEBUG } 05:27:24 DEBUG } 05:27:24 INFO [http_cmd]: http status code OK 05:27:24 DEBUG --- status code --- 05:27:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:24 DEBUG --- http response --- 05:27:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:27:24 DEBUG ----- output ----- 05:27:24 DEBUG { 05:27:24 DEBUG "status": "success", 05:27:24 DEBUG "data": { 05:27:24 DEBUG "resultType": "vector", 05:27:24 DEBUG "result": [ 05:27:24 DEBUG { 05:27:24 DEBUG "metric": { 05:27:24 DEBUG "instance": "10.142.0.25:9100" 05:27:24 DEBUG }, 05:27:24 DEBUG "value": [ 05:27:24 DEBUG 05:27:24 DEBUG 05:27:24 DEBUG 05:27:24 DEBUG [...] 05:27:24 DEBUG 05:27:24 DEBUG 05:27:24 DEBUG 05:27:24 DEBUG "instance": "10.142.0.77:9100" 05:27:24 DEBUG }, 05:27:24 DEBUG "value": [ 05:27:24 DEBUG 1712723244, 05:27:24 DEBUG "37264413.766666666" 05:27:24 DEBUG ] 05:27:24 DEBUG } 05:27:24 DEBUG ] 05:27:24 DEBUG } 05:27:24 DEBUG } 05:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723244" 05:27:24 INFO [http_cmd]: http status code OK 05:27:24 DEBUG --- status code --- 05:27:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:24 DEBUG --- http response --- 05:27:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:27:24 DEBUG ----- output ----- 05:27:24 DEBUG { 05:27:24 DEBUG "status": "success", 05:27:24 DEBUG "data": { 05:27:24 DEBUG "resultType": "vector", 05:27:24 DEBUG "result": [ 05:27:24 DEBUG { 05:27:24 DEBUG "metric": { 05:27:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:27:24 DEBUG }, 05:27:24 DEBUG "value": [ 05:27:24 DEBUG 05:27:24 DEBUG 05:27:24 DEBUG 05:27:24 DEBUG [...] 05:27:24 DEBUG 05:27:24 DEBUG 05:27:24 DEBUG 05:27:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:27:24 DEBUG }, 05:27:24 DEBUG "value": [ 05:27:24 DEBUG 1712723244, 05:27:24 DEBUG "226274.5" 05:27:24 DEBUG ] 05:27:24 DEBUG } 05:27:24 DEBUG ] 05:27:24 DEBUG } 05:27:24 DEBUG } 05:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723244" 05:27:24 INFO [http_cmd]: http status code OK 05:27:24 DEBUG --- status code --- 05:27:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:24 DEBUG --- http response --- 05:27:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:27:24 DEBUG ----- output ----- 05:27:24 DEBUG { 05:27:24 DEBUG "status": "success", 05:27:24 DEBUG "data": { 05:27:24 DEBUG "resultType": "vector", 05:27:24 DEBUG "result": [ 05:27:24 DEBUG { 05:27:24 DEBUG "metric": { 05:27:24 DEBUG "pod": "ds-idrepo-0" 05:27:24 DEBUG }, 05:27:24 DEBUG "value": [ 05:27:24 DEBUG 05:27:24 DEBUG 05:27:24 DEBUG 05:27:24 DEBUG [...] 05:27:24 DEBUG 05:27:24 DEBUG 05:27:24 DEBUG 05:27:24 DEBUG "pod": "ds-idrepo-1" 05:27:24 DEBUG }, 05:27:24 DEBUG "value": [ 05:27:24 DEBUG 1712723244, 05:27:24 DEBUG "0" 05:27:24 DEBUG ] 05:27:24 DEBUG } 05:27:24 DEBUG ] 05:27:24 DEBUG } 05:27:24 DEBUG } 05:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723244" 05:27:24 INFO [http_cmd]: http status code OK 05:27:24 DEBUG --- status code --- 05:27:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:24 DEBUG --- http response --- 05:27:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:27:24 DEBUG ----- output ----- 05:27:24 DEBUG { 05:27:24 DEBUG "status": "success", 05:27:24 DEBUG "data": { 05:27:24 DEBUG "resultType": "vector", 05:27:24 DEBUG "result": [ 05:27:24 DEBUG { 05:27:24 DEBUG "metric": { 05:27:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:27:24 DEBUG }, 05:27:24 DEBUG "value": [ 05:27:24 DEBUG 05:27:24 DEBUG 05:27:24 DEBUG 05:27:24 DEBUG [...] 05:27:24 DEBUG 05:27:24 DEBUG 05:27:24 DEBUG 05:27:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:27:24 DEBUG }, 05:27:24 DEBUG "value": [ 05:27:24 DEBUG 1712723244, 05:27:24 DEBUG "1257.3666666666666" 05:27:24 DEBUG ] 05:27:24 DEBUG } 05:27:24 DEBUG ] 05:27:24 DEBUG } 05:27:24 DEBUG } 05:27:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723245" 05:27:25 INFO [http_cmd]: http status code OK 05:27:25 DEBUG --- status code --- 05:27:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:25 DEBUG --- http response --- 05:27:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:27:25 DEBUG ----- output ----- 05:27:25 DEBUG { 05:27:25 DEBUG "status": "success", 05:27:25 DEBUG "data": { 05:27:25 DEBUG "resultType": "vector", 05:27:25 DEBUG "result": [ 05:27:25 DEBUG { 05:27:25 DEBUG "metric": { 05:27:25 DEBUG "instance": "10.142.0.25:9100" 05:27:25 DEBUG }, 05:27:25 DEBUG "value": [ 05:27:25 DEBUG 05:27:25 DEBUG 05:27:25 DEBUG 05:27:25 DEBUG [...] 05:27:25 DEBUG 05:27:25 DEBUG 05:27:25 DEBUG 05:27:25 DEBUG "instance": "10.142.0.77:9100" 05:27:25 DEBUG }, 05:27:25 DEBUG "value": [ 05:27:25 DEBUG 1712723245, 05:27:25 DEBUG "0.12176666666666686" 05:27:25 DEBUG ] 05:27:25 DEBUG } 05:27:25 DEBUG ] 05:27:25 DEBUG } 05:27:25 DEBUG } 05:27:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723245" 05:27:25 INFO [http_cmd]: http status code OK 05:27:25 DEBUG --- status code --- 05:27:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:25 DEBUG --- http response --- 05:27:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:27:25 DEBUG ----- output ----- 05:27:25 DEBUG { 05:27:25 DEBUG "status": "success", 05:27:25 DEBUG "data": { 05:27:25 DEBUG "resultType": "vector", 05:27:25 DEBUG "result": [ 05:27:25 DEBUG { 05:27:25 DEBUG "metric": { 05:27:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:27:25 DEBUG }, 05:27:25 DEBUG "value": [ 05:27:25 DEBUG 05:27:25 DEBUG 05:27:25 DEBUG 05:27:25 DEBUG [...] 05:27:25 DEBUG 05:27:25 DEBUG 05:27:25 DEBUG 05:27:25 DEBUG "pod": "ds-idrepo-1" 05:27:25 DEBUG }, 05:27:25 DEBUG "value": [ 05:27:25 DEBUG 1712723245, 05:27:25 DEBUG "0" 05:27:25 DEBUG ] 05:27:25 DEBUG } 05:27:25 DEBUG ] 05:27:25 DEBUG } 05:27:25 DEBUG } 05:27:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712723245" 05:27:26 INFO [http_cmd]: http status code OK 05:27:26 DEBUG --- status code --- 05:27:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:26 DEBUG --- http response --- 05:27:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:27:26 DEBUG ----- output ----- 05:27:26 DEBUG { 05:27:26 DEBUG "status": "success", 05:27:26 DEBUG "data": { 05:27:26 DEBUG "resultType": "vector", 05:27:26 DEBUG "result": [ 05:27:26 DEBUG { 05:27:26 DEBUG "metric": { 05:27:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:27:26 DEBUG }, 05:27:26 DEBUG "value": [ 05:27:26 DEBUG 05:27:26 DEBUG 05:27:26 DEBUG 05:27:26 DEBUG [...] 05:27:26 DEBUG 05:27:26 DEBUG 05:27:26 DEBUG 05:27:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:27:26 DEBUG }, 05:27:26 DEBUG "value": [ 05:27:26 DEBUG 1712723245, 05:27:26 DEBUG "3.584983497058882" 05:27:26 DEBUG ] 05:27:26 DEBUG } 05:27:26 DEBUG ] 05:27:26 DEBUG } 05:27:26 DEBUG } 05:27:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712723246" 05:27:26 INFO [http_cmd]: http status code OK 05:27:26 DEBUG --- status code --- 05:27:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:26 DEBUG --- http response --- 05:27:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:27:26 DEBUG ----- output ----- 05:27:26 DEBUG { 05:27:26 DEBUG "status": "success", 05:27:26 DEBUG "data": { 05:27:26 DEBUG "resultType": "vector", 05:27:26 DEBUG "result": [ 05:27:26 DEBUG { 05:27:26 DEBUG "metric": { 05:27:26 DEBUG "pod": "ds-idrepo-0" 05:27:26 DEBUG }, 05:27:26 DEBUG "value": [ 05:27:26 DEBUG 05:27:26 DEBUG 05:27:26 DEBUG 05:27:26 DEBUG [...] 05:27:26 DEBUG 05:27:26 DEBUG 05:27:26 DEBUG 05:27:26 DEBUG "pod": "ds-cts-2" 05:27:26 DEBUG }, 05:27:26 DEBUG "value": [ 05:27:26 DEBUG 1712723246, 05:27:26 DEBUG "0" 05:27:26 DEBUG ] 05:27:26 DEBUG } 05:27:26 DEBUG ] 05:27:26 DEBUG } 05:27:26 DEBUG } 05:27:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723246" 05:27:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723246" 05:27:26 INFO [http_cmd]: http status code OK 05:27:26 INFO [http_cmd]: http status code OK 05:27:26 DEBUG --- status code --- 05:27:26 DEBUG --- status code --- 05:27:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:26 DEBUG --- http response --- 05:27:26 DEBUG --- http response --- 05:27:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:27:26 DEBUG ----- output ----- 05:27:26 DEBUG { 05:27:26 DEBUG "status": "success", 05:27:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:27:26 DEBUG "data": { 05:27:26 DEBUG ----- output ----- 05:27:26 DEBUG "resultType": "vector", 05:27:26 DEBUG { 05:27:26 DEBUG "status": "success", 05:27:26 DEBUG "result": [ 05:27:26 DEBUG "data": { 05:27:26 DEBUG { 05:27:26 DEBUG "resultType": "vector", 05:27:26 DEBUG "result": [ 05:27:26 DEBUG "metric": { 05:27:26 DEBUG { 05:27:26 DEBUG "pod": "ds-idrepo-2" 05:27:26 DEBUG "metric": { 05:27:26 DEBUG }, 05:27:26 DEBUG "value": [ 05:27:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:27:26 DEBUG 05:27:26 DEBUG }, 05:27:26 DEBUG 05:27:26 DEBUG "value": [ 05:27:26 DEBUG 05:27:26 DEBUG 05:27:26 DEBUG [...] 05:27:26 DEBUG 05:27:26 DEBUG 05:27:26 DEBUG 05:27:26 DEBUG 05:27:26 DEBUG [...] 05:27:26 DEBUG 05:27:26 DEBUG 05:27:26 DEBUG "pod": "ds-cts-2" 05:27:26 DEBUG 05:27:26 DEBUG }, 05:27:26 DEBUG 05:27:26 DEBUG "pod": "ds-idrepo-1" 05:27:26 DEBUG }, 05:27:26 DEBUG "value": [ 05:27:26 DEBUG "value": [ 05:27:26 DEBUG 1712723246, 05:27:26 DEBUG 1712723246, 05:27:26 DEBUG "0.00019132666666666666" 05:27:26 DEBUG "0" 05:27:26 DEBUG ] 05:27:26 DEBUG ] 05:27:26 DEBUG } 05:27:26 DEBUG } 05:27:26 DEBUG ] 05:27:26 DEBUG ] 05:27:26 DEBUG } 05:27:26 DEBUG } 05:27:26 DEBUG } 05:27:26 DEBUG } 05:27:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723246" 05:27:26 INFO [http_cmd]: http status code OK 05:27:26 DEBUG --- status code --- 05:27:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:26 DEBUG --- http response --- 05:27:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:27:26 DEBUG ----- output ----- 05:27:26 DEBUG { 05:27:26 DEBUG "status": "success", 05:27:26 DEBUG "data": { 05:27:26 DEBUG "resultType": "vector", 05:27:26 DEBUG "result": [ 05:27:26 DEBUG { 05:27:26 DEBUG "metric": { 05:27:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:27:26 DEBUG }, 05:27:26 DEBUG "value": [ 05:27:26 DEBUG 05:27:26 DEBUG 05:27:26 DEBUG 05:27:26 DEBUG [...] 05:27:26 DEBUG 05:27:26 DEBUG 05:27:26 DEBUG 05:27:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:27:26 DEBUG }, 05:27:26 DEBUG "value": [ 05:27:26 DEBUG 1712723246, 05:27:26 DEBUG "1257.1666666666667" 05:27:26 DEBUG ] 05:27:26 DEBUG } 05:27:26 DEBUG ] 05:27:26 DEBUG } 05:27:26 DEBUG } 05:27:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712723246" 05:27:26 INFO [http_cmd]: http status code OK 05:27:26 DEBUG --- status code --- 05:27:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:26 DEBUG --- http response --- 05:27:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:27:26 DEBUG ----- output ----- 05:27:26 DEBUG { 05:27:26 DEBUG "status": "success", 05:27:26 DEBUG "data": { 05:27:26 DEBUG "resultType": "vector", 05:27:26 DEBUG "result": [ 05:27:26 DEBUG { 05:27:26 DEBUG "metric": { 05:27:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:27:26 DEBUG }, 05:27:26 DEBUG "value": [ 05:27:26 DEBUG 05:27:26 DEBUG 05:27:26 DEBUG 05:27:26 DEBUG [...] 05:27:26 DEBUG 05:27:26 DEBUG 05:27:26 DEBUG 05:27:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:27:26 DEBUG }, 05:27:26 DEBUG "value": [ 05:27:26 DEBUG 1712723246, 05:27:26 DEBUG "0.042" 05:27:26 DEBUG ] 05:27:26 DEBUG } 05:27:26 DEBUG ] 05:27:26 DEBUG } 05:27:26 DEBUG } 05:27:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723246" 05:27:26 INFO [http_cmd]: http status code OK 05:27:26 DEBUG --- status code --- 05:27:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:26 DEBUG --- http response --- 05:27:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:27:26 DEBUG ----- output ----- 05:27:26 DEBUG { 05:27:26 DEBUG "status": "success", 05:27:26 DEBUG "data": { 05:27:26 DEBUG "resultType": "vector", 05:27:26 DEBUG "result": [ 05:27:26 DEBUG { 05:27:26 DEBUG "metric": { 05:27:26 DEBUG "pod": "ds-idrepo-0" 05:27:26 DEBUG }, 05:27:26 DEBUG "value": [ 05:27:26 DEBUG 05:27:26 DEBUG 05:27:26 DEBUG 05:27:26 DEBUG [...] 05:27:26 DEBUG 05:27:26 DEBUG 05:27:26 DEBUG 05:27:26 DEBUG "pod": "ds-idrepo-1" 05:27:26 DEBUG }, 05:27:26 DEBUG "value": [ 05:27:26 DEBUG 1712723246, 05:27:26 DEBUG "0" 05:27:26 DEBUG ] 05:27:26 DEBUG } 05:27:26 DEBUG ] 05:27:26 DEBUG } 05:27:26 DEBUG } 05:27:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723247" 05:27:27 INFO [http_cmd]: http status code OK 05:27:27 DEBUG --- status code --- 05:27:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:27 DEBUG --- http response --- 05:27:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:27:27 DEBUG ----- output ----- 05:27:27 DEBUG { 05:27:27 DEBUG "status": "success", 05:27:27 DEBUG "data": { 05:27:27 DEBUG "resultType": "vector", 05:27:27 DEBUG "result": [ 05:27:27 DEBUG { 05:27:27 DEBUG "metric": { 05:27:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:27:27 DEBUG }, 05:27:27 DEBUG "value": [ 05:27:27 DEBUG 05:27:27 DEBUG 05:27:27 DEBUG 05:27:27 DEBUG [...] 05:27:27 DEBUG 05:27:27 DEBUG 05:27:27 DEBUG 05:27:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:27:27 DEBUG }, 05:27:27 DEBUG "value": [ 05:27:27 DEBUG 1712723247, 05:27:27 DEBUG "1260.6" 05:27:27 DEBUG ] 05:27:27 DEBUG } 05:27:27 DEBUG ] 05:27:27 DEBUG } 05:27:27 DEBUG } 05:27:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723247" 05:27:27 INFO [http_cmd]: http status code OK 05:27:27 DEBUG --- status code --- 05:27:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:27 DEBUG --- http response --- 05:27:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:27:27 DEBUG ----- output ----- 05:27:27 DEBUG { 05:27:27 DEBUG "status": "success", 05:27:27 DEBUG "data": { 05:27:27 DEBUG "resultType": "vector", 05:27:27 DEBUG "result": [ 05:27:27 DEBUG { 05:27:27 DEBUG "metric": { 05:27:27 DEBUG "pod": "ds-idrepo-0" 05:27:27 DEBUG }, 05:27:27 DEBUG "value": [ 05:27:27 DEBUG 05:27:27 DEBUG 05:27:27 DEBUG 05:27:27 DEBUG [...] 05:27:27 DEBUG 05:27:27 DEBUG 05:27:27 DEBUG 05:27:27 DEBUG "pod": "ds-idrepo-1" 05:27:27 DEBUG }, 05:27:27 DEBUG "value": [ 05:27:27 DEBUG 1712723247, 05:27:27 DEBUG "0" 05:27:27 DEBUG ] 05:27:27 DEBUG } 05:27:27 DEBUG ] 05:27:27 DEBUG } 05:27:27 DEBUG } 05:27:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712723247" 05:27:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723247" 05:27:27 INFO [http_cmd]: http status code OK 05:27:27 DEBUG --- status code --- 05:27:27 INFO [http_cmd]: http status code OK 05:27:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:27 DEBUG --- http response --- 05:27:27 DEBUG --- status code --- 05:27:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:27:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:27 DEBUG ----- output ----- 05:27:27 DEBUG --- http response --- 05:27:27 DEBUG { 05:27:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:27:27 DEBUG "status": "success", 05:27:27 DEBUG ----- output ----- 05:27:27 DEBUG "data": { 05:27:27 DEBUG { 05:27:27 DEBUG "resultType": "vector", 05:27:27 DEBUG "status": "success", 05:27:27 DEBUG "result": [ 05:27:27 DEBUG "data": { 05:27:27 DEBUG { 05:27:27 DEBUG "resultType": "vector", 05:27:27 DEBUG "metric": { 05:27:27 DEBUG "result": [ 05:27:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:27:27 DEBUG { 05:27:27 DEBUG }, 05:27:27 DEBUG "metric": { 05:27:27 DEBUG "value": [ 05:27:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:27:27 DEBUG 05:27:27 DEBUG }, 05:27:27 DEBUG 05:27:27 DEBUG "value": [ 05:27:27 DEBUG 05:27:27 DEBUG 05:27:27 DEBUG [...] 05:27:27 DEBUG 05:27:27 DEBUG 05:27:27 DEBUG 05:27:27 DEBUG [...] 05:27:27 DEBUG 05:27:27 DEBUG 05:27:27 DEBUG 05:27:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:27:27 DEBUG }, 05:27:27 DEBUG "value": [ 05:27:27 DEBUG 1712723247, 05:27:27 DEBUG "2488578048" 05:27:27 DEBUG ] 05:27:27 DEBUG } 05:27:27 DEBUG ] 05:27:27 DEBUG } 05:27:27 DEBUG } 05:27:27 DEBUG 05:27:27 DEBUG 05:27:27 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:27:27 DEBUG }, 05:27:27 DEBUG "value": [ 05:27:27 DEBUG 1712723247, 05:27:27 DEBUG "0" 05:27:27 DEBUG ] 05:27:27 DEBUG } 05:27:27 DEBUG ] 05:27:27 DEBUG } 05:27:27 DEBUG } 05:27:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723247" 05:27:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723247" 05:27:27 INFO [http_cmd]: http status code OK 05:27:27 DEBUG --- status code --- 05:27:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:27 DEBUG --- http response --- 05:27:27 INFO [http_cmd]: http status code OK 05:27:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:27:27 DEBUG --- status code --- 05:27:27 DEBUG ----- output ----- 05:27:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:27:27 DEBUG { 05:27:27 DEBUG --- http response --- 05:27:27 DEBUG "status": "success", 05:27:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:27:27 DEBUG "data": { 05:27:27 DEBUG ----- output ----- 05:27:27 DEBUG "resultType": "vector", 05:27:27 DEBUG { 05:27:27 DEBUG "result": [ 05:27:27 DEBUG "status": "success", 05:27:27 DEBUG { 05:27:27 DEBUG "data": { 05:27:27 DEBUG "metric": { 05:27:27 DEBUG "resultType": "vector", 05:27:27 DEBUG "instance": "10.142.0.25:9100" 05:27:27 DEBUG "result": [ 05:27:27 DEBUG }, 05:27:27 DEBUG { 05:27:27 DEBUG "value": [ 05:27:27 DEBUG "metric": { 05:27:27 DEBUG 05:27:27 DEBUG "instance": "10.142.0.25:9100" 05:27:27 DEBUG 05:27:27 DEBUG }, 05:27:27 DEBUG 05:27:27 DEBUG "value": [ 05:27:27 DEBUG [...] 05:27:27 DEBUG 05:27:27 DEBUG 05:27:27 DEBUG 05:27:27 DEBUG 05:27:27 DEBUG 05:27:27 DEBUG 05:27:27 DEBUG [...] 05:27:27 DEBUG "instance": "10.142.0.77:9100" 05:27:27 DEBUG 05:27:27 DEBUG }, 05:27:27 DEBUG 05:27:27 DEBUG "value": [ 05:27:27 DEBUG 05:27:27 DEBUG 1712723247, 05:27:27 DEBUG "instance": "10.142.0.77:9100" 05:27:27 DEBUG "37263877.86666667" 05:27:27 DEBUG }, 05:27:27 DEBUG ] 05:27:27 DEBUG "value": [ 05:27:27 DEBUG } 05:27:27 DEBUG 1712723247, 05:27:27 DEBUG ] 05:27:27 DEBUG "0.04199999999999987" 05:27:27 DEBUG } 05:27:27 DEBUG ] 05:27:27 DEBUG } 05:27:27 DEBUG } 05:27:27 DEBUG ] 05:27:27 DEBUG } 05:27:27 DEBUG } 05:27:41 INFO 05:27:41 INFO [loop_until]: kubectl --namespace=xlou top pods 05:27:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:27:42 INFO [loop_until]: OK (rc = 0) 05:27:42 DEBUG --- stdout --- 05:27:42 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 8813m 8805Mi am-654c4fcbcf-9scrj 8618m 8418Mi am-654c4fcbcf-v5qb5 7289m 8397Mi ds-cts-0 7245m 12421Mi ds-cts-1 7575m 12571Mi ds-cts-2 6621m 13285Mi ds-idrepo-0 9619m 5621Mi ds-idrepo-1 311m 5669Mi ds-idrepo-2 330m 5698Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 5m 2213Mi lodemon-b776b7d88-thrqf 6m 70Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3567m 1662Mi 05:27:42 DEBUG --- stderr --- 05:27:42 DEBUG 05:27:48 INFO 05:27:48 INFO [loop_until]: kubectl --namespace=xlou top node 05:27:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:27:48 INFO [loop_until]: OK (rc = 0) 05:27:48 DEBUG --- stdout --- 05:27:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 94m 0% 2303Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8137m 51% 9280Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 83m 0% 3259Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10428m 65% 6504Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9565m 60% 9311Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9606m 60% 9645Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 76m 0% 859Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9753m 61% 6281Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 7458m 46% 13086Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 6672m 41% 13895Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 384m 2% 6365Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7538m 47% 13169Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 370m 2% 6330Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3701m 23% 5340Mi 9% 05:27:48 DEBUG --- stderr --- 05:27:48 DEBUG 05:28:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723303" 05:28:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712723303" 05:28:23 ERROR [http_cmd]: ERROR 05:28:23 INFO [http_cmd]: http status code OK 05:28:23 ERROR --- status code --- 05:28:23 DEBUG --- status code --- 05:28:23 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:28:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:23 ERROR --- http response --- 05:28:23 DEBUG --- http response --- 05:28:23 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:28:23 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:28:23 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:28:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723303" 05:28:23 INFO [http_cmd]: http status code OK 05:28:23 DEBUG --- status code --- 05:28:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:23 DEBUG --- http response --- 05:28:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:28:23 DEBUG ----- output ----- 05:28:23 DEBUG { 05:28:23 DEBUG "status": "success", 05:28:23 DEBUG "data": { 05:28:23 DEBUG "resultType": "vector", 05:28:23 DEBUG "result": [ 05:28:23 DEBUG { 05:28:23 DEBUG "metric": { 05:28:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:28:23 DEBUG }, 05:28:23 DEBUG "value": [ 05:28:23 DEBUG 05:28:23 DEBUG 05:28:23 DEBUG 05:28:23 DEBUG [...] 05:28:23 DEBUG 05:28:23 DEBUG 05:28:23 DEBUG 05:28:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:28:23 DEBUG }, 05:28:23 DEBUG "value": [ 05:28:23 DEBUG 1712723303, 05:28:23 DEBUG "0" 05:28:23 DEBUG ] 05:28:23 DEBUG } 05:28:23 DEBUG ] 05:28:23 DEBUG } 05:28:23 DEBUG } 05:28:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723303" 05:28:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723303" 05:28:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723303" 05:28:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723303" 05:28:23 INFO [http_cmd]: http status code OK 05:28:23 INFO [http_cmd]: http status code OK 05:28:23 DEBUG --- status code --- 05:28:23 DEBUG --- status code --- 05:28:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:23 DEBUG --- http response --- 05:28:23 DEBUG --- http response --- 05:28:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:28:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:28:23 DEBUG ----- output ----- 05:28:23 DEBUG ----- output ----- 05:28:23 DEBUG { 05:28:23 DEBUG { 05:28:23 DEBUG "status": "success", 05:28:23 DEBUG "status": "success", 05:28:23 DEBUG "data": { 05:28:23 DEBUG "data": { 05:28:23 DEBUG "resultType": "vector", 05:28:23 DEBUG "resultType": "vector", 05:28:23 DEBUG "result": [ 05:28:23 DEBUG "result": [ 05:28:23 DEBUG { 05:28:23 DEBUG { 05:28:23 DEBUG "metric": { 05:28:23 DEBUG "metric": { 05:28:23 DEBUG "pod": "ds-idrepo-0" 05:28:23 DEBUG "pod": "ds-idrepo-0" 05:28:23 DEBUG }, 05:28:23 DEBUG }, 05:28:23 DEBUG "value": [ 05:28:23 DEBUG "value": [ 05:28:23 DEBUG 05:28:23 DEBUG 05:28:23 INFO [http_cmd]: http status code OK 05:28:23 DEBUG --- status code --- 05:28:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:23 DEBUG 05:28:23 DEBUG --- http response --- 05:28:23 DEBUG 05:28:23 DEBUG 05:28:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:28:23 DEBUG [...] 05:28:23 DEBUG 05:28:23 DEBUG ----- output ----- 05:28:23 DEBUG 05:28:23 DEBUG 05:28:23 DEBUG { 05:28:23 DEBUG "pod": "ds-cts-2" 05:28:23 DEBUG }, 05:28:23 DEBUG "value": [ 05:28:23 DEBUG 1712723303, 05:28:23 DEBUG "status": "success", 05:28:23 DEBUG "29.8" 05:28:23 DEBUG ] 05:28:23 DEBUG "data": { 05:28:23 DEBUG } 05:28:23 DEBUG 05:28:23 DEBUG "resultType": "vector", 05:28:23 DEBUG ] 05:28:23 DEBUG [...] 05:28:23 DEBUG "result": [ 05:28:23 INFO [http_cmd]: http status code OK 05:28:23 DEBUG } 05:28:23 DEBUG 05:28:23 DEBUG { 05:28:23 DEBUG --- status code --- 05:28:23 DEBUG } 05:28:23 DEBUG 05:28:23 DEBUG "metric": { 05:28:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:23 DEBUG 05:28:23 DEBUG "instance": "10.142.0.25:9100" 05:28:23 DEBUG --- http response --- 05:28:23 DEBUG "pod": "ds-cts-2" 05:28:23 DEBUG }, 05:28:23 DEBUG }, 05:28:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:28:23 DEBUG "value": [ 05:28:23 DEBUG "value": [ 05:28:23 DEBUG ----- output ----- 05:28:23 DEBUG 05:28:23 DEBUG 1712723303, 05:28:23 DEBUG "0" 05:28:23 DEBUG 05:28:23 DEBUG ] 05:28:23 DEBUG { 05:28:23 DEBUG 05:28:23 DEBUG } 05:28:23 DEBUG "status": "success", 05:28:23 DEBUG [...] 05:28:23 DEBUG ] 05:28:23 DEBUG "data": { 05:28:23 DEBUG 05:28:23 DEBUG } 05:28:23 DEBUG } 05:28:23 DEBUG 05:28:23 DEBUG "resultType": "vector", 05:28:23 DEBUG "result": [ 05:28:23 DEBUG 05:28:23 DEBUG { 05:28:23 DEBUG "instance": "10.142.0.77:9100" 05:28:23 DEBUG "metric": { 05:28:23 DEBUG }, 05:28:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:28:23 DEBUG "value": [ 05:28:23 DEBUG }, 05:28:23 DEBUG 1712723303, 05:28:23 DEBUG "value": [ 05:28:23 DEBUG "284933790720" 05:28:23 DEBUG 05:28:23 DEBUG ] 05:28:23 DEBUG 05:28:23 DEBUG } 05:28:23 DEBUG 05:28:23 DEBUG [...] 05:28:23 DEBUG 05:28:23 DEBUG 05:28:23 DEBUG 05:28:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:28:23 DEBUG }, 05:28:23 DEBUG "value": [ 05:28:23 DEBUG 1712723303, 05:28:23 DEBUG "1261.4333333333334" 05:28:23 DEBUG ] 05:28:23 DEBUG } 05:28:23 DEBUG ] 05:28:23 DEBUG } 05:28:23 DEBUG } 05:28:23 DEBUG ] 05:28:23 DEBUG } 05:28:23 DEBUG } 05:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723304" 05:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723304" 05:28:24 INFO [http_cmd]: http status code OK 05:28:24 DEBUG --- status code --- 05:28:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:24 DEBUG --- http response --- 05:28:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:28:24 DEBUG ----- output ----- 05:28:24 DEBUG { 05:28:24 DEBUG "status": "success", 05:28:24 DEBUG "data": { 05:28:24 DEBUG "resultType": "vector", 05:28:24 DEBUG "result": [ 05:28:24 DEBUG { 05:28:24 DEBUG "metric": { 05:28:24 DEBUG "instance": "10.142.0.25:9100" 05:28:24 DEBUG }, 05:28:24 DEBUG "value": [ 05:28:24 DEBUG 05:28:24 DEBUG 05:28:24 DEBUG 05:28:24 DEBUG [...] 05:28:24 DEBUG 05:28:24 DEBUG 05:28:24 DEBUG 05:28:24 DEBUG "instance": "10.142.0.77:9100" 05:28:24 DEBUG }, 05:28:24 DEBUG "value": [ 05:28:24 DEBUG 1712723304, 05:28:24 DEBUG "37683197.5" 05:28:24 DEBUG ] 05:28:24 DEBUG } 05:28:24 DEBUG ] 05:28:24 DEBUG } 05:28:24 DEBUG } 05:28:24 INFO [http_cmd]: http status code OK 05:28:24 DEBUG --- status code --- 05:28:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:24 DEBUG --- http response --- 05:28:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 05:28:24 DEBUG ----- output ----- 05:28:24 DEBUG { 05:28:24 DEBUG "status": "success", 05:28:24 DEBUG "data": { 05:28:24 DEBUG "resultType": "vector", 05:28:24 DEBUG "result": [ 05:28:24 DEBUG { 05:28:24 DEBUG "metric": { 05:28:24 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:28:24 DEBUG }, 05:28:24 DEBUG "value": [ 05:28:24 DEBUG 05:28:24 DEBUG 05:28:24 DEBUG 05:28:24 DEBUG [...] 05:28:24 DEBUG 05:28:24 DEBUG 05:28:24 DEBUG 05:28:24 DEBUG "pod": "ds-idrepo-1" 05:28:24 DEBUG }, 05:28:24 DEBUG "value": [ 05:28:24 DEBUG 1712723304, 05:28:24 DEBUG "0" 05:28:24 DEBUG ] 05:28:24 DEBUG } 05:28:24 DEBUG ] 05:28:24 DEBUG } 05:28:24 DEBUG } 05:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723304" 05:28:24 INFO [http_cmd]: http status code OK 05:28:24 DEBUG --- status code --- 05:28:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:24 DEBUG --- http response --- 05:28:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:28:24 DEBUG ----- output ----- 05:28:24 DEBUG { 05:28:24 DEBUG "status": "success", 05:28:24 DEBUG "data": { 05:28:24 DEBUG "resultType": "vector", 05:28:24 DEBUG "result": [ 05:28:24 DEBUG { 05:28:24 DEBUG "metric": { 05:28:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:28:24 DEBUG }, 05:28:24 DEBUG "value": [ 05:28:24 DEBUG 05:28:24 DEBUG 05:28:24 DEBUG 05:28:24 DEBUG [...] 05:28:24 DEBUG 05:28:24 DEBUG 05:28:24 DEBUG 05:28:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:28:24 DEBUG }, 05:28:24 DEBUG "value": [ 05:28:24 DEBUG 1712723304, 05:28:24 DEBUG "226150.5" 05:28:24 DEBUG ] 05:28:24 DEBUG } 05:28:24 DEBUG ] 05:28:24 DEBUG } 05:28:24 DEBUG } 05:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723304" 05:28:24 INFO [http_cmd]: http status code OK 05:28:24 DEBUG --- status code --- 05:28:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:24 DEBUG --- http response --- 05:28:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:28:24 DEBUG ----- output ----- 05:28:24 DEBUG { 05:28:24 DEBUG "status": "success", 05:28:24 DEBUG "data": { 05:28:24 DEBUG "resultType": "vector", 05:28:24 DEBUG "result": [ 05:28:24 DEBUG { 05:28:24 DEBUG "metric": { 05:28:24 DEBUG "pod": "ds-idrepo-0" 05:28:24 DEBUG }, 05:28:24 DEBUG "value": [ 05:28:24 DEBUG 05:28:24 DEBUG 05:28:24 DEBUG 05:28:24 DEBUG [...] 05:28:24 DEBUG 05:28:24 DEBUG 05:28:24 DEBUG 05:28:24 DEBUG "pod": "ds-idrepo-1" 05:28:24 DEBUG }, 05:28:24 DEBUG "value": [ 05:28:24 DEBUG 1712723304, 05:28:24 DEBUG "0" 05:28:24 DEBUG ] 05:28:24 DEBUG } 05:28:24 DEBUG ] 05:28:24 DEBUG } 05:28:24 DEBUG } 05:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723304" 05:28:24 INFO [http_cmd]: http status code OK 05:28:24 DEBUG --- status code --- 05:28:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:24 DEBUG --- http response --- 05:28:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:28:24 DEBUG ----- output ----- 05:28:24 DEBUG { 05:28:24 DEBUG "status": "success", 05:28:24 DEBUG "data": { 05:28:24 DEBUG "resultType": "vector", 05:28:24 DEBUG "result": [ 05:28:24 DEBUG { 05:28:24 DEBUG "metric": { 05:28:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:28:24 DEBUG }, 05:28:24 DEBUG "value": [ 05:28:24 DEBUG 05:28:24 DEBUG 05:28:24 DEBUG 05:28:24 DEBUG [...] 05:28:24 DEBUG 05:28:24 DEBUG 05:28:24 DEBUG 05:28:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:28:24 DEBUG }, 05:28:24 DEBUG "value": [ 05:28:24 DEBUG 1712723304, 05:28:24 DEBUG "1263.5333333333333" 05:28:24 DEBUG ] 05:28:24 DEBUG } 05:28:24 DEBUG ] 05:28:24 DEBUG } 05:28:24 DEBUG } 05:28:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723305" 05:28:25 INFO [http_cmd]: http status code OK 05:28:25 DEBUG --- status code --- 05:28:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:25 DEBUG --- http response --- 05:28:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:28:25 DEBUG ----- output ----- 05:28:25 DEBUG { 05:28:25 DEBUG "status": "success", 05:28:25 DEBUG "data": { 05:28:25 DEBUG "resultType": "vector", 05:28:25 DEBUG "result": [ 05:28:25 DEBUG { 05:28:25 DEBUG "metric": { 05:28:25 DEBUG "instance": "10.142.0.25:9100" 05:28:25 DEBUG }, 05:28:25 DEBUG "value": [ 05:28:25 DEBUG 05:28:25 DEBUG 05:28:25 DEBUG 05:28:25 DEBUG [...] 05:28:25 DEBUG 05:28:25 DEBUG 05:28:25 DEBUG 05:28:25 DEBUG "instance": "10.142.0.77:9100" 05:28:25 DEBUG }, 05:28:25 DEBUG "value": [ 05:28:25 DEBUG 1712723305, 05:28:25 DEBUG "0.03403333333333146" 05:28:25 DEBUG ] 05:28:25 DEBUG } 05:28:25 DEBUG ] 05:28:25 DEBUG } 05:28:25 DEBUG } 05:28:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723305" 05:28:25 INFO [http_cmd]: http status code OK 05:28:25 DEBUG --- status code --- 05:28:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:25 DEBUG --- http response --- 05:28:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:28:25 DEBUG ----- output ----- 05:28:25 DEBUG { 05:28:25 DEBUG "status": "success", 05:28:25 DEBUG "data": { 05:28:25 DEBUG "resultType": "vector", 05:28:25 DEBUG "result": [ 05:28:25 DEBUG { 05:28:25 DEBUG "metric": { 05:28:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:28:25 DEBUG }, 05:28:25 DEBUG "value": [ 05:28:25 DEBUG 05:28:25 DEBUG 05:28:25 DEBUG 05:28:25 DEBUG [...] 05:28:25 DEBUG 05:28:25 DEBUG 05:28:25 DEBUG 05:28:25 DEBUG "pod": "ds-idrepo-1" 05:28:25 DEBUG }, 05:28:25 DEBUG "value": [ 05:28:25 DEBUG 1712723305, 05:28:25 DEBUG "12968.20317287397" 05:28:25 DEBUG ] 05:28:25 DEBUG } 05:28:25 DEBUG ] 05:28:25 DEBUG } 05:28:25 DEBUG } 05:28:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712723306" 05:28:26 INFO [http_cmd]: http status code OK 05:28:26 DEBUG --- status code --- 05:28:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:26 DEBUG --- http response --- 05:28:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:28:26 DEBUG ----- output ----- 05:28:26 DEBUG { 05:28:26 DEBUG "status": "success", 05:28:26 DEBUG "data": { 05:28:26 DEBUG "resultType": "vector", 05:28:26 DEBUG "result": [ 05:28:26 DEBUG { 05:28:26 DEBUG "metric": { 05:28:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:28:26 DEBUG }, 05:28:26 DEBUG "value": [ 05:28:26 DEBUG 05:28:26 DEBUG 05:28:26 DEBUG 05:28:26 DEBUG [...] 05:28:26 DEBUG 05:28:26 DEBUG 05:28:26 DEBUG 05:28:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:28:26 DEBUG }, 05:28:26 DEBUG "value": [ 05:28:26 DEBUG 1712723306, 05:28:26 DEBUG "3.618775643078146" 05:28:26 DEBUG ] 05:28:26 DEBUG } 05:28:26 DEBUG ] 05:28:26 DEBUG } 05:28:26 DEBUG } 05:28:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712723306" 05:28:26 INFO [http_cmd]: http status code OK 05:28:26 DEBUG --- status code --- 05:28:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:26 DEBUG --- http response --- 05:28:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:28:26 DEBUG ----- output ----- 05:28:26 DEBUG { 05:28:26 DEBUG "status": "success", 05:28:26 DEBUG "data": { 05:28:26 DEBUG "resultType": "vector", 05:28:26 DEBUG "result": [ 05:28:26 DEBUG { 05:28:26 DEBUG "metric": { 05:28:26 DEBUG "pod": "ds-idrepo-0" 05:28:26 DEBUG }, 05:28:26 DEBUG "value": [ 05:28:26 DEBUG 05:28:26 DEBUG 05:28:26 DEBUG 05:28:26 DEBUG [...] 05:28:26 DEBUG 05:28:26 DEBUG 05:28:26 DEBUG 05:28:26 DEBUG "pod": "ds-cts-2" 05:28:26 DEBUG }, 05:28:26 DEBUG "value": [ 05:28:26 DEBUG 1712723306, 05:28:26 DEBUG "0" 05:28:26 DEBUG ] 05:28:26 DEBUG } 05:28:26 DEBUG ] 05:28:26 DEBUG } 05:28:26 DEBUG } 05:28:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723306" 05:28:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723306" 05:28:26 INFO [http_cmd]: http status code OK 05:28:26 INFO [http_cmd]: http status code OK 05:28:26 DEBUG --- status code --- 05:28:26 DEBUG --- status code --- 05:28:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:26 DEBUG --- http response --- 05:28:26 DEBUG --- http response --- 05:28:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:28:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:28:26 DEBUG ----- output ----- 05:28:26 DEBUG ----- output ----- 05:28:26 DEBUG { 05:28:26 DEBUG { 05:28:26 DEBUG "status": "success", 05:28:26 DEBUG "status": "success", 05:28:26 DEBUG "data": { 05:28:26 DEBUG "data": { 05:28:26 DEBUG "resultType": "vector", 05:28:26 DEBUG "resultType": "vector", 05:28:26 DEBUG "result": [ 05:28:26 DEBUG "result": [ 05:28:26 DEBUG { 05:28:26 DEBUG { 05:28:26 DEBUG "metric": { 05:28:26 DEBUG "metric": { 05:28:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:28:26 DEBUG "pod": "ds-idrepo-2" 05:28:26 DEBUG }, 05:28:26 DEBUG }, 05:28:26 DEBUG "value": [ 05:28:26 DEBUG "value": [ 05:28:26 DEBUG 05:28:26 DEBUG 05:28:26 DEBUG 05:28:26 DEBUG 05:28:26 DEBUG 05:28:26 DEBUG 05:28:26 DEBUG [...] 05:28:26 DEBUG [...] 05:28:26 DEBUG 05:28:26 DEBUG 05:28:26 DEBUG 05:28:26 DEBUG 05:28:26 DEBUG 05:28:26 DEBUG 05:28:26 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:28:26 DEBUG "pod": "ds-cts-2" 05:28:26 DEBUG }, 05:28:26 DEBUG "value": [ 05:28:26 DEBUG }, 05:28:26 DEBUG 1712723306, 05:28:26 DEBUG "value": [ 05:28:26 DEBUG "3.577228017458057" 05:28:26 DEBUG 1712723306, 05:28:26 DEBUG ] 05:28:26 DEBUG "0.00016131" 05:28:26 DEBUG } 05:28:26 DEBUG ] 05:28:26 DEBUG ] 05:28:26 DEBUG } 05:28:26 DEBUG } 05:28:26 DEBUG ] 05:28:26 DEBUG } 05:28:26 DEBUG } 05:28:26 DEBUG } 05:28:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723306" 05:28:26 INFO [http_cmd]: http status code OK 05:28:26 DEBUG --- status code --- 05:28:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:26 DEBUG --- http response --- 05:28:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:28:26 DEBUG ----- output ----- 05:28:26 DEBUG { 05:28:26 DEBUG "status": "success", 05:28:26 DEBUG "data": { 05:28:26 DEBUG "resultType": "vector", 05:28:26 DEBUG "result": [ 05:28:26 DEBUG { 05:28:26 DEBUG "metric": { 05:28:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:28:26 DEBUG }, 05:28:26 DEBUG "value": [ 05:28:26 DEBUG 05:28:26 DEBUG 05:28:26 DEBUG 05:28:26 DEBUG [...] 05:28:26 DEBUG 05:28:26 DEBUG 05:28:26 DEBUG 05:28:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:28:26 DEBUG }, 05:28:26 DEBUG "value": [ 05:28:26 DEBUG 1712723306, 05:28:26 DEBUG "1264.0666666666666" 05:28:26 DEBUG ] 05:28:26 DEBUG } 05:28:26 DEBUG ] 05:28:26 DEBUG } 05:28:26 DEBUG } 05:28:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712723306" 05:28:26 INFO [http_cmd]: http status code OK 05:28:26 DEBUG --- status code --- 05:28:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:26 DEBUG --- http response --- 05:28:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:28:26 DEBUG ----- output ----- 05:28:26 DEBUG { 05:28:26 DEBUG "status": "success", 05:28:26 DEBUG "data": { 05:28:26 DEBUG "resultType": "vector", 05:28:26 DEBUG "result": [ 05:28:26 DEBUG { 05:28:26 DEBUG "metric": { 05:28:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:28:26 DEBUG }, 05:28:26 DEBUG "value": [ 05:28:26 DEBUG 05:28:26 DEBUG 05:28:26 DEBUG 05:28:26 DEBUG [...] 05:28:26 DEBUG 05:28:26 DEBUG 05:28:26 DEBUG 05:28:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:28:26 DEBUG }, 05:28:26 DEBUG "value": [ 05:28:26 DEBUG 1712723306, 05:28:26 DEBUG "0.042" 05:28:26 DEBUG ] 05:28:26 DEBUG } 05:28:26 DEBUG ] 05:28:26 DEBUG } 05:28:26 DEBUG } 05:28:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723307" 05:28:27 INFO [http_cmd]: http status code OK 05:28:27 DEBUG --- status code --- 05:28:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:27 DEBUG --- http response --- 05:28:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:28:27 DEBUG ----- output ----- 05:28:27 DEBUG { 05:28:27 DEBUG "status": "success", 05:28:27 DEBUG "data": { 05:28:27 DEBUG "resultType": "vector", 05:28:27 DEBUG "result": [ 05:28:27 DEBUG { 05:28:27 DEBUG "metric": { 05:28:27 DEBUG "pod": "ds-idrepo-0" 05:28:27 DEBUG }, 05:28:27 DEBUG "value": [ 05:28:27 DEBUG 05:28:27 DEBUG 05:28:27 DEBUG 05:28:27 DEBUG [...] 05:28:27 DEBUG 05:28:27 DEBUG 05:28:27 DEBUG 05:28:27 DEBUG "pod": "ds-idrepo-1" 05:28:27 DEBUG }, 05:28:27 DEBUG "value": [ 05:28:27 DEBUG 1712723307, 05:28:27 DEBUG "0" 05:28:27 DEBUG ] 05:28:27 DEBUG } 05:28:27 DEBUG ] 05:28:27 DEBUG } 05:28:27 DEBUG } 05:28:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723307" 05:28:27 INFO [http_cmd]: http status code OK 05:28:27 DEBUG --- status code --- 05:28:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:27 DEBUG --- http response --- 05:28:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:28:27 DEBUG ----- output ----- 05:28:27 DEBUG { 05:28:27 DEBUG "status": "success", 05:28:27 DEBUG "data": { 05:28:27 DEBUG "resultType": "vector", 05:28:27 DEBUG "result": [ 05:28:27 DEBUG { 05:28:27 DEBUG "metric": { 05:28:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:28:27 DEBUG }, 05:28:27 DEBUG "value": [ 05:28:27 DEBUG 05:28:27 DEBUG 05:28:27 DEBUG 05:28:27 DEBUG [...] 05:28:27 DEBUG 05:28:27 DEBUG 05:28:27 DEBUG 05:28:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:28:27 DEBUG }, 05:28:27 DEBUG "value": [ 05:28:27 DEBUG 1712723307, 05:28:27 DEBUG "1264.6333333333332" 05:28:27 DEBUG ] 05:28:27 DEBUG } 05:28:27 DEBUG ] 05:28:27 DEBUG } 05:28:27 DEBUG } 05:28:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723307" 05:28:27 INFO [http_cmd]: http status code OK 05:28:27 DEBUG --- status code --- 05:28:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:27 DEBUG --- http response --- 05:28:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:28:27 DEBUG ----- output ----- 05:28:27 DEBUG { 05:28:27 DEBUG "status": "success", 05:28:27 DEBUG "data": { 05:28:27 DEBUG "resultType": "vector", 05:28:27 DEBUG "result": [ 05:28:27 DEBUG { 05:28:27 DEBUG "metric": { 05:28:27 DEBUG "pod": "ds-idrepo-0" 05:28:27 DEBUG }, 05:28:27 DEBUG "value": [ 05:28:27 DEBUG 05:28:27 DEBUG 05:28:27 DEBUG 05:28:27 DEBUG [...] 05:28:27 DEBUG 05:28:27 DEBUG 05:28:27 DEBUG 05:28:27 DEBUG "pod": "ds-idrepo-1" 05:28:27 DEBUG }, 05:28:27 DEBUG "value": [ 05:28:27 DEBUG 1712723307, 05:28:27 DEBUG "0" 05:28:27 DEBUG ] 05:28:27 DEBUG } 05:28:27 DEBUG ] 05:28:27 DEBUG } 05:28:27 DEBUG } 05:28:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712723307" 05:28:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723307" 05:28:27 INFO [http_cmd]: http status code OK 05:28:27 DEBUG --- status code --- 05:28:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:27 DEBUG --- http response --- 05:28:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:28:27 DEBUG ----- output ----- 05:28:27 DEBUG { 05:28:27 DEBUG "status": "success", 05:28:27 DEBUG "data": { 05:28:27 DEBUG "resultType": "vector", 05:28:27 DEBUG "result": [ 05:28:27 DEBUG { 05:28:27 DEBUG "metric": { 05:28:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:28:27 DEBUG }, 05:28:27 DEBUG "value": [ 05:28:27 DEBUG 05:28:27 DEBUG 05:28:27 DEBUG 05:28:27 DEBUG [...] 05:28:27 DEBUG 05:28:27 DEBUG 05:28:27 DEBUG 05:28:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:28:27 DEBUG }, 05:28:27 DEBUG "value": [ 05:28:27 DEBUG 1712723307, 05:28:27 DEBUG "2489397248" 05:28:27 DEBUG ] 05:28:27 DEBUG } 05:28:27 DEBUG ] 05:28:27 DEBUG } 05:28:27 DEBUG } 05:28:27 INFO [http_cmd]: http status code OK 05:28:27 DEBUG --- status code --- 05:28:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:27 DEBUG --- http response --- 05:28:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:28:27 DEBUG ----- output ----- 05:28:27 DEBUG { 05:28:27 DEBUG "status": "success", 05:28:27 DEBUG "data": { 05:28:27 DEBUG "resultType": "vector", 05:28:27 DEBUG "result": [ 05:28:27 DEBUG { 05:28:27 DEBUG "metric": { 05:28:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:28:27 DEBUG }, 05:28:27 DEBUG "value": [ 05:28:27 DEBUG 05:28:27 DEBUG 05:28:27 DEBUG 05:28:27 DEBUG [...] 05:28:27 DEBUG 05:28:27 DEBUG 05:28:27 DEBUG 05:28:27 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:28:27 DEBUG }, 05:28:27 DEBUG "value": [ 05:28:27 DEBUG 1712723307, 05:28:27 DEBUG "0" 05:28:27 DEBUG ] 05:28:27 DEBUG } 05:28:27 DEBUG ] 05:28:27 DEBUG } 05:28:27 DEBUG } 05:28:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723307" 05:28:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723307" 05:28:27 INFO [http_cmd]: http status code OK 05:28:27 DEBUG --- status code --- 05:28:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:27 DEBUG --- http response --- 05:28:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:28:27 DEBUG ----- output ----- 05:28:27 DEBUG { 05:28:27 DEBUG "status": "success", 05:28:27 DEBUG "data": { 05:28:27 DEBUG "resultType": "vector", 05:28:27 DEBUG "result": [ 05:28:27 DEBUG { 05:28:27 DEBUG "metric": { 05:28:27 DEBUG "instance": "10.142.0.25:9100" 05:28:27 DEBUG }, 05:28:27 DEBUG "value": [ 05:28:27 DEBUG 05:28:27 DEBUG 05:28:27 DEBUG 05:28:27 DEBUG [...] 05:28:27 DEBUG 05:28:27 DEBUG 05:28:27 DEBUG 05:28:27 DEBUG "instance": "10.142.0.77:9100" 05:28:27 DEBUG }, 05:28:27 DEBUG "value": [ 05:28:27 DEBUG 1712723307, 05:28:27 DEBUG "37684400.8" 05:28:27 DEBUG ] 05:28:27 DEBUG } 05:28:27 INFO [http_cmd]: http status code OK 05:28:27 DEBUG ] 05:28:27 DEBUG --- status code --- 05:28:27 DEBUG } 05:28:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:28:27 DEBUG --- http response --- 05:28:27 DEBUG } 05:28:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:28:27 DEBUG ----- output ----- 05:28:27 DEBUG { 05:28:27 DEBUG "status": "success", 05:28:27 DEBUG "data": { 05:28:27 DEBUG "resultType": "vector", 05:28:27 DEBUG "result": [ 05:28:27 DEBUG { 05:28:27 DEBUG "metric": { 05:28:27 DEBUG "instance": "10.142.0.25:9100" 05:28:27 DEBUG }, 05:28:27 DEBUG "value": [ 05:28:27 DEBUG 05:28:27 DEBUG 05:28:27 DEBUG 05:28:27 DEBUG [...] 05:28:27 DEBUG 05:28:27 DEBUG 05:28:27 DEBUG 05:28:27 DEBUG "instance": "10.142.0.77:9100" 05:28:27 DEBUG }, 05:28:27 DEBUG "value": [ 05:28:27 DEBUG 1712723307, 05:28:27 DEBUG "0.015999999999999657" 05:28:27 DEBUG ] 05:28:27 DEBUG } 05:28:27 DEBUG ] 05:28:27 DEBUG } 05:28:27 DEBUG } 05:28:42 INFO 05:28:42 INFO [loop_until]: kubectl --namespace=xlou top pods 05:28:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:28:42 INFO [loop_until]: OK (rc = 0) 05:28:42 DEBUG --- stdout --- 05:28:42 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 8920m 8803Mi am-654c4fcbcf-9scrj 8705m 8421Mi am-654c4fcbcf-v5qb5 7342m 8398Mi ds-cts-0 7294m 13079Mi ds-cts-1 7607m 13141Mi ds-cts-2 6675m 13334Mi ds-idrepo-0 9664m 5622Mi ds-idrepo-1 319m 5669Mi ds-idrepo-2 321m 5698Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 8m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3676m 1662Mi 05:28:42 DEBUG --- stderr --- 05:28:42 DEBUG 05:28:48 INFO 05:28:48 INFO [loop_until]: kubectl --namespace=xlou top node 05:28:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:28:48 INFO [loop_until]: OK (rc = 0) 05:28:48 DEBUG --- stdout --- 05:28:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 97m 0% 2302Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8154m 51% 9279Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 86m 0% 3260Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10811m 68% 6516Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9589m 60% 9309Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9945m 62% 9643Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 80m 0% 855Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9787m 61% 6280Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 7484m 47% 13709Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 6474m 40% 13936Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 383m 2% 6369Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7636m 48% 13796Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 374m 2% 6330Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3709m 23% 5341Mi 9% 05:28:48 DEBUG --- stderr --- 05:28:48 DEBUG 05:29:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712723363" 05:29:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723363" 05:29:23 ERROR [http_cmd]: ERROR 05:29:23 INFO [http_cmd]: http status code OK 05:29:23 ERROR --- status code --- 05:29:23 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:29:23 DEBUG --- status code --- 05:29:23 ERROR --- http response --- 05:29:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:23 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:29:23 DEBUG --- http response --- 05:29:23 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:29:23 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:29:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723363" 05:29:23 INFO [http_cmd]: http status code OK 05:29:23 DEBUG --- status code --- 05:29:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:23 DEBUG --- http response --- 05:29:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:29:23 DEBUG ----- output ----- 05:29:23 DEBUG { 05:29:23 DEBUG "status": "success", 05:29:23 DEBUG "data": { 05:29:23 DEBUG "resultType": "vector", 05:29:23 DEBUG "result": [ 05:29:23 DEBUG { 05:29:23 DEBUG "metric": { 05:29:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:29:23 DEBUG }, 05:29:23 DEBUG "value": [ 05:29:23 DEBUG 05:29:23 DEBUG 05:29:23 DEBUG 05:29:23 DEBUG [...] 05:29:23 DEBUG 05:29:23 DEBUG 05:29:23 DEBUG 05:29:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:29:23 DEBUG }, 05:29:23 DEBUG "value": [ 05:29:23 DEBUG 1712723363, 05:29:23 DEBUG "0" 05:29:23 DEBUG ] 05:29:23 DEBUG } 05:29:23 DEBUG ] 05:29:23 DEBUG } 05:29:23 DEBUG } 05:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723364" 05:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723364" 05:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723364" 05:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723364" 05:29:24 INFO [http_cmd]: http status code OK 05:29:24 INFO [http_cmd]: http status code OK 05:29:24 DEBUG --- status code --- 05:29:24 DEBUG --- status code --- 05:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:24 DEBUG --- http response --- 05:29:24 DEBUG --- http response --- 05:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:29:24 DEBUG ----- output ----- 05:29:24 DEBUG ----- output ----- 05:29:24 DEBUG { 05:29:24 DEBUG { 05:29:24 DEBUG "status": "success", 05:29:24 DEBUG "status": "success", 05:29:24 DEBUG "data": { 05:29:24 DEBUG "data": { 05:29:24 DEBUG "resultType": "vector", 05:29:24 DEBUG "resultType": "vector", 05:29:24 DEBUG "result": [ 05:29:24 DEBUG "result": [ 05:29:24 DEBUG { 05:29:24 DEBUG { 05:29:24 DEBUG "metric": { 05:29:24 DEBUG "metric": { 05:29:24 DEBUG "pod": "ds-idrepo-0" 05:29:24 DEBUG "pod": "ds-idrepo-0" 05:29:24 DEBUG }, 05:29:24 DEBUG }, 05:29:24 DEBUG "value": [ 05:29:24 DEBUG "value": [ 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG [...] 05:29:24 DEBUG [...] 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG "pod": "ds-cts-2" 05:29:24 DEBUG "pod": "ds-cts-2" 05:29:24 DEBUG }, 05:29:24 DEBUG }, 05:29:24 DEBUG "value": [ 05:29:24 DEBUG "value": [ 05:29:24 DEBUG 1712723364, 05:29:24 DEBUG 1712723364, 05:29:24 DEBUG "0" 05:29:24 DEBUG "30" 05:29:24 DEBUG ] 05:29:24 DEBUG ] 05:29:24 DEBUG } 05:29:24 DEBUG } 05:29:24 DEBUG ] 05:29:24 DEBUG ] 05:29:24 DEBUG } 05:29:24 DEBUG } 05:29:24 DEBUG } 05:29:24 DEBUG } 05:29:24 INFO [http_cmd]: http status code OK 05:29:24 DEBUG --- status code --- 05:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:24 DEBUG --- http response --- 05:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:29:24 DEBUG ----- output ----- 05:29:24 DEBUG { 05:29:24 DEBUG "status": "success", 05:29:24 DEBUG "data": { 05:29:24 DEBUG "resultType": "vector", 05:29:24 DEBUG "result": [ 05:29:24 DEBUG { 05:29:24 DEBUG "metric": { 05:29:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:29:24 DEBUG }, 05:29:24 DEBUG "value": [ 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG [...] 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:29:24 DEBUG }, 05:29:24 DEBUG "value": [ 05:29:24 DEBUG 1712723364, 05:29:24 DEBUG "1260.5666666666666" 05:29:24 DEBUG ] 05:29:24 DEBUG } 05:29:24 DEBUG ] 05:29:24 DEBUG } 05:29:24 DEBUG } 05:29:24 INFO [http_cmd]: http status code OK 05:29:24 DEBUG --- status code --- 05:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:24 DEBUG --- http response --- 05:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:29:24 DEBUG ----- output ----- 05:29:24 DEBUG { 05:29:24 DEBUG "status": "success", 05:29:24 DEBUG "data": { 05:29:24 DEBUG "resultType": "vector", 05:29:24 DEBUG "result": [ 05:29:24 DEBUG { 05:29:24 DEBUG "metric": { 05:29:24 DEBUG "instance": "10.142.0.25:9100" 05:29:24 DEBUG }, 05:29:24 DEBUG "value": [ 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG [...] 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG "instance": "10.142.0.77:9100" 05:29:24 DEBUG }, 05:29:24 DEBUG "value": [ 05:29:24 DEBUG 1712723364, 05:29:24 DEBUG "286912656384" 05:29:24 DEBUG ] 05:29:24 DEBUG } 05:29:24 DEBUG ] 05:29:24 DEBUG } 05:29:24 DEBUG } 05:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723364" 05:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723364" 05:29:24 INFO [http_cmd]: http status code OK 05:29:24 DEBUG --- status code --- 05:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:24 DEBUG --- http response --- 05:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:29:24 DEBUG ----- output ----- 05:29:24 DEBUG { 05:29:24 DEBUG "status": "success", 05:29:24 DEBUG "data": { 05:29:24 DEBUG "resultType": "vector", 05:29:24 DEBUG "result": [ 05:29:24 DEBUG { 05:29:24 DEBUG "metric": { 05:29:24 DEBUG "instance": "10.142.0.25:9100" 05:29:24 DEBUG }, 05:29:24 DEBUG "value": [ 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG [...] 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG "instance": "10.142.0.77:9100" 05:29:24 DEBUG }, 05:29:24 DEBUG "value": [ 05:29:24 DEBUG 1712723364, 05:29:24 DEBUG "37694244.63333333" 05:29:24 DEBUG ] 05:29:24 DEBUG } 05:29:24 DEBUG ] 05:29:24 DEBUG } 05:29:24 DEBUG } 05:29:24 INFO [http_cmd]: http status code OK 05:29:24 DEBUG --- status code --- 05:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:24 DEBUG --- http response --- 05:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:29:24 DEBUG ----- output ----- 05:29:24 DEBUG { 05:29:24 DEBUG "status": "success", 05:29:24 DEBUG "data": { 05:29:24 DEBUG "resultType": "vector", 05:29:24 DEBUG "result": [ 05:29:24 DEBUG { 05:29:24 DEBUG "metric": { 05:29:24 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:29:24 DEBUG }, 05:29:24 DEBUG "value": [ 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG [...] 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:29:24 DEBUG }, 05:29:24 DEBUG "value": [ 05:29:24 DEBUG 1712723364, 05:29:24 DEBUG "0" 05:29:24 DEBUG ] 05:29:24 DEBUG } 05:29:24 DEBUG ] 05:29:24 DEBUG } 05:29:24 DEBUG } 05:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723364" 05:29:24 INFO [http_cmd]: http status code OK 05:29:24 DEBUG --- status code --- 05:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:24 DEBUG --- http response --- 05:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:29:24 DEBUG ----- output ----- 05:29:24 DEBUG { 05:29:24 DEBUG "status": "success", 05:29:24 DEBUG "data": { 05:29:24 DEBUG "resultType": "vector", 05:29:24 DEBUG "result": [ 05:29:24 DEBUG { 05:29:24 DEBUG "metric": { 05:29:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:29:24 DEBUG }, 05:29:24 DEBUG "value": [ 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG [...] 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:29:24 DEBUG }, 05:29:24 DEBUG "value": [ 05:29:24 DEBUG 1712723364, 05:29:24 DEBUG "226401.5" 05:29:24 DEBUG ] 05:29:24 DEBUG } 05:29:24 DEBUG ] 05:29:24 DEBUG } 05:29:24 DEBUG } 05:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723364" 05:29:24 INFO [http_cmd]: http status code OK 05:29:24 DEBUG --- status code --- 05:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:24 DEBUG --- http response --- 05:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:29:24 DEBUG ----- output ----- 05:29:24 DEBUG { 05:29:24 DEBUG "status": "success", 05:29:24 DEBUG "data": { 05:29:24 DEBUG "resultType": "vector", 05:29:24 DEBUG "result": [ 05:29:24 DEBUG { 05:29:24 DEBUG "metric": { 05:29:24 DEBUG "pod": "ds-idrepo-0" 05:29:24 DEBUG }, 05:29:24 DEBUG "value": [ 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG [...] 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG 05:29:24 DEBUG "pod": "ds-idrepo-1" 05:29:24 DEBUG }, 05:29:24 DEBUG "value": [ 05:29:24 DEBUG 1712723364, 05:29:24 DEBUG "0" 05:29:24 DEBUG ] 05:29:24 DEBUG } 05:29:24 DEBUG ] 05:29:24 DEBUG } 05:29:24 DEBUG } 05:29:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723365" 05:29:25 INFO [http_cmd]: http status code OK 05:29:25 DEBUG --- status code --- 05:29:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:25 DEBUG --- http response --- 05:29:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:29:25 DEBUG ----- output ----- 05:29:25 DEBUG { 05:29:25 DEBUG "status": "success", 05:29:25 DEBUG "data": { 05:29:25 DEBUG "resultType": "vector", 05:29:25 DEBUG "result": [ 05:29:25 DEBUG { 05:29:25 DEBUG "metric": { 05:29:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:29:25 DEBUG }, 05:29:25 DEBUG "value": [ 05:29:25 DEBUG 05:29:25 DEBUG 05:29:25 DEBUG 05:29:25 DEBUG [...] 05:29:25 DEBUG 05:29:25 DEBUG 05:29:25 DEBUG 05:29:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:29:25 DEBUG }, 05:29:25 DEBUG "value": [ 05:29:25 DEBUG 1712723365, 05:29:25 DEBUG "1260.3666666666666" 05:29:25 DEBUG ] 05:29:25 DEBUG } 05:29:25 DEBUG ] 05:29:25 DEBUG } 05:29:25 DEBUG } 05:29:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723365" 05:29:25 INFO [http_cmd]: http status code OK 05:29:25 DEBUG --- status code --- 05:29:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:25 DEBUG --- http response --- 05:29:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:29:25 DEBUG ----- output ----- 05:29:25 DEBUG { 05:29:25 DEBUG "status": "success", 05:29:25 DEBUG "data": { 05:29:25 DEBUG "resultType": "vector", 05:29:25 DEBUG "result": [ 05:29:25 DEBUG { 05:29:25 DEBUG "metric": { 05:29:25 DEBUG "instance": "10.142.0.25:9100" 05:29:25 DEBUG }, 05:29:25 DEBUG "value": [ 05:29:25 DEBUG 05:29:25 DEBUG 05:29:25 DEBUG 05:29:25 DEBUG [...] 05:29:25 DEBUG 05:29:25 DEBUG 05:29:25 DEBUG 05:29:25 DEBUG "instance": "10.142.0.77:9100" 05:29:25 DEBUG }, 05:29:25 DEBUG "value": [ 05:29:25 DEBUG 1712723365, 05:29:25 DEBUG "0.1446999999999985" 05:29:25 DEBUG ] 05:29:25 DEBUG } 05:29:25 DEBUG ] 05:29:25 DEBUG } 05:29:25 DEBUG } 05:29:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723366" 05:29:26 INFO [http_cmd]: http status code OK 05:29:26 DEBUG --- status code --- 05:29:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:26 DEBUG --- http response --- 05:29:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:29:26 DEBUG ----- output ----- 05:29:26 DEBUG { 05:29:26 DEBUG "status": "success", 05:29:26 DEBUG "data": { 05:29:26 DEBUG "resultType": "vector", 05:29:26 DEBUG "result": [ 05:29:26 DEBUG { 05:29:26 DEBUG "metric": { 05:29:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:29:26 DEBUG }, 05:29:26 DEBUG "value": [ 05:29:26 DEBUG 05:29:26 DEBUG 05:29:26 DEBUG 05:29:26 DEBUG [...] 05:29:26 DEBUG 05:29:26 DEBUG 05:29:26 DEBUG 05:29:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:29:26 DEBUG }, 05:29:26 DEBUG "value": [ 05:29:26 DEBUG 1712723366, 05:29:26 DEBUG "14536.980614055647" 05:29:26 DEBUG ] 05:29:26 DEBUG } 05:29:26 DEBUG ] 05:29:26 DEBUG } 05:29:26 DEBUG } 05:29:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712723366" 05:29:26 INFO [http_cmd]: http status code OK 05:29:26 DEBUG --- status code --- 05:29:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:26 DEBUG --- http response --- 05:29:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:29:26 DEBUG ----- output ----- 05:29:26 DEBUG { 05:29:26 DEBUG "status": "success", 05:29:26 DEBUG "data": { 05:29:26 DEBUG "resultType": "vector", 05:29:26 DEBUG "result": [ 05:29:26 DEBUG { 05:29:26 DEBUG "metric": { 05:29:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:29:26 DEBUG }, 05:29:26 DEBUG "value": [ 05:29:26 DEBUG 05:29:26 DEBUG 05:29:26 DEBUG 05:29:26 DEBUG [...] 05:29:26 DEBUG 05:29:26 DEBUG 05:29:26 DEBUG 05:29:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:29:26 DEBUG }, 05:29:26 DEBUG "value": [ 05:29:26 DEBUG 1712723366, 05:29:26 DEBUG "3.668256373495732" 05:29:26 DEBUG ] 05:29:26 DEBUG } 05:29:26 DEBUG ] 05:29:26 DEBUG } 05:29:26 DEBUG } 05:29:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712723366" 05:29:26 INFO [http_cmd]: http status code OK 05:29:26 DEBUG --- status code --- 05:29:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:26 DEBUG --- http response --- 05:29:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:29:26 DEBUG ----- output ----- 05:29:26 DEBUG { 05:29:26 DEBUG "status": "success", 05:29:26 DEBUG "data": { 05:29:26 DEBUG "resultType": "vector", 05:29:26 DEBUG "result": [ 05:29:26 DEBUG { 05:29:26 DEBUG "metric": { 05:29:26 DEBUG "pod": "ds-idrepo-0" 05:29:26 DEBUG }, 05:29:26 DEBUG "value": [ 05:29:26 DEBUG 05:29:26 DEBUG 05:29:26 DEBUG 05:29:26 DEBUG [...] 05:29:26 DEBUG 05:29:26 DEBUG 05:29:26 DEBUG 05:29:26 DEBUG "pod": "ds-cts-2" 05:29:26 DEBUG }, 05:29:26 DEBUG "value": [ 05:29:26 DEBUG 1712723366, 05:29:26 DEBUG "0" 05:29:26 DEBUG ] 05:29:26 DEBUG } 05:29:26 DEBUG ] 05:29:26 DEBUG } 05:29:26 DEBUG } 05:29:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723366" 05:29:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723366" 05:29:26 INFO [http_cmd]: http status code OK 05:29:26 INFO [http_cmd]: http status code OK 05:29:26 DEBUG --- status code --- 05:29:26 DEBUG --- status code --- 05:29:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:26 DEBUG --- http response --- 05:29:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:29:26 DEBUG ----- output ----- 05:29:26 DEBUG { 05:29:26 DEBUG "status": "success", 05:29:26 DEBUG "data": { 05:29:26 DEBUG "resultType": "vector", 05:29:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:26 DEBUG "result": [ 05:29:26 DEBUG --- http response --- 05:29:26 DEBUG { 05:29:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:29:26 DEBUG "metric": { 05:29:26 DEBUG "pod": "ds-idrepo-2" 05:29:26 DEBUG ----- output ----- 05:29:26 DEBUG }, 05:29:26 DEBUG { 05:29:26 DEBUG "value": [ 05:29:26 DEBUG "status": "success", 05:29:26 DEBUG 05:29:26 DEBUG "data": { 05:29:26 DEBUG 05:29:26 DEBUG "resultType": "vector", 05:29:26 DEBUG 05:29:26 DEBUG "result": [ 05:29:26 DEBUG [...] 05:29:26 DEBUG { 05:29:26 DEBUG 05:29:26 DEBUG "metric": { 05:29:26 DEBUG 05:29:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:29:26 DEBUG 05:29:26 DEBUG }, 05:29:26 DEBUG "pod": "ds-cts-2" 05:29:26 DEBUG "value": [ 05:29:26 DEBUG }, 05:29:26 DEBUG 05:29:26 DEBUG "value": [ 05:29:26 DEBUG 05:29:26 DEBUG 1712723366, 05:29:26 DEBUG 05:29:26 DEBUG "0" 05:29:26 DEBUG [...] 05:29:26 DEBUG ] 05:29:26 DEBUG 05:29:26 DEBUG } 05:29:26 DEBUG 05:29:26 DEBUG ] 05:29:26 DEBUG 05:29:26 DEBUG } 05:29:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:29:26 DEBUG } 05:29:26 DEBUG }, 05:29:26 DEBUG "value": [ 05:29:26 DEBUG 1712723366, 05:29:26 DEBUG "7.449225587437491" 05:29:26 DEBUG ] 05:29:26 DEBUG } 05:29:26 DEBUG ] 05:29:26 DEBUG } 05:29:26 DEBUG } 05:29:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723366" 05:29:26 INFO [http_cmd]: http status code OK 05:29:26 DEBUG --- status code --- 05:29:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:26 DEBUG --- http response --- 05:29:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:29:26 DEBUG ----- output ----- 05:29:26 DEBUG { 05:29:26 DEBUG "status": "success", 05:29:26 DEBUG "data": { 05:29:26 DEBUG "resultType": "vector", 05:29:26 DEBUG "result": [ 05:29:26 DEBUG { 05:29:26 DEBUG "metric": { 05:29:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:29:26 DEBUG }, 05:29:26 DEBUG "value": [ 05:29:26 DEBUG 05:29:26 DEBUG 05:29:26 DEBUG 05:29:26 DEBUG [...] 05:29:26 DEBUG 05:29:26 DEBUG 05:29:26 DEBUG 05:29:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:29:26 DEBUG }, 05:29:26 DEBUG "value": [ 05:29:26 DEBUG 1712723366, 05:29:26 DEBUG "1260.1666666666667" 05:29:26 DEBUG ] 05:29:26 DEBUG } 05:29:26 DEBUG ] 05:29:26 DEBUG } 05:29:26 DEBUG } 05:29:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712723366" 05:29:26 INFO [http_cmd]: http status code OK 05:29:26 DEBUG --- status code --- 05:29:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:26 DEBUG --- http response --- 05:29:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:29:26 DEBUG ----- output ----- 05:29:26 DEBUG { 05:29:26 DEBUG "status": "success", 05:29:26 DEBUG "data": { 05:29:26 DEBUG "resultType": "vector", 05:29:26 DEBUG "result": [ 05:29:26 DEBUG { 05:29:26 DEBUG "metric": { 05:29:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:29:26 DEBUG }, 05:29:26 DEBUG "value": [ 05:29:26 DEBUG 05:29:26 DEBUG 05:29:26 DEBUG 05:29:26 DEBUG [...] 05:29:26 DEBUG 05:29:26 DEBUG 05:29:26 DEBUG 05:29:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:29:26 DEBUG }, 05:29:26 DEBUG "value": [ 05:29:26 DEBUG 1712723366, 05:29:26 DEBUG "0.042" 05:29:26 DEBUG ] 05:29:26 DEBUG } 05:29:26 DEBUG ] 05:29:26 DEBUG } 05:29:26 DEBUG } 05:29:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723367" 05:29:27 INFO [http_cmd]: http status code OK 05:29:27 DEBUG --- status code --- 05:29:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:27 DEBUG --- http response --- 05:29:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:29:27 DEBUG ----- output ----- 05:29:27 DEBUG { 05:29:27 DEBUG "status": "success", 05:29:27 DEBUG "data": { 05:29:27 DEBUG "resultType": "vector", 05:29:27 DEBUG "result": [ 05:29:27 DEBUG { 05:29:27 DEBUG "metric": { 05:29:27 DEBUG "pod": "ds-idrepo-0" 05:29:27 DEBUG }, 05:29:27 DEBUG "value": [ 05:29:27 DEBUG 05:29:27 DEBUG 05:29:27 DEBUG 05:29:27 DEBUG [...] 05:29:27 DEBUG 05:29:27 DEBUG 05:29:27 DEBUG 05:29:27 DEBUG "pod": "ds-idrepo-1" 05:29:27 DEBUG }, 05:29:27 DEBUG "value": [ 05:29:27 DEBUG 1712723367, 05:29:27 DEBUG "0" 05:29:27 DEBUG ] 05:29:27 DEBUG } 05:29:27 DEBUG ] 05:29:27 DEBUG } 05:29:27 DEBUG } 05:29:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723367" 05:29:27 INFO [http_cmd]: http status code OK 05:29:27 DEBUG --- status code --- 05:29:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:27 DEBUG --- http response --- 05:29:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:29:27 DEBUG ----- output ----- 05:29:27 DEBUG { 05:29:27 DEBUG "status": "success", 05:29:27 DEBUG "data": { 05:29:27 DEBUG "resultType": "vector", 05:29:27 DEBUG "result": [ 05:29:27 DEBUG { 05:29:27 DEBUG "metric": { 05:29:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:29:27 DEBUG }, 05:29:27 DEBUG "value": [ 05:29:27 DEBUG 05:29:27 DEBUG 05:29:27 DEBUG 05:29:27 DEBUG [...] 05:29:27 DEBUG 05:29:27 DEBUG 05:29:27 DEBUG 05:29:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:29:27 DEBUG }, 05:29:27 DEBUG "value": [ 05:29:27 DEBUG 1712723367, 05:29:27 DEBUG "1245.1" 05:29:27 DEBUG ] 05:29:27 DEBUG } 05:29:27 DEBUG ] 05:29:27 DEBUG } 05:29:27 DEBUG } 05:29:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723367" 05:29:27 INFO [http_cmd]: http status code OK 05:29:27 DEBUG --- status code --- 05:29:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:27 DEBUG --- http response --- 05:29:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:29:27 DEBUG ----- output ----- 05:29:27 DEBUG { 05:29:27 DEBUG "status": "success", 05:29:27 DEBUG "data": { 05:29:27 DEBUG "resultType": "vector", 05:29:27 DEBUG "result": [ 05:29:27 DEBUG { 05:29:27 DEBUG "metric": { 05:29:27 DEBUG "pod": "ds-idrepo-0" 05:29:27 DEBUG }, 05:29:27 DEBUG "value": [ 05:29:27 DEBUG 05:29:27 DEBUG 05:29:27 DEBUG 05:29:27 DEBUG [...] 05:29:27 DEBUG 05:29:27 DEBUG 05:29:27 DEBUG 05:29:27 DEBUG "pod": "ds-idrepo-1" 05:29:27 DEBUG }, 05:29:27 DEBUG "value": [ 05:29:27 DEBUG 1712723367, 05:29:27 DEBUG "0" 05:29:27 DEBUG ] 05:29:27 DEBUG } 05:29:27 DEBUG ] 05:29:27 DEBUG } 05:29:27 DEBUG } 05:29:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712723367" 05:29:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723367" 05:29:27 INFO [http_cmd]: http status code OK 05:29:27 DEBUG --- status code --- 05:29:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:27 DEBUG --- http response --- 05:29:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:29:27 DEBUG ----- output ----- 05:29:27 DEBUG { 05:29:27 DEBUG "status": "success", 05:29:27 DEBUG "data": { 05:29:27 DEBUG "resultType": "vector", 05:29:27 DEBUG "result": [ 05:29:27 DEBUG { 05:29:27 DEBUG "metric": { 05:29:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:29:27 DEBUG }, 05:29:27 DEBUG "value": [ 05:29:27 DEBUG 05:29:27 DEBUG 05:29:27 DEBUG 05:29:27 DEBUG [...] 05:29:27 DEBUG 05:29:27 DEBUG 05:29:27 DEBUG 05:29:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:29:27 DEBUG }, 05:29:27 DEBUG "value": [ 05:29:27 DEBUG 1712723367, 05:29:27 DEBUG "2491019264" 05:29:27 DEBUG ] 05:29:27 DEBUG } 05:29:27 DEBUG ] 05:29:27 DEBUG } 05:29:27 DEBUG } 05:29:27 INFO [http_cmd]: http status code OK 05:29:27 DEBUG --- status code --- 05:29:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:27 DEBUG --- http response --- 05:29:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:29:27 DEBUG ----- output ----- 05:29:27 DEBUG { 05:29:27 DEBUG "status": "success", 05:29:27 DEBUG "data": { 05:29:27 DEBUG "resultType": "vector", 05:29:27 DEBUG "result": [ 05:29:27 DEBUG { 05:29:27 DEBUG "metric": { 05:29:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:29:27 DEBUG }, 05:29:27 DEBUG "value": [ 05:29:27 DEBUG 05:29:27 DEBUG 05:29:27 DEBUG 05:29:27 DEBUG [...] 05:29:27 DEBUG 05:29:27 DEBUG 05:29:27 DEBUG 05:29:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:29:27 DEBUG }, 05:29:27 DEBUG "value": [ 05:29:27 DEBUG 1712723367, 05:29:27 DEBUG "0" 05:29:27 DEBUG ] 05:29:27 DEBUG } 05:29:27 DEBUG ] 05:29:27 DEBUG } 05:29:27 DEBUG } 05:29:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723367" 05:29:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723367" 05:29:27 INFO [http_cmd]: http status code OK 05:29:27 DEBUG --- status code --- 05:29:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:27 DEBUG --- http response --- 05:29:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:29:27 DEBUG ----- output ----- 05:29:27 DEBUG { 05:29:27 DEBUG "status": "success", 05:29:27 DEBUG "data": { 05:29:27 DEBUG "resultType": "vector", 05:29:27 DEBUG "result": [ 05:29:27 DEBUG { 05:29:27 DEBUG "metric": { 05:29:27 DEBUG "instance": "10.142.0.25:9100" 05:29:27 DEBUG }, 05:29:27 DEBUG "value": [ 05:29:27 DEBUG 05:29:27 DEBUG 05:29:27 DEBUG 05:29:27 DEBUG [...] 05:29:27 DEBUG 05:29:27 DEBUG 05:29:27 DEBUG 05:29:27 DEBUG "instance": "10.142.0.77:9100" 05:29:27 DEBUG }, 05:29:27 DEBUG "value": [ 05:29:27 DEBUG 1712723367, 05:29:27 DEBUG "37693970.666666664" 05:29:27 DEBUG ] 05:29:27 DEBUG } 05:29:27 DEBUG ] 05:29:27 DEBUG } 05:29:27 DEBUG } 05:29:27 INFO [http_cmd]: http status code OK 05:29:27 DEBUG --- status code --- 05:29:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:29:27 DEBUG --- http response --- 05:29:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:29:27 DEBUG ----- output ----- 05:29:27 DEBUG { 05:29:27 DEBUG "status": "success", 05:29:27 DEBUG "data": { 05:29:27 DEBUG "resultType": "vector", 05:29:27 DEBUG "result": [ 05:29:27 DEBUG { 05:29:27 DEBUG "metric": { 05:29:27 DEBUG "instance": "10.142.0.25:9100" 05:29:27 DEBUG }, 05:29:27 DEBUG "value": [ 05:29:27 DEBUG 05:29:27 DEBUG 05:29:27 DEBUG 05:29:27 DEBUG [...] 05:29:27 DEBUG 05:29:27 DEBUG 05:29:27 DEBUG 05:29:27 DEBUG "instance": "10.142.0.77:9100" 05:29:27 DEBUG }, 05:29:27 DEBUG "value": [ 05:29:27 DEBUG 1712723367, 05:29:27 DEBUG "0.049666666666666956" 05:29:27 DEBUG ] 05:29:27 DEBUG } 05:29:27 DEBUG ] 05:29:27 DEBUG } 05:29:27 DEBUG } 05:29:42 INFO 05:29:42 INFO [loop_until]: kubectl --namespace=xlou top pods 05:29:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:29:42 INFO [loop_until]: OK (rc = 0) 05:29:42 DEBUG --- stdout --- 05:29:42 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 9005m 8804Mi am-654c4fcbcf-9scrj 8579m 8421Mi am-654c4fcbcf-v5qb5 7248m 8400Mi ds-cts-0 7274m 13302Mi ds-cts-1 7381m 13268Mi ds-cts-2 6463m 12908Mi ds-idrepo-0 9628m 5621Mi ds-idrepo-1 316m 5669Mi ds-idrepo-2 322m 5698Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2212Mi lodemon-b776b7d88-thrqf 9m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3562m 1663Mi 05:29:42 DEBUG --- stderr --- 05:29:42 DEBUG 05:29:48 INFO 05:29:48 INFO [loop_until]: kubectl --namespace=xlou top node 05:29:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:29:48 INFO [loop_until]: OK (rc = 0) 05:29:48 DEBUG --- stdout --- 05:29:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 93m 0% 2307Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8139m 51% 9279Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 86m 0% 3260Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10819m 68% 6494Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9552m 60% 9310Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9895m 62% 9646Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 78m 0% 851Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9776m 61% 6280Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 7438m 46% 13944Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 6541m 41% 13524Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 386m 2% 6366Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7490m 47% 13944Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 427m 2% 6329Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3704m 23% 5343Mi 9% 05:29:48 DEBUG --- stderr --- 05:29:48 DEBUG 05:30:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712723423" 05:30:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723423" 05:30:23 ERROR [http_cmd]: ERROR 05:30:23 INFO [http_cmd]: http status code OK 05:30:23 ERROR --- status code --- 05:30:23 DEBUG --- status code --- 05:30:23 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:30:23 ERROR --- http response --- 05:30:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:23 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:30:23 DEBUG --- http response --- 05:30:23 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:30:23 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:30:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723423" 05:30:23 INFO [http_cmd]: http status code OK 05:30:23 DEBUG --- status code --- 05:30:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:23 DEBUG --- http response --- 05:30:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:30:23 DEBUG ----- output ----- 05:30:23 DEBUG { 05:30:23 DEBUG "status": "success", 05:30:23 DEBUG "data": { 05:30:23 DEBUG "resultType": "vector", 05:30:23 DEBUG "result": [ 05:30:23 DEBUG { 05:30:23 DEBUG "metric": { 05:30:23 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:30:23 DEBUG }, 05:30:23 DEBUG "value": [ 05:30:23 DEBUG 05:30:23 DEBUG 05:30:23 DEBUG 05:30:23 DEBUG [...] 05:30:23 DEBUG 05:30:23 DEBUG 05:30:23 DEBUG 05:30:23 DEBUG "pod": "am-654c4fcbcf-487cv" 05:30:23 DEBUG }, 05:30:23 DEBUG "value": [ 05:30:23 DEBUG 1712723423, 05:30:23 DEBUG "0" 05:30:23 DEBUG ] 05:30:23 DEBUG } 05:30:23 DEBUG ] 05:30:23 DEBUG } 05:30:23 DEBUG } 05:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723424" 05:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723424" 05:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723424" 05:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723424" 05:30:24 INFO [http_cmd]: http status code OK 05:30:24 INFO [http_cmd]: http status code OK 05:30:24 DEBUG --- status code --- 05:30:24 INFO [http_cmd]: http status code OK 05:30:24 INFO [http_cmd]: http status code OK 05:30:24 DEBUG --- status code --- 05:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:24 DEBUG --- status code --- 05:30:24 DEBUG --- status code --- 05:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:24 DEBUG --- http response --- 05:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:24 DEBUG --- http response --- 05:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:30:24 DEBUG --- http response --- 05:30:24 DEBUG --- http response --- 05:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:30:24 DEBUG ----- output ----- 05:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:30:24 DEBUG ----- output ----- 05:30:24 DEBUG { 05:30:24 DEBUG "status": "success", 05:30:24 DEBUG "data": { 05:30:24 DEBUG { 05:30:24 DEBUG ----- output ----- 05:30:24 DEBUG { 05:30:24 DEBUG "status": "success", 05:30:24 DEBUG "data": { 05:30:24 DEBUG "resultType": "vector", 05:30:24 DEBUG "status": "success", 05:30:24 DEBUG "result": [ 05:30:24 DEBUG { 05:30:24 DEBUG "metric": { 05:30:24 DEBUG "instance": "10.142.0.25:9100" 05:30:24 DEBUG "data": { 05:30:24 DEBUG }, 05:30:24 DEBUG "resultType": "vector", 05:30:24 DEBUG "result": [ 05:30:24 DEBUG { 05:30:24 DEBUG "metric": { 05:30:24 DEBUG "pod": "ds-idrepo-0" 05:30:24 DEBUG }, 05:30:24 DEBUG "resultType": "vector", 05:30:24 DEBUG ----- output ----- 05:30:24 DEBUG "value": [ 05:30:24 DEBUG "value": [ 05:30:24 DEBUG "result": [ 05:30:24 DEBUG { 05:30:24 DEBUG 05:30:24 DEBUG 05:30:24 DEBUG { 05:30:24 DEBUG "status": "success", 05:30:24 DEBUG 05:30:24 DEBUG 05:30:24 DEBUG "metric": { 05:30:24 DEBUG "data": { 05:30:24 DEBUG 05:30:24 DEBUG 05:30:24 DEBUG "pod": "ds-idrepo-0" 05:30:24 DEBUG "resultType": "vector", 05:30:24 DEBUG [...] 05:30:24 DEBUG [...] 05:30:24 DEBUG }, 05:30:24 DEBUG "result": [ 05:30:24 DEBUG { 05:30:24 DEBUG 05:30:24 DEBUG "value": [ 05:30:24 DEBUG 05:30:24 DEBUG "metric": { 05:30:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:30:24 DEBUG 05:30:24 DEBUG 05:30:24 DEBUG }, 05:30:24 DEBUG 05:30:24 DEBUG 05:30:24 DEBUG 05:30:24 DEBUG "value": [ 05:30:24 DEBUG "pod": "ds-cts-2" 05:30:24 DEBUG 05:30:24 DEBUG "instance": "10.142.0.77:9100" 05:30:24 DEBUG 05:30:24 DEBUG }, 05:30:24 DEBUG 05:30:24 DEBUG }, 05:30:24 DEBUG 05:30:24 DEBUG "value": [ 05:30:24 DEBUG [...] 05:30:24 DEBUG "value": [ 05:30:24 DEBUG 05:30:24 DEBUG 1712723424, 05:30:24 DEBUG 05:30:24 DEBUG 1712723424, 05:30:24 DEBUG "288668537856" 05:30:24 DEBUG "0" 05:30:24 DEBUG 05:30:24 DEBUG 05:30:24 DEBUG [...] 05:30:24 DEBUG ] 05:30:24 DEBUG "pod": "ds-cts-2" 05:30:24 DEBUG ] 05:30:24 DEBUG 05:30:24 DEBUG 05:30:24 DEBUG }, 05:30:24 DEBUG } 05:30:24 DEBUG 05:30:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:30:24 DEBUG "value": [ 05:30:24 DEBUG ] 05:30:24 DEBUG } 05:30:24 DEBUG ] 05:30:24 DEBUG 1712723424, 05:30:24 DEBUG } 05:30:24 DEBUG } 05:30:24 DEBUG } 05:30:24 DEBUG "29.933333333333334" 05:30:24 DEBUG }, 05:30:24 DEBUG "value": [ 05:30:24 DEBUG ] 05:30:24 DEBUG 1712723424, 05:30:24 DEBUG } 05:30:24 DEBUG } 05:30:24 DEBUG "1264.5" 05:30:24 DEBUG ] 05:30:24 DEBUG ] 05:30:24 DEBUG } 05:30:24 DEBUG } 05:30:24 DEBUG } 05:30:24 DEBUG ] 05:30:24 DEBUG } 05:30:24 DEBUG } 05:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723424" 05:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723424" 05:30:24 INFO [http_cmd]: http status code OK 05:30:24 DEBUG --- status code --- 05:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:24 DEBUG --- http response --- 05:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:30:24 DEBUG ----- output ----- 05:30:24 DEBUG { 05:30:24 DEBUG "status": "success", 05:30:24 DEBUG "data": { 05:30:24 DEBUG "resultType": "vector", 05:30:24 DEBUG "result": [ 05:30:24 DEBUG { 05:30:24 DEBUG "metric": { 05:30:24 DEBUG "instance": "10.142.0.25:9100" 05:30:24 DEBUG }, 05:30:24 DEBUG "value": [ 05:30:24 DEBUG 05:30:24 DEBUG 05:30:24 DEBUG 05:30:24 DEBUG [...] 05:30:24 DEBUG 05:30:24 DEBUG 05:30:24 DEBUG 05:30:24 DEBUG "instance": "10.142.0.77:9100" 05:30:24 DEBUG }, 05:30:24 INFO [http_cmd]: http status code OK 05:30:24 DEBUG "value": [ 05:30:24 DEBUG --- status code --- 05:30:24 DEBUG 1712723424, 05:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:24 DEBUG "37564519.96666667" 05:30:24 DEBUG --- http response --- 05:30:24 DEBUG ] 05:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 05:30:24 DEBUG } 05:30:24 DEBUG ----- output ----- 05:30:24 DEBUG ] 05:30:24 DEBUG { 05:30:24 DEBUG } 05:30:24 DEBUG "status": "success", 05:30:24 DEBUG } 05:30:24 DEBUG "data": { 05:30:24 DEBUG "resultType": "vector", 05:30:24 DEBUG "result": [ 05:30:24 DEBUG { 05:30:24 DEBUG "metric": { 05:30:24 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:30:24 DEBUG }, 05:30:24 DEBUG "value": [ 05:30:24 DEBUG 05:30:24 DEBUG 05:30:24 DEBUG 05:30:24 DEBUG [...] 05:30:24 DEBUG 05:30:24 DEBUG 05:30:24 DEBUG 05:30:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:30:24 DEBUG }, 05:30:24 DEBUG "value": [ 05:30:24 DEBUG 1712723424, 05:30:24 DEBUG "0" 05:30:24 DEBUG ] 05:30:24 DEBUG } 05:30:24 DEBUG ] 05:30:24 DEBUG } 05:30:24 DEBUG } 05:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723424" 05:30:24 INFO [http_cmd]: http status code OK 05:30:24 DEBUG --- status code --- 05:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:24 DEBUG --- http response --- 05:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:30:24 DEBUG ----- output ----- 05:30:24 DEBUG { 05:30:24 DEBUG "status": "success", 05:30:24 DEBUG "data": { 05:30:24 DEBUG "resultType": "vector", 05:30:24 DEBUG "result": [ 05:30:24 DEBUG { 05:30:24 DEBUG "metric": { 05:30:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:30:24 DEBUG }, 05:30:24 DEBUG "value": [ 05:30:24 DEBUG 05:30:24 DEBUG 05:30:24 DEBUG 05:30:24 DEBUG [...] 05:30:24 DEBUG 05:30:24 DEBUG 05:30:24 DEBUG 05:30:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:30:24 DEBUG }, 05:30:24 DEBUG "value": [ 05:30:24 DEBUG 1712723424, 05:30:24 DEBUG "226841.5" 05:30:24 DEBUG ] 05:30:24 DEBUG } 05:30:24 DEBUG ] 05:30:24 DEBUG } 05:30:24 DEBUG } 05:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723424" 05:30:24 INFO [http_cmd]: http status code OK 05:30:24 DEBUG --- status code --- 05:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:24 DEBUG --- http response --- 05:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:30:24 DEBUG ----- output ----- 05:30:24 DEBUG { 05:30:24 DEBUG "status": "success", 05:30:24 DEBUG "data": { 05:30:24 DEBUG "resultType": "vector", 05:30:24 DEBUG "result": [ 05:30:24 DEBUG { 05:30:24 DEBUG "metric": { 05:30:24 DEBUG "pod": "ds-idrepo-0" 05:30:24 DEBUG }, 05:30:24 DEBUG "value": [ 05:30:24 DEBUG 05:30:24 DEBUG 05:30:24 DEBUG 05:30:24 DEBUG [...] 05:30:24 DEBUG 05:30:24 DEBUG 05:30:24 DEBUG 05:30:24 DEBUG "pod": "ds-idrepo-1" 05:30:24 DEBUG }, 05:30:24 DEBUG "value": [ 05:30:24 DEBUG 1712723424, 05:30:24 DEBUG "0" 05:30:24 DEBUG ] 05:30:24 DEBUG } 05:30:24 DEBUG ] 05:30:24 DEBUG } 05:30:24 DEBUG } 05:30:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723425" 05:30:25 INFO [http_cmd]: http status code OK 05:30:25 DEBUG --- status code --- 05:30:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:25 DEBUG --- http response --- 05:30:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:30:25 DEBUG ----- output ----- 05:30:25 DEBUG { 05:30:25 DEBUG "status": "success", 05:30:25 DEBUG "data": { 05:30:25 DEBUG "resultType": "vector", 05:30:25 DEBUG "result": [ 05:30:25 DEBUG { 05:30:25 DEBUG "metric": { 05:30:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:30:25 DEBUG }, 05:30:25 DEBUG "value": [ 05:30:25 DEBUG 05:30:25 DEBUG 05:30:25 DEBUG 05:30:25 DEBUG [...] 05:30:25 DEBUG 05:30:25 DEBUG 05:30:25 DEBUG 05:30:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:30:25 DEBUG }, 05:30:25 DEBUG "value": [ 05:30:25 DEBUG 1712723425, 05:30:25 DEBUG "1265.3333333333333" 05:30:25 DEBUG ] 05:30:25 DEBUG } 05:30:25 DEBUG ] 05:30:25 DEBUG } 05:30:25 DEBUG } 05:30:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723425" 05:30:25 INFO [http_cmd]: http status code OK 05:30:25 DEBUG --- status code --- 05:30:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:25 DEBUG --- http response --- 05:30:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:30:25 DEBUG ----- output ----- 05:30:25 DEBUG { 05:30:25 DEBUG "status": "success", 05:30:25 DEBUG "data": { 05:30:25 DEBUG "resultType": "vector", 05:30:25 DEBUG "result": [ 05:30:25 DEBUG { 05:30:25 DEBUG "metric": { 05:30:25 DEBUG "instance": "10.142.0.25:9100" 05:30:25 DEBUG }, 05:30:25 DEBUG "value": [ 05:30:25 DEBUG 05:30:25 DEBUG 05:30:25 DEBUG 05:30:25 DEBUG [...] 05:30:25 DEBUG 05:30:25 DEBUG 05:30:25 DEBUG 05:30:25 DEBUG "instance": "10.142.0.77:9100" 05:30:25 DEBUG }, 05:30:25 DEBUG "value": [ 05:30:25 DEBUG 1712723425, 05:30:25 DEBUG "0.1523666666666634" 05:30:25 DEBUG ] 05:30:25 DEBUG } 05:30:25 DEBUG ] 05:30:25 DEBUG } 05:30:25 DEBUG } 05:30:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723426" 05:30:26 INFO [http_cmd]: http status code OK 05:30:26 DEBUG --- status code --- 05:30:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:26 DEBUG --- http response --- 05:30:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:30:26 DEBUG ----- output ----- 05:30:26 DEBUG { 05:30:26 DEBUG "status": "success", 05:30:26 DEBUG "data": { 05:30:26 DEBUG "resultType": "vector", 05:30:26 DEBUG "result": [ 05:30:26 DEBUG { 05:30:26 DEBUG "metric": { 05:30:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:30:26 DEBUG }, 05:30:26 DEBUG "value": [ 05:30:26 DEBUG 05:30:26 DEBUG 05:30:26 DEBUG 05:30:26 DEBUG [...] 05:30:26 DEBUG 05:30:26 DEBUG 05:30:26 DEBUG 05:30:26 DEBUG "pod": "ds-cts-1" 05:30:26 DEBUG }, 05:30:26 DEBUG "value": [ 05:30:26 DEBUG 1712723426, 05:30:26 DEBUG "0" 05:30:26 DEBUG ] 05:30:26 DEBUG } 05:30:26 DEBUG ] 05:30:26 DEBUG } 05:30:26 DEBUG } 05:30:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712723426" 05:30:26 INFO [http_cmd]: http status code OK 05:30:26 DEBUG --- status code --- 05:30:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:26 DEBUG --- http response --- 05:30:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:30:26 DEBUG ----- output ----- 05:30:26 DEBUG { 05:30:26 DEBUG "status": "success", 05:30:26 DEBUG "data": { 05:30:26 DEBUG "resultType": "vector", 05:30:26 DEBUG "result": [ 05:30:26 DEBUG { 05:30:26 DEBUG "metric": { 05:30:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:30:26 DEBUG }, 05:30:26 DEBUG "value": [ 05:30:26 DEBUG 05:30:26 DEBUG 05:30:26 DEBUG 05:30:26 DEBUG [...] 05:30:26 DEBUG 05:30:26 DEBUG 05:30:26 DEBUG 05:30:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:30:26 DEBUG }, 05:30:26 DEBUG "value": [ 05:30:26 DEBUG 1712723426, 05:30:26 DEBUG "3.618380829239677" 05:30:26 DEBUG ] 05:30:26 DEBUG } 05:30:26 DEBUG ] 05:30:26 DEBUG } 05:30:26 DEBUG } 05:30:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712723426" 05:30:26 INFO [http_cmd]: http status code OK 05:30:26 DEBUG --- status code --- 05:30:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:26 DEBUG --- http response --- 05:30:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:30:26 DEBUG ----- output ----- 05:30:26 DEBUG { 05:30:26 DEBUG "status": "success", 05:30:26 DEBUG "data": { 05:30:26 DEBUG "resultType": "vector", 05:30:26 DEBUG "result": [ 05:30:26 DEBUG { 05:30:26 DEBUG "metric": { 05:30:26 DEBUG "pod": "ds-idrepo-0" 05:30:26 DEBUG }, 05:30:26 DEBUG "value": [ 05:30:26 DEBUG 05:30:26 DEBUG 05:30:26 DEBUG 05:30:26 DEBUG [...] 05:30:26 DEBUG 05:30:26 DEBUG 05:30:26 DEBUG 05:30:26 DEBUG "pod": "ds-cts-2" 05:30:26 DEBUG }, 05:30:26 DEBUG "value": [ 05:30:26 DEBUG 1712723426, 05:30:26 DEBUG "0" 05:30:26 DEBUG ] 05:30:26 DEBUG } 05:30:26 DEBUG ] 05:30:26 DEBUG } 05:30:26 DEBUG } 05:30:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723426" 05:30:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723426" 05:30:26 INFO [http_cmd]: http status code OK 05:30:26 DEBUG --- status code --- 05:30:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:26 DEBUG --- http response --- 05:30:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:30:26 DEBUG ----- output ----- 05:30:26 DEBUG { 05:30:26 DEBUG "status": "success", 05:30:26 DEBUG "data": { 05:30:26 DEBUG "resultType": "vector", 05:30:26 DEBUG "result": [ 05:30:26 DEBUG { 05:30:26 DEBUG "metric": { 05:30:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:30:26 DEBUG }, 05:30:26 DEBUG "value": [ 05:30:26 DEBUG 05:30:26 DEBUG 05:30:26 DEBUG 05:30:26 DEBUG [...] 05:30:26 DEBUG 05:30:26 DEBUG 05:30:26 DEBUG 05:30:26 DEBUG "pod": "ds-cts-1" 05:30:26 DEBUG }, 05:30:26 DEBUG "value": [ 05:30:26 DEBUG 1712723426, 05:30:26 DEBUG "0" 05:30:26 DEBUG ] 05:30:26 DEBUG } 05:30:26 DEBUG ] 05:30:26 DEBUG } 05:30:26 DEBUG } 05:30:26 INFO [http_cmd]: http status code OK 05:30:26 DEBUG --- status code --- 05:30:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:26 DEBUG --- http response --- 05:30:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:30:26 DEBUG ----- output ----- 05:30:26 DEBUG { 05:30:26 DEBUG "status": "success", 05:30:26 DEBUG "data": { 05:30:26 DEBUG "resultType": "vector", 05:30:26 DEBUG "result": [ 05:30:26 DEBUG { 05:30:26 DEBUG "metric": { 05:30:26 DEBUG "pod": "ds-idrepo-2" 05:30:26 DEBUG }, 05:30:26 DEBUG "value": [ 05:30:26 DEBUG 05:30:26 DEBUG 05:30:26 DEBUG 05:30:26 DEBUG [...] 05:30:26 DEBUG 05:30:26 DEBUG 05:30:26 DEBUG 05:30:26 DEBUG "pod": "ds-cts-2" 05:30:26 DEBUG }, 05:30:26 DEBUG "value": [ 05:30:26 DEBUG 1712723426, 05:30:26 DEBUG "0" 05:30:26 DEBUG ] 05:30:26 DEBUG } 05:30:26 DEBUG ] 05:30:26 DEBUG } 05:30:26 DEBUG } 05:30:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723426" 05:30:26 INFO [http_cmd]: http status code OK 05:30:26 DEBUG --- status code --- 05:30:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:26 DEBUG --- http response --- 05:30:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:30:26 DEBUG ----- output ----- 05:30:26 DEBUG { 05:30:26 DEBUG "status": "success", 05:30:26 DEBUG "data": { 05:30:26 DEBUG "resultType": "vector", 05:30:26 DEBUG "result": [ 05:30:26 DEBUG { 05:30:26 DEBUG "metric": { 05:30:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:30:26 DEBUG }, 05:30:26 DEBUG "value": [ 05:30:26 DEBUG 05:30:26 DEBUG 05:30:26 DEBUG 05:30:26 DEBUG [...] 05:30:26 DEBUG 05:30:26 DEBUG 05:30:26 DEBUG 05:30:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:30:26 DEBUG }, 05:30:26 DEBUG "value": [ 05:30:26 DEBUG 1712723426, 05:30:26 DEBUG "1265.1333333333332" 05:30:26 DEBUG ] 05:30:26 DEBUG } 05:30:26 DEBUG ] 05:30:26 DEBUG } 05:30:26 DEBUG } 05:30:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712723426" 05:30:26 INFO [http_cmd]: http status code OK 05:30:26 DEBUG --- status code --- 05:30:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:26 DEBUG --- http response --- 05:30:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:30:26 DEBUG ----- output ----- 05:30:26 DEBUG { 05:30:26 DEBUG "status": "success", 05:30:26 DEBUG "data": { 05:30:26 DEBUG "resultType": "vector", 05:30:26 DEBUG "result": [ 05:30:26 DEBUG { 05:30:26 DEBUG "metric": { 05:30:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:30:26 DEBUG }, 05:30:26 DEBUG "value": [ 05:30:26 DEBUG 05:30:26 DEBUG 05:30:26 DEBUG 05:30:26 DEBUG [...] 05:30:26 DEBUG 05:30:26 DEBUG 05:30:26 DEBUG 05:30:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:30:26 DEBUG }, 05:30:26 DEBUG "value": [ 05:30:26 DEBUG 1712723426, 05:30:26 DEBUG "0.042" 05:30:26 DEBUG ] 05:30:26 DEBUG } 05:30:26 DEBUG ] 05:30:26 DEBUG } 05:30:26 DEBUG } 05:30:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723427" 05:30:27 INFO [http_cmd]: http status code OK 05:30:27 DEBUG --- status code --- 05:30:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:27 DEBUG --- http response --- 05:30:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:30:27 DEBUG ----- output ----- 05:30:27 DEBUG { 05:30:27 DEBUG "status": "success", 05:30:27 DEBUG "data": { 05:30:27 DEBUG "resultType": "vector", 05:30:27 DEBUG "result": [ 05:30:27 DEBUG { 05:30:27 DEBUG "metric": { 05:30:27 DEBUG "pod": "ds-idrepo-0" 05:30:27 DEBUG }, 05:30:27 DEBUG "value": [ 05:30:27 DEBUG 05:30:27 DEBUG 05:30:27 DEBUG 05:30:27 DEBUG [...] 05:30:27 DEBUG 05:30:27 DEBUG 05:30:27 DEBUG 05:30:27 DEBUG "pod": "ds-idrepo-1" 05:30:27 DEBUG }, 05:30:27 DEBUG "value": [ 05:30:27 DEBUG 1712723427, 05:30:27 DEBUG "0" 05:30:27 DEBUG ] 05:30:27 DEBUG } 05:30:27 DEBUG ] 05:30:27 DEBUG } 05:30:27 DEBUG } 05:30:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723427" 05:30:27 INFO [http_cmd]: http status code OK 05:30:27 DEBUG --- status code --- 05:30:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:27 DEBUG --- http response --- 05:30:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:30:27 DEBUG ----- output ----- 05:30:27 DEBUG { 05:30:27 DEBUG "status": "success", 05:30:27 DEBUG "data": { 05:30:27 DEBUG "resultType": "vector", 05:30:27 DEBUG "result": [ 05:30:27 DEBUG { 05:30:27 DEBUG "metric": { 05:30:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:30:27 DEBUG }, 05:30:27 DEBUG "value": [ 05:30:27 DEBUG 05:30:27 DEBUG 05:30:27 DEBUG 05:30:27 DEBUG [...] 05:30:27 DEBUG 05:30:27 DEBUG 05:30:27 DEBUG 05:30:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:30:27 DEBUG }, 05:30:27 DEBUG "value": [ 05:30:27 DEBUG 1712723427, 05:30:27 DEBUG "1257" 05:30:27 DEBUG ] 05:30:27 DEBUG } 05:30:27 DEBUG ] 05:30:27 DEBUG } 05:30:27 DEBUG } 05:30:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723427" 05:30:27 INFO [http_cmd]: http status code OK 05:30:27 DEBUG --- status code --- 05:30:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:27 DEBUG --- http response --- 05:30:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:30:27 DEBUG ----- output ----- 05:30:27 DEBUG { 05:30:27 DEBUG "status": "success", 05:30:27 DEBUG "data": { 05:30:27 DEBUG "resultType": "vector", 05:30:27 DEBUG "result": [ 05:30:27 DEBUG { 05:30:27 DEBUG "metric": { 05:30:27 DEBUG "pod": "ds-idrepo-0" 05:30:27 DEBUG }, 05:30:27 DEBUG "value": [ 05:30:27 DEBUG 05:30:27 DEBUG 05:30:27 DEBUG 05:30:27 DEBUG [...] 05:30:27 DEBUG 05:30:27 DEBUG 05:30:27 DEBUG 05:30:27 DEBUG "pod": "ds-idrepo-1" 05:30:27 DEBUG }, 05:30:27 DEBUG "value": [ 05:30:27 DEBUG 1712723427, 05:30:27 DEBUG "0" 05:30:27 DEBUG ] 05:30:27 DEBUG } 05:30:27 DEBUG ] 05:30:27 DEBUG } 05:30:27 DEBUG } 05:30:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712723427" 05:30:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723427" 05:30:27 INFO [http_cmd]: http status code OK 05:30:27 DEBUG --- status code --- 05:30:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:27 DEBUG --- http response --- 05:30:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:30:27 DEBUG ----- output ----- 05:30:27 DEBUG { 05:30:27 DEBUG "status": "success", 05:30:27 DEBUG "data": { 05:30:27 DEBUG "resultType": "vector", 05:30:27 DEBUG "result": [ 05:30:27 DEBUG { 05:30:27 DEBUG "metric": { 05:30:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:30:27 DEBUG }, 05:30:27 DEBUG "value": [ 05:30:27 DEBUG 05:30:27 DEBUG 05:30:27 DEBUG 05:30:27 DEBUG [...] 05:30:27 DEBUG 05:30:27 DEBUG 05:30:27 DEBUG 05:30:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:30:27 DEBUG }, 05:30:27 DEBUG "value": [ 05:30:27 DEBUG 1712723427, 05:30:27 DEBUG "2486435840" 05:30:27 DEBUG ] 05:30:27 DEBUG } 05:30:27 DEBUG ] 05:30:27 DEBUG } 05:30:27 DEBUG } 05:30:27 INFO [http_cmd]: http status code OK 05:30:27 DEBUG --- status code --- 05:30:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:27 DEBUG --- http response --- 05:30:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:30:27 DEBUG ----- output ----- 05:30:27 DEBUG { 05:30:27 DEBUG "status": "success", 05:30:27 DEBUG "data": { 05:30:27 DEBUG "resultType": "vector", 05:30:27 DEBUG "result": [ 05:30:27 DEBUG { 05:30:27 DEBUG "metric": { 05:30:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:30:27 DEBUG }, 05:30:27 DEBUG "value": [ 05:30:27 DEBUG 05:30:27 DEBUG 05:30:27 DEBUG 05:30:27 DEBUG [...] 05:30:27 DEBUG 05:30:27 DEBUG 05:30:27 DEBUG 05:30:27 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 05:30:27 DEBUG }, 05:30:27 DEBUG "value": [ 05:30:27 DEBUG 1712723427, 05:30:27 DEBUG "0" 05:30:27 DEBUG ] 05:30:27 DEBUG } 05:30:27 DEBUG ] 05:30:27 DEBUG } 05:30:27 DEBUG } 05:30:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723427" 05:30:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723427" 05:30:27 INFO [http_cmd]: http status code OK 05:30:27 DEBUG --- status code --- 05:30:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:27 DEBUG --- http response --- 05:30:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:30:27 DEBUG ----- output ----- 05:30:27 DEBUG { 05:30:27 DEBUG "status": "success", 05:30:27 DEBUG "data": { 05:30:27 DEBUG "resultType": "vector", 05:30:27 DEBUG "result": [ 05:30:27 DEBUG { 05:30:27 DEBUG "metric": { 05:30:27 DEBUG "instance": "10.142.0.25:9100" 05:30:27 DEBUG }, 05:30:27 DEBUG "value": [ 05:30:27 DEBUG 05:30:27 DEBUG 05:30:27 DEBUG 05:30:27 DEBUG [...] 05:30:27 DEBUG 05:30:27 DEBUG 05:30:27 DEBUG 05:30:27 DEBUG "instance": "10.142.0.77:9100" 05:30:27 DEBUG }, 05:30:27 DEBUG "value": [ 05:30:27 DEBUG 1712723427, 05:30:27 DEBUG "37563946.66666667" 05:30:27 DEBUG ] 05:30:27 DEBUG } 05:30:27 DEBUG ] 05:30:27 INFO [http_cmd]: http status code OK 05:30:27 DEBUG } 05:30:27 DEBUG --- status code --- 05:30:27 DEBUG } 05:30:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:30:27 DEBUG --- http response --- 05:30:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:30:27 DEBUG ----- output ----- 05:30:27 DEBUG { 05:30:27 DEBUG "status": "success", 05:30:27 DEBUG "data": { 05:30:27 DEBUG "resultType": "vector", 05:30:27 DEBUG "result": [ 05:30:27 DEBUG { 05:30:27 DEBUG "metric": { 05:30:27 DEBUG "instance": "10.142.0.25:9100" 05:30:27 DEBUG }, 05:30:27 DEBUG "value": [ 05:30:27 DEBUG 05:30:27 DEBUG 05:30:27 DEBUG 05:30:27 DEBUG [...] 05:30:27 DEBUG 05:30:27 DEBUG 05:30:27 DEBUG 05:30:27 DEBUG "instance": "10.142.0.77:9100" 05:30:27 DEBUG }, 05:30:27 DEBUG "value": [ 05:30:27 DEBUG 1712723427, 05:30:27 DEBUG "0.05933333333333397" 05:30:27 DEBUG ] 05:30:27 DEBUG } 05:30:27 DEBUG ] 05:30:27 DEBUG } 05:30:27 DEBUG } 05:30:42 INFO 05:30:42 INFO [loop_until]: kubectl --namespace=xlou top pods 05:30:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:30:42 INFO [loop_until]: OK (rc = 0) 05:30:42 DEBUG --- stdout --- 05:30:42 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 8944m 8806Mi am-654c4fcbcf-9scrj 8808m 8421Mi am-654c4fcbcf-v5qb5 7723m 8402Mi ds-cts-0 7085m 13319Mi ds-cts-1 7566m 12373Mi ds-cts-2 6799m 13171Mi ds-idrepo-0 9599m 5622Mi ds-idrepo-1 318m 5669Mi ds-idrepo-2 325m 5698Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2212Mi lodemon-b776b7d88-thrqf 2m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3496m 1664Mi 05:30:42 DEBUG --- stderr --- 05:30:42 DEBUG 05:30:48 INFO 05:30:48 INFO [loop_until]: kubectl --namespace=xlou top node 05:30:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:30:49 INFO [loop_until]: OK (rc = 0) 05:30:49 DEBUG --- stdout --- 05:30:49 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 92m 0% 2306Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8138m 51% 9283Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 82m 0% 3257Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10746m 67% 6509Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9521m 59% 9311Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9605m 60% 9646Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 77m 0% 851Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9711m 61% 6282Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 6987m 43% 13941Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 7186m 45% 13786Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 389m 2% 6365Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7479m 47% 13087Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 395m 2% 6331Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3700m 23% 5346Mi 9% 05:30:49 DEBUG --- stderr --- 05:30:49 DEBUG 05:31:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712723483" 05:31:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723483" 05:31:23 ERROR [http_cmd]: ERROR 05:31:23 INFO [http_cmd]: http status code OK 05:31:23 ERROR --- status code --- 05:31:23 DEBUG --- status code --- 05:31:23 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:31:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:23 ERROR --- http response --- 05:31:23 DEBUG --- http response --- 05:31:23 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:31:23 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:31:23 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:31:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723484" 05:31:24 INFO [http_cmd]: http status code OK 05:31:24 DEBUG --- status code --- 05:31:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:24 DEBUG --- http response --- 05:31:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:31:24 DEBUG ----- output ----- 05:31:24 DEBUG { 05:31:24 DEBUG "status": "success", 05:31:24 DEBUG "data": { 05:31:24 DEBUG "resultType": "vector", 05:31:24 DEBUG "result": [ 05:31:24 DEBUG { 05:31:24 DEBUG "metric": { 05:31:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:31:24 DEBUG }, 05:31:24 DEBUG "value": [ 05:31:24 DEBUG 05:31:24 DEBUG 05:31:24 DEBUG 05:31:24 DEBUG [...] 05:31:24 DEBUG 05:31:24 DEBUG 05:31:24 DEBUG 05:31:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:31:24 DEBUG }, 05:31:24 DEBUG "value": [ 05:31:24 DEBUG 1712723484, 05:31:24 DEBUG "0" 05:31:24 DEBUG ] 05:31:24 DEBUG } 05:31:24 DEBUG ] 05:31:24 DEBUG } 05:31:24 DEBUG } 05:31:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723484" 05:31:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723484" 05:31:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723484" 05:31:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723484" 05:31:24 INFO [http_cmd]: http status code OK 05:31:24 INFO [http_cmd]: http status code OK 05:31:24 INFO [http_cmd]: http status code OK 05:31:24 DEBUG --- status code --- 05:31:24 INFO [http_cmd]: http status code OK 05:31:24 DEBUG --- status code --- 05:31:24 DEBUG --- status code --- 05:31:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:24 DEBUG --- status code --- 05:31:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:24 DEBUG --- http response --- 05:31:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:24 DEBUG --- http response --- 05:31:24 DEBUG --- http response --- 05:31:24 DEBUG --- http response --- 05:31:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:31:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:31:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:31:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:31:24 DEBUG ----- output ----- 05:31:24 DEBUG ----- output ----- 05:31:24 DEBUG ----- output ----- 05:31:24 DEBUG ----- output ----- 05:31:24 DEBUG { 05:31:24 DEBUG { 05:31:24 DEBUG { 05:31:24 DEBUG { 05:31:24 DEBUG "status": "success", 05:31:24 DEBUG "status": "success", 05:31:24 DEBUG "status": "success", 05:31:24 DEBUG "status": "success", 05:31:24 DEBUG "data": { 05:31:24 DEBUG "data": { 05:31:24 DEBUG "data": { 05:31:24 DEBUG "data": { 05:31:24 DEBUG "resultType": "vector", 05:31:24 DEBUG "result": [ 05:31:24 DEBUG "resultType": "vector", 05:31:24 DEBUG "resultType": "vector", 05:31:24 DEBUG "result": [ 05:31:24 DEBUG { 05:31:24 DEBUG "metric": { 05:31:24 DEBUG "instance": "10.142.0.25:9100" 05:31:24 DEBUG }, 05:31:24 DEBUG "value": [ 05:31:24 DEBUG 05:31:24 DEBUG 05:31:24 DEBUG 05:31:24 DEBUG [...] 05:31:24 DEBUG 05:31:24 DEBUG 05:31:24 DEBUG 05:31:24 DEBUG "result": [ 05:31:24 DEBUG "instance": "10.142.0.77:9100" 05:31:24 DEBUG "resultType": "vector", 05:31:24 DEBUG { 05:31:24 DEBUG { 05:31:24 DEBUG }, 05:31:24 DEBUG "result": [ 05:31:24 DEBUG "metric": { 05:31:24 DEBUG "metric": { 05:31:24 DEBUG "pod": "ds-idrepo-0" 05:31:24 DEBUG { 05:31:24 DEBUG "pod": "ds-idrepo-0" 05:31:24 DEBUG "value": [ 05:31:24 DEBUG }, 05:31:24 DEBUG "metric": { 05:31:24 DEBUG }, 05:31:24 DEBUG 1712723484, 05:31:24 DEBUG "value": [ 05:31:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:31:24 DEBUG "value": [ 05:31:24 DEBUG "290545974272" 05:31:24 DEBUG 05:31:24 DEBUG }, 05:31:24 DEBUG 05:31:24 DEBUG ] 05:31:24 DEBUG 05:31:24 DEBUG "value": [ 05:31:24 DEBUG 05:31:24 DEBUG } 05:31:24 DEBUG 05:31:24 DEBUG [...] 05:31:24 DEBUG 05:31:24 DEBUG ] 05:31:24 DEBUG 05:31:24 DEBUG 05:31:24 DEBUG [...] 05:31:24 DEBUG } 05:31:24 DEBUG 05:31:24 DEBUG 05:31:24 DEBUG 05:31:24 DEBUG } 05:31:24 DEBUG 05:31:24 DEBUG 05:31:24 DEBUG 05:31:24 DEBUG [...] 05:31:24 DEBUG "pod": "ds-cts-2" 05:31:24 DEBUG 05:31:24 DEBUG 05:31:24 DEBUG }, 05:31:24 DEBUG "pod": "ds-cts-2" 05:31:24 DEBUG 05:31:24 DEBUG "value": [ 05:31:24 DEBUG }, 05:31:24 DEBUG 05:31:24 DEBUG 1712723484, 05:31:24 DEBUG "value": [ 05:31:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:31:24 DEBUG "29.7" 05:31:24 DEBUG 1712723484, 05:31:24 DEBUG }, 05:31:24 DEBUG "value": [ 05:31:24 DEBUG "0" 05:31:24 DEBUG 1712723484, 05:31:24 DEBUG "1262.3" 05:31:24 DEBUG ] 05:31:24 DEBUG ] 05:31:24 DEBUG ] 05:31:24 DEBUG } 05:31:24 DEBUG } 05:31:24 DEBUG } 05:31:24 DEBUG ] 05:31:24 DEBUG ] 05:31:24 DEBUG ] 05:31:24 DEBUG } 05:31:24 DEBUG } 05:31:24 DEBUG } 05:31:24 DEBUG } 05:31:24 DEBUG } 05:31:24 DEBUG } 05:31:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723484" 05:31:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723484" 05:31:24 INFO [http_cmd]: http status code OK 05:31:24 INFO [http_cmd]: http status code OK 05:31:24 DEBUG --- status code --- 05:31:24 DEBUG --- status code --- 05:31:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:24 DEBUG --- http response --- 05:31:24 DEBUG --- http response --- 05:31:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 05:31:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:31:24 DEBUG ----- output ----- 05:31:24 DEBUG ----- output ----- 05:31:24 DEBUG { 05:31:24 DEBUG { 05:31:24 DEBUG "status": "success", 05:31:24 DEBUG "data": { 05:31:24 DEBUG "resultType": "vector", 05:31:24 DEBUG "status": "success", 05:31:24 DEBUG "result": [ 05:31:24 DEBUG "data": { 05:31:24 DEBUG { 05:31:24 DEBUG "resultType": "vector", 05:31:24 DEBUG "metric": { 05:31:24 DEBUG "result": [ 05:31:24 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:31:24 DEBUG { 05:31:24 DEBUG }, 05:31:24 DEBUG "metric": { 05:31:24 DEBUG "value": [ 05:31:24 DEBUG "instance": "10.142.0.25:9100" 05:31:24 DEBUG 05:31:24 DEBUG }, 05:31:24 DEBUG 05:31:24 DEBUG "value": [ 05:31:24 DEBUG 05:31:24 DEBUG 05:31:24 DEBUG [...] 05:31:24 DEBUG 05:31:24 DEBUG 05:31:24 DEBUG 05:31:24 DEBUG 05:31:24 DEBUG [...] 05:31:24 DEBUG 05:31:24 DEBUG 05:31:24 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:31:24 DEBUG 05:31:24 DEBUG }, 05:31:24 DEBUG 05:31:24 DEBUG "value": [ 05:31:24 DEBUG "instance": "10.142.0.77:9100" 05:31:24 DEBUG 1712723484, 05:31:24 DEBUG }, 05:31:24 DEBUG "0" 05:31:24 DEBUG "value": [ 05:31:24 DEBUG ] 05:31:24 DEBUG 1712723484, 05:31:24 DEBUG } 05:31:24 DEBUG "37521184.9" 05:31:24 DEBUG ] 05:31:24 DEBUG ] 05:31:24 DEBUG } 05:31:24 DEBUG } 05:31:24 DEBUG } 05:31:24 DEBUG ] 05:31:24 DEBUG } 05:31:24 DEBUG } 05:31:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723484" 05:31:24 INFO [http_cmd]: http status code OK 05:31:24 DEBUG --- status code --- 05:31:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:24 DEBUG --- http response --- 05:31:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:31:24 DEBUG ----- output ----- 05:31:24 DEBUG { 05:31:24 DEBUG "status": "success", 05:31:24 DEBUG "data": { 05:31:24 DEBUG "resultType": "vector", 05:31:24 DEBUG "result": [ 05:31:24 DEBUG { 05:31:24 DEBUG "metric": { 05:31:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:31:24 DEBUG }, 05:31:24 DEBUG "value": [ 05:31:24 DEBUG 05:31:24 DEBUG 05:31:24 DEBUG 05:31:24 DEBUG [...] 05:31:24 DEBUG 05:31:24 DEBUG 05:31:24 DEBUG 05:31:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:31:24 DEBUG }, 05:31:24 DEBUG "value": [ 05:31:24 DEBUG 1712723484, 05:31:24 DEBUG "227395.5" 05:31:24 DEBUG ] 05:31:24 DEBUG } 05:31:24 DEBUG ] 05:31:24 DEBUG } 05:31:24 DEBUG } 05:31:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723484" 05:31:24 INFO [http_cmd]: http status code OK 05:31:24 DEBUG --- status code --- 05:31:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:24 DEBUG --- http response --- 05:31:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:31:24 DEBUG ----- output ----- 05:31:24 DEBUG { 05:31:24 DEBUG "status": "success", 05:31:24 DEBUG "data": { 05:31:24 DEBUG "resultType": "vector", 05:31:24 DEBUG "result": [ 05:31:24 DEBUG { 05:31:24 DEBUG "metric": { 05:31:24 DEBUG "pod": "ds-idrepo-0" 05:31:24 DEBUG }, 05:31:24 DEBUG "value": [ 05:31:24 DEBUG 05:31:24 DEBUG 05:31:24 DEBUG 05:31:24 DEBUG [...] 05:31:24 DEBUG 05:31:24 DEBUG 05:31:24 DEBUG 05:31:24 DEBUG "pod": "ds-idrepo-1" 05:31:24 DEBUG }, 05:31:24 DEBUG "value": [ 05:31:24 DEBUG 1712723484, 05:31:24 DEBUG "0" 05:31:24 DEBUG ] 05:31:24 DEBUG } 05:31:24 DEBUG ] 05:31:24 DEBUG } 05:31:24 DEBUG } 05:31:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723485" 05:31:25 INFO [http_cmd]: http status code OK 05:31:25 DEBUG --- status code --- 05:31:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:25 DEBUG --- http response --- 05:31:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:31:25 DEBUG ----- output ----- 05:31:25 DEBUG { 05:31:25 DEBUG "status": "success", 05:31:25 DEBUG "data": { 05:31:25 DEBUG "resultType": "vector", 05:31:25 DEBUG "result": [ 05:31:25 DEBUG { 05:31:25 DEBUG "metric": { 05:31:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:31:25 DEBUG }, 05:31:25 DEBUG "value": [ 05:31:25 DEBUG 05:31:25 DEBUG 05:31:25 DEBUG 05:31:25 DEBUG [...] 05:31:25 DEBUG 05:31:25 DEBUG 05:31:25 DEBUG 05:31:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:31:25 DEBUG }, 05:31:25 DEBUG "value": [ 05:31:25 DEBUG 1712723485, 05:31:25 DEBUG "1262.7" 05:31:25 DEBUG ] 05:31:25 DEBUG } 05:31:25 DEBUG ] 05:31:25 DEBUG } 05:31:25 DEBUG } 05:31:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723485" 05:31:25 INFO [http_cmd]: http status code OK 05:31:25 DEBUG --- status code --- 05:31:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:25 DEBUG --- http response --- 05:31:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:31:25 DEBUG ----- output ----- 05:31:25 DEBUG { 05:31:25 DEBUG "status": "success", 05:31:25 DEBUG "data": { 05:31:25 DEBUG "resultType": "vector", 05:31:25 DEBUG "result": [ 05:31:25 DEBUG { 05:31:25 DEBUG "metric": { 05:31:25 DEBUG "instance": "10.142.0.25:9100" 05:31:25 DEBUG }, 05:31:25 DEBUG "value": [ 05:31:25 DEBUG 05:31:25 DEBUG 05:31:25 DEBUG 05:31:25 DEBUG [...] 05:31:25 DEBUG 05:31:25 DEBUG 05:31:25 DEBUG 05:31:25 DEBUG "instance": "10.142.0.77:9100" 05:31:25 DEBUG }, 05:31:25 DEBUG "value": [ 05:31:25 DEBUG 1712723485, 05:31:25 DEBUG "0.03196666666667174" 05:31:25 DEBUG ] 05:31:25 DEBUG } 05:31:25 DEBUG ] 05:31:25 DEBUG } 05:31:25 DEBUG } 05:31:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723486" 05:31:26 INFO [http_cmd]: http status code OK 05:31:26 DEBUG --- status code --- 05:31:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:26 DEBUG --- http response --- 05:31:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:31:26 DEBUG ----- output ----- 05:31:26 DEBUG { 05:31:26 DEBUG "status": "success", 05:31:26 DEBUG "data": { 05:31:26 DEBUG "resultType": "vector", 05:31:26 DEBUG "result": [ 05:31:26 DEBUG { 05:31:26 DEBUG "metric": { 05:31:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:31:26 DEBUG }, 05:31:26 DEBUG "value": [ 05:31:26 DEBUG 05:31:26 DEBUG 05:31:26 DEBUG 05:31:26 DEBUG [...] 05:31:26 DEBUG 05:31:26 DEBUG 05:31:26 DEBUG 05:31:26 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:31:26 DEBUG }, 05:31:26 DEBUG "value": [ 05:31:26 DEBUG 1712723486, 05:31:26 DEBUG "0" 05:31:26 DEBUG ] 05:31:26 DEBUG } 05:31:26 DEBUG ] 05:31:26 DEBUG } 05:31:26 DEBUG } 05:31:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712723486" 05:31:26 INFO [http_cmd]: http status code OK 05:31:26 DEBUG --- status code --- 05:31:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:26 DEBUG --- http response --- 05:31:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:31:26 DEBUG ----- output ----- 05:31:26 DEBUG { 05:31:26 DEBUG "status": "success", 05:31:26 DEBUG "data": { 05:31:26 DEBUG "resultType": "vector", 05:31:26 DEBUG "result": [ 05:31:26 DEBUG { 05:31:26 DEBUG "metric": { 05:31:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:31:26 DEBUG }, 05:31:26 DEBUG "value": [ 05:31:26 DEBUG 05:31:26 DEBUG 05:31:26 DEBUG 05:31:26 DEBUG [...] 05:31:26 DEBUG 05:31:26 DEBUG 05:31:26 DEBUG 05:31:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:31:26 DEBUG }, 05:31:26 DEBUG "value": [ 05:31:26 DEBUG 1712723486, 05:31:26 DEBUG "3.6172043137025063" 05:31:26 DEBUG ] 05:31:26 DEBUG } 05:31:26 DEBUG ] 05:31:26 DEBUG } 05:31:26 DEBUG } 05:31:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712723486" 05:31:26 INFO [http_cmd]: http status code OK 05:31:26 DEBUG --- status code --- 05:31:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:26 DEBUG --- http response --- 05:31:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:31:26 DEBUG ----- output ----- 05:31:26 DEBUG { 05:31:26 DEBUG "status": "success", 05:31:26 DEBUG "data": { 05:31:26 DEBUG "resultType": "vector", 05:31:26 DEBUG "result": [ 05:31:26 DEBUG { 05:31:26 DEBUG "metric": { 05:31:26 DEBUG "pod": "ds-idrepo-0" 05:31:26 DEBUG }, 05:31:26 DEBUG "value": [ 05:31:26 DEBUG 05:31:26 DEBUG 05:31:26 DEBUG 05:31:26 DEBUG [...] 05:31:26 DEBUG 05:31:26 DEBUG 05:31:26 DEBUG 05:31:26 DEBUG "pod": "ds-cts-2" 05:31:26 DEBUG }, 05:31:26 DEBUG "value": [ 05:31:26 DEBUG 1712723486, 05:31:26 DEBUG "0" 05:31:26 DEBUG ] 05:31:26 DEBUG } 05:31:26 DEBUG ] 05:31:26 DEBUG } 05:31:26 DEBUG } 05:31:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723486" 05:31:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723486" 05:31:26 INFO [http_cmd]: http status code OK 05:31:26 DEBUG --- status code --- 05:31:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:26 DEBUG --- http response --- 05:31:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:31:26 DEBUG ----- output ----- 05:31:26 DEBUG { 05:31:26 DEBUG "status": "success", 05:31:26 DEBUG "data": { 05:31:26 DEBUG "resultType": "vector", 05:31:26 DEBUG "result": [ 05:31:26 DEBUG { 05:31:26 DEBUG "metric": { 05:31:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:31:26 DEBUG }, 05:31:26 DEBUG "value": [ 05:31:26 DEBUG 05:31:26 DEBUG 05:31:26 DEBUG 05:31:26 DEBUG [...] 05:31:26 DEBUG 05:31:26 DEBUG 05:31:26 DEBUG 05:31:26 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:31:26 DEBUG }, 05:31:26 DEBUG "value": [ 05:31:26 DEBUG 1712723486, 05:31:26 DEBUG "0" 05:31:26 DEBUG ] 05:31:26 DEBUG } 05:31:26 DEBUG ] 05:31:26 DEBUG } 05:31:26 DEBUG } 05:31:26 INFO [http_cmd]: http status code OK 05:31:26 DEBUG --- status code --- 05:31:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:26 DEBUG --- http response --- 05:31:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:31:26 DEBUG ----- output ----- 05:31:26 DEBUG { 05:31:26 DEBUG "status": "success", 05:31:26 DEBUG "data": { 05:31:26 DEBUG "resultType": "vector", 05:31:26 DEBUG "result": [ 05:31:26 DEBUG { 05:31:26 DEBUG "metric": { 05:31:26 DEBUG "pod": "ds-idrepo-2" 05:31:26 DEBUG }, 05:31:26 DEBUG "value": [ 05:31:26 DEBUG 05:31:26 DEBUG 05:31:26 DEBUG 05:31:26 DEBUG [...] 05:31:26 DEBUG 05:31:26 DEBUG 05:31:26 DEBUG 05:31:26 DEBUG "pod": "ds-cts-2" 05:31:26 DEBUG }, 05:31:26 DEBUG "value": [ 05:31:26 DEBUG 1712723486, 05:31:26 DEBUG "0.00005399666666666667" 05:31:26 DEBUG ] 05:31:26 DEBUG } 05:31:26 DEBUG ] 05:31:26 DEBUG } 05:31:26 DEBUG } 05:31:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723486" 05:31:26 INFO [http_cmd]: http status code OK 05:31:26 DEBUG --- status code --- 05:31:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:26 DEBUG --- http response --- 05:31:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:31:26 DEBUG ----- output ----- 05:31:26 DEBUG { 05:31:26 DEBUG "status": "success", 05:31:26 DEBUG "data": { 05:31:26 DEBUG "resultType": "vector", 05:31:26 DEBUG "result": [ 05:31:26 DEBUG { 05:31:26 DEBUG "metric": { 05:31:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:31:26 DEBUG }, 05:31:26 DEBUG "value": [ 05:31:26 DEBUG 05:31:26 DEBUG 05:31:26 DEBUG 05:31:26 DEBUG [...] 05:31:26 DEBUG 05:31:26 DEBUG 05:31:26 DEBUG 05:31:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:31:26 DEBUG }, 05:31:26 DEBUG "value": [ 05:31:26 DEBUG 1712723486, 05:31:26 DEBUG "1263" 05:31:26 DEBUG ] 05:31:26 DEBUG } 05:31:26 DEBUG ] 05:31:26 DEBUG } 05:31:26 DEBUG } 05:31:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712723486" 05:31:26 INFO [http_cmd]: http status code OK 05:31:26 DEBUG --- status code --- 05:31:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:26 DEBUG --- http response --- 05:31:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:31:26 DEBUG ----- output ----- 05:31:26 DEBUG { 05:31:26 DEBUG "status": "success", 05:31:26 DEBUG "data": { 05:31:26 DEBUG "resultType": "vector", 05:31:26 DEBUG "result": [ 05:31:26 DEBUG { 05:31:26 DEBUG "metric": { 05:31:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:31:26 DEBUG }, 05:31:26 DEBUG "value": [ 05:31:26 DEBUG 05:31:26 DEBUG 05:31:26 DEBUG 05:31:26 DEBUG [...] 05:31:26 DEBUG 05:31:26 DEBUG 05:31:26 DEBUG 05:31:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:31:26 DEBUG }, 05:31:26 DEBUG "value": [ 05:31:26 DEBUG 1712723486, 05:31:26 DEBUG "0.042" 05:31:26 DEBUG ] 05:31:26 DEBUG } 05:31:26 DEBUG ] 05:31:26 DEBUG } 05:31:26 DEBUG } 05:31:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723487" 05:31:27 INFO [http_cmd]: http status code OK 05:31:27 DEBUG --- status code --- 05:31:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:27 DEBUG --- http response --- 05:31:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:31:27 DEBUG ----- output ----- 05:31:27 DEBUG { 05:31:27 DEBUG "status": "success", 05:31:27 DEBUG "data": { 05:31:27 DEBUG "resultType": "vector", 05:31:27 DEBUG "result": [ 05:31:27 DEBUG { 05:31:27 DEBUG "metric": { 05:31:27 DEBUG "pod": "ds-idrepo-0" 05:31:27 DEBUG }, 05:31:27 DEBUG "value": [ 05:31:27 DEBUG 05:31:27 DEBUG 05:31:27 DEBUG 05:31:27 DEBUG [...] 05:31:27 DEBUG 05:31:27 DEBUG 05:31:27 DEBUG 05:31:27 DEBUG "pod": "ds-idrepo-1" 05:31:27 DEBUG }, 05:31:27 DEBUG "value": [ 05:31:27 DEBUG 1712723487, 05:31:27 DEBUG "0" 05:31:27 DEBUG ] 05:31:27 DEBUG } 05:31:27 DEBUG ] 05:31:27 DEBUG } 05:31:27 DEBUG } 05:31:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723487" 05:31:27 INFO [http_cmd]: http status code OK 05:31:27 DEBUG --- status code --- 05:31:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:27 DEBUG --- http response --- 05:31:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:31:27 DEBUG ----- output ----- 05:31:27 DEBUG { 05:31:27 DEBUG "status": "success", 05:31:27 DEBUG "data": { 05:31:27 DEBUG "resultType": "vector", 05:31:27 DEBUG "result": [ 05:31:27 DEBUG { 05:31:27 DEBUG "metric": { 05:31:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:31:27 DEBUG }, 05:31:27 DEBUG "value": [ 05:31:27 DEBUG 05:31:27 DEBUG 05:31:27 DEBUG 05:31:27 DEBUG [...] 05:31:27 DEBUG 05:31:27 DEBUG 05:31:27 DEBUG 05:31:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:31:27 DEBUG }, 05:31:27 DEBUG "value": [ 05:31:27 DEBUG 1712723487, 05:31:27 DEBUG "1264.1666666666667" 05:31:27 DEBUG ] 05:31:27 DEBUG } 05:31:27 DEBUG ] 05:31:27 DEBUG } 05:31:27 DEBUG } 05:31:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723487" 05:31:27 INFO [http_cmd]: http status code OK 05:31:27 DEBUG --- status code --- 05:31:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:27 DEBUG --- http response --- 05:31:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:31:27 DEBUG ----- output ----- 05:31:27 DEBUG { 05:31:27 DEBUG "status": "success", 05:31:27 DEBUG "data": { 05:31:27 DEBUG "resultType": "vector", 05:31:27 DEBUG "result": [ 05:31:27 DEBUG { 05:31:27 DEBUG "metric": { 05:31:27 DEBUG "pod": "ds-idrepo-0" 05:31:27 DEBUG }, 05:31:27 DEBUG "value": [ 05:31:27 DEBUG 05:31:27 DEBUG 05:31:27 DEBUG 05:31:27 DEBUG [...] 05:31:27 DEBUG 05:31:27 DEBUG 05:31:27 DEBUG 05:31:27 DEBUG "pod": "ds-idrepo-1" 05:31:27 DEBUG }, 05:31:27 DEBUG "value": [ 05:31:27 DEBUG 1712723487, 05:31:27 DEBUG "0" 05:31:27 DEBUG ] 05:31:27 DEBUG } 05:31:27 DEBUG ] 05:31:27 DEBUG } 05:31:27 DEBUG } 05:31:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712723487" 05:31:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723487" 05:31:27 INFO [http_cmd]: http status code OK 05:31:27 INFO [http_cmd]: http status code OK 05:31:27 DEBUG --- status code --- 05:31:27 DEBUG --- status code --- 05:31:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:27 DEBUG --- http response --- 05:31:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:27 DEBUG --- http response --- 05:31:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:31:27 DEBUG ----- output ----- 05:31:27 DEBUG { 05:31:27 DEBUG "status": "success", 05:31:27 DEBUG "data": { 05:31:27 DEBUG "resultType": "vector", 05:31:27 DEBUG "result": [ 05:31:27 DEBUG { 05:31:27 DEBUG "metric": { 05:31:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:31:27 DEBUG }, 05:31:27 DEBUG "value": [ 05:31:27 DEBUG 05:31:27 DEBUG 05:31:27 DEBUG 05:31:27 DEBUG [...] 05:31:27 DEBUG 05:31:27 DEBUG 05:31:27 DEBUG 05:31:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:31:27 DEBUG }, 05:31:27 DEBUG "value": [ 05:31:27 DEBUG 1712723487, 05:31:27 DEBUG "2489028608" 05:31:27 DEBUG ] 05:31:27 DEBUG } 05:31:27 DEBUG ] 05:31:27 DEBUG } 05:31:27 DEBUG } 05:31:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:31:27 DEBUG ----- output ----- 05:31:27 DEBUG { 05:31:27 DEBUG "status": "success", 05:31:27 DEBUG "data": { 05:31:27 DEBUG "resultType": "vector", 05:31:27 DEBUG "result": [ 05:31:27 DEBUG { 05:31:27 DEBUG "metric": { 05:31:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:31:27 DEBUG }, 05:31:27 DEBUG "value": [ 05:31:27 DEBUG 05:31:27 DEBUG 05:31:27 DEBUG 05:31:27 DEBUG [...] 05:31:27 DEBUG 05:31:27 DEBUG 05:31:27 DEBUG 05:31:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:31:27 DEBUG }, 05:31:27 DEBUG "value": [ 05:31:27 DEBUG 1712723487, 05:31:27 DEBUG "0" 05:31:27 DEBUG ] 05:31:27 DEBUG } 05:31:27 DEBUG ] 05:31:27 DEBUG } 05:31:27 DEBUG } 05:31:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723487" 05:31:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723487" 05:31:27 INFO [http_cmd]: http status code OK 05:31:27 DEBUG --- status code --- 05:31:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:27 DEBUG --- http response --- 05:31:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:31:27 DEBUG ----- output ----- 05:31:27 DEBUG { 05:31:27 DEBUG "status": "success", 05:31:27 DEBUG "data": { 05:31:27 DEBUG "resultType": "vector", 05:31:27 DEBUG "result": [ 05:31:27 DEBUG { 05:31:27 DEBUG "metric": { 05:31:27 DEBUG "instance": "10.142.0.25:9100" 05:31:27 DEBUG }, 05:31:27 DEBUG "value": [ 05:31:27 DEBUG 05:31:27 INFO [http_cmd]: http status code OK 05:31:27 DEBUG 05:31:27 DEBUG --- status code --- 05:31:27 DEBUG 05:31:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:31:27 DEBUG [...] 05:31:27 DEBUG --- http response --- 05:31:27 DEBUG 05:31:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:31:27 DEBUG 05:31:27 DEBUG 05:31:27 DEBUG ----- output ----- 05:31:27 DEBUG "instance": "10.142.0.77:9100" 05:31:27 DEBUG }, 05:31:27 DEBUG "value": [ 05:31:27 DEBUG 1712723487, 05:31:27 DEBUG "37520777.56666666" 05:31:27 DEBUG ] 05:31:27 DEBUG } 05:31:27 DEBUG ] 05:31:27 DEBUG } 05:31:27 DEBUG } 05:31:27 DEBUG { 05:31:27 DEBUG "status": "success", 05:31:27 DEBUG "data": { 05:31:27 DEBUG "resultType": "vector", 05:31:27 DEBUG "result": [ 05:31:27 DEBUG { 05:31:27 DEBUG "metric": { 05:31:27 DEBUG "instance": "10.142.0.25:9100" 05:31:27 DEBUG }, 05:31:27 DEBUG "value": [ 05:31:27 DEBUG 05:31:27 DEBUG 05:31:27 DEBUG 05:31:27 DEBUG [...] 05:31:27 DEBUG 05:31:27 DEBUG 05:31:27 DEBUG 05:31:27 DEBUG "instance": "10.142.0.77:9100" 05:31:27 DEBUG }, 05:31:27 DEBUG "value": [ 05:31:27 DEBUG 1712723487, 05:31:27 DEBUG "0.015333333333333064" 05:31:27 DEBUG ] 05:31:27 DEBUG } 05:31:27 DEBUG ] 05:31:27 DEBUG } 05:31:27 DEBUG } 05:31:42 INFO 05:31:42 INFO [loop_until]: kubectl --namespace=xlou top pods 05:31:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:31:42 INFO [loop_until]: OK (rc = 0) 05:31:42 DEBUG --- stdout --- 05:31:42 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 9006m 8806Mi am-654c4fcbcf-9scrj 8678m 8420Mi am-654c4fcbcf-v5qb5 7361m 8399Mi ds-cts-0 7111m 12892Mi ds-cts-1 7438m 13127Mi ds-cts-2 7164m 13224Mi ds-idrepo-0 9458m 5622Mi ds-idrepo-1 314m 5669Mi ds-idrepo-2 330m 5698Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 6m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3601m 1665Mi 05:31:42 DEBUG --- stderr --- 05:31:42 DEBUG 05:31:49 INFO 05:31:49 INFO [loop_until]: kubectl --namespace=xlou top node 05:31:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:31:49 INFO [loop_until]: OK (rc = 0) 05:31:49 DEBUG --- stdout --- 05:31:49 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 93m 0% 2307Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8199m 51% 9282Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 78m 0% 3259Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10799m 67% 6528Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9628m 60% 9310Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9935m 62% 9648Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 77m 0% 853Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9758m 61% 6282Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 7307m 45% 13550Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 7459m 46% 13868Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 391m 2% 6365Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7661m 48% 13710Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 427m 2% 6333Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3716m 23% 5343Mi 9% 05:31:49 DEBUG --- stderr --- 05:31:49 DEBUG 05:32:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723544" 05:32:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712723544" 05:32:24 ERROR [http_cmd]: ERROR 05:32:24 ERROR --- status code --- 05:32:24 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:32:24 ERROR --- http response --- 05:32:24 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:32:24 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:32:24 INFO [http_cmd]: http status code OK 05:32:24 DEBUG --- status code --- 05:32:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:24 DEBUG --- http response --- 05:32:24 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:32:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723544" 05:32:24 INFO [http_cmd]: http status code OK 05:32:24 DEBUG --- status code --- 05:32:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:24 DEBUG --- http response --- 05:32:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:32:24 DEBUG ----- output ----- 05:32:24 DEBUG { 05:32:24 DEBUG "status": "success", 05:32:24 DEBUG "data": { 05:32:24 DEBUG "resultType": "vector", 05:32:24 DEBUG "result": [ 05:32:24 DEBUG { 05:32:24 DEBUG "metric": { 05:32:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:32:24 DEBUG }, 05:32:24 DEBUG "value": [ 05:32:24 DEBUG 05:32:24 DEBUG 05:32:24 DEBUG 05:32:24 DEBUG [...] 05:32:24 DEBUG 05:32:24 DEBUG 05:32:24 DEBUG 05:32:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:32:24 DEBUG }, 05:32:24 DEBUG "value": [ 05:32:24 DEBUG 1712723544, 05:32:24 DEBUG "0" 05:32:24 DEBUG ] 05:32:24 DEBUG } 05:32:24 DEBUG ] 05:32:24 DEBUG } 05:32:24 DEBUG } 05:32:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723544" 05:32:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723544" 05:32:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723544" 05:32:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723544" 05:32:24 INFO [http_cmd]: http status code OK 05:32:24 INFO [http_cmd]: http status code OK 05:32:24 DEBUG --- status code --- 05:32:24 DEBUG --- status code --- 05:32:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:24 DEBUG --- http response --- 05:32:24 DEBUG --- http response --- 05:32:24 INFO [http_cmd]: http status code OK 05:32:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:32:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:32:24 INFO [http_cmd]: http status code OK 05:32:24 DEBUG --- status code --- 05:32:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:24 DEBUG ----- output ----- 05:32:24 DEBUG --- status code --- 05:32:24 DEBUG --- http response --- 05:32:24 DEBUG ----- output ----- 05:32:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:32:24 DEBUG ----- output ----- 05:32:24 DEBUG { 05:32:24 DEBUG { 05:32:24 DEBUG "status": "success", 05:32:24 DEBUG "status": "success", 05:32:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:24 DEBUG --- http response --- 05:32:24 DEBUG "data": { 05:32:24 DEBUG "resultType": "vector", 05:32:24 DEBUG { 05:32:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:32:24 DEBUG "result": [ 05:32:24 DEBUG "data": { 05:32:24 DEBUG "status": "success", 05:32:24 DEBUG ----- output ----- 05:32:24 DEBUG { 05:32:24 DEBUG "resultType": "vector", 05:32:24 DEBUG "data": { 05:32:24 DEBUG { 05:32:24 DEBUG "status": "success", 05:32:24 DEBUG "result": [ 05:32:24 DEBUG "resultType": "vector", 05:32:24 DEBUG "data": { 05:32:24 DEBUG "metric": { 05:32:24 DEBUG { 05:32:24 DEBUG "result": [ 05:32:24 DEBUG "resultType": "vector", 05:32:24 DEBUG "instance": "10.142.0.25:9100" 05:32:24 DEBUG "metric": { 05:32:24 DEBUG { 05:32:24 DEBUG "result": [ 05:32:24 DEBUG }, 05:32:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:32:24 DEBUG "metric": { 05:32:24 DEBUG { 05:32:24 DEBUG "value": [ 05:32:24 DEBUG }, 05:32:24 DEBUG "pod": "ds-idrepo-0" 05:32:24 DEBUG "metric": { 05:32:24 DEBUG 05:32:24 DEBUG "value": [ 05:32:24 DEBUG }, 05:32:24 DEBUG "pod": "ds-idrepo-0" 05:32:24 DEBUG 05:32:24 DEBUG 05:32:24 DEBUG "value": [ 05:32:24 DEBUG 05:32:24 DEBUG }, 05:32:24 DEBUG 05:32:24 DEBUG 05:32:24 DEBUG 05:32:24 DEBUG "value": [ 05:32:24 DEBUG 05:32:24 DEBUG [...] 05:32:24 DEBUG 05:32:24 DEBUG 05:32:24 DEBUG [...] 05:32:24 DEBUG 05:32:24 DEBUG [...] 05:32:24 DEBUG 05:32:24 DEBUG 05:32:24 DEBUG 05:32:24 DEBUG 05:32:24 DEBUG 05:32:24 DEBUG 05:32:24 DEBUG 05:32:24 DEBUG 05:32:24 DEBUG [...] 05:32:24 DEBUG 05:32:24 DEBUG "instance": "10.142.0.77:9100" 05:32:24 DEBUG 05:32:24 DEBUG 05:32:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:32:24 DEBUG }, 05:32:24 DEBUG "pod": "ds-cts-2" 05:32:24 DEBUG 05:32:24 DEBUG }, 05:32:24 DEBUG "value": [ 05:32:24 DEBUG }, 05:32:24 DEBUG 05:32:24 DEBUG "value": [ 05:32:24 DEBUG 1712723544, 05:32:24 DEBUG "value": [ 05:32:24 DEBUG "pod": "ds-cts-2" 05:32:24 DEBUG 1712723544, 05:32:24 DEBUG "292335993856" 05:32:24 DEBUG 1712723544, 05:32:24 DEBUG }, 05:32:24 DEBUG "1271.4" 05:32:24 DEBUG ] 05:32:24 DEBUG "29.766666666666666" 05:32:24 DEBUG "value": [ 05:32:24 DEBUG ] 05:32:24 DEBUG } 05:32:24 DEBUG ] 05:32:24 DEBUG 1712723544, 05:32:24 DEBUG } 05:32:24 DEBUG ] 05:32:24 DEBUG } 05:32:24 DEBUG "0" 05:32:24 DEBUG ] 05:32:24 DEBUG } 05:32:24 DEBUG ] 05:32:24 DEBUG ] 05:32:24 DEBUG } 05:32:24 DEBUG } 05:32:24 DEBUG } 05:32:24 DEBUG } 05:32:24 DEBUG } 05:32:24 DEBUG } 05:32:24 DEBUG ] 05:32:24 DEBUG } 05:32:24 DEBUG } 05:32:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723544" 05:32:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723544" 05:32:24 INFO [http_cmd]: http status code OK 05:32:24 INFO [http_cmd]: http status code OK 05:32:24 DEBUG --- status code --- 05:32:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:24 DEBUG --- http response --- 05:32:24 DEBUG --- status code --- 05:32:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:32:24 DEBUG --- http response --- 05:32:24 DEBUG ----- output ----- 05:32:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:32:24 DEBUG { 05:32:24 DEBUG "status": "success", 05:32:24 DEBUG "data": { 05:32:24 DEBUG "resultType": "vector", 05:32:24 DEBUG "result": [ 05:32:24 DEBUG { 05:32:24 DEBUG "metric": { 05:32:24 DEBUG "instance": "10.142.0.25:9100" 05:32:24 DEBUG }, 05:32:24 DEBUG "value": [ 05:32:24 DEBUG 05:32:24 DEBUG 05:32:24 DEBUG 05:32:24 DEBUG [...] 05:32:24 DEBUG 05:32:24 DEBUG 05:32:24 DEBUG 05:32:24 DEBUG "instance": "10.142.0.77:9100" 05:32:24 DEBUG }, 05:32:24 DEBUG "value": [ 05:32:24 DEBUG 1712723544, 05:32:24 DEBUG "37636190.0476619" 05:32:24 DEBUG ] 05:32:24 DEBUG ----- output ----- 05:32:24 DEBUG } 05:32:24 DEBUG { 05:32:24 DEBUG "status": "success", 05:32:24 DEBUG "data": { 05:32:24 DEBUG "resultType": "vector", 05:32:24 DEBUG "result": [ 05:32:24 DEBUG { 05:32:24 DEBUG "metric": { 05:32:24 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:32:24 DEBUG }, 05:32:24 DEBUG "value": [ 05:32:24 DEBUG 05:32:24 DEBUG 05:32:24 DEBUG 05:32:24 DEBUG ] 05:32:24 DEBUG [...] 05:32:24 DEBUG 05:32:24 DEBUG 05:32:24 DEBUG } 05:32:24 DEBUG 05:32:24 DEBUG } 05:32:24 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 05:32:24 DEBUG }, 05:32:24 DEBUG "value": [ 05:32:24 DEBUG 1712723544, 05:32:24 DEBUG "0" 05:32:24 DEBUG ] 05:32:24 DEBUG } 05:32:24 DEBUG ] 05:32:24 DEBUG } 05:32:24 DEBUG } 05:32:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723544" 05:32:24 INFO [http_cmd]: http status code OK 05:32:24 DEBUG --- status code --- 05:32:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:24 DEBUG --- http response --- 05:32:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:32:24 DEBUG ----- output ----- 05:32:24 DEBUG { 05:32:24 DEBUG "status": "success", 05:32:24 DEBUG "data": { 05:32:24 DEBUG "resultType": "vector", 05:32:24 DEBUG "result": [ 05:32:24 DEBUG { 05:32:24 DEBUG "metric": { 05:32:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:32:24 DEBUG }, 05:32:24 DEBUG "value": [ 05:32:24 DEBUG 05:32:24 DEBUG 05:32:24 DEBUG 05:32:24 DEBUG [...] 05:32:24 DEBUG 05:32:24 DEBUG 05:32:24 DEBUG 05:32:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:32:24 DEBUG }, 05:32:24 DEBUG "value": [ 05:32:24 DEBUG 1712723544, 05:32:24 DEBUG "227634.5" 05:32:24 DEBUG ] 05:32:24 DEBUG } 05:32:24 DEBUG ] 05:32:24 DEBUG } 05:32:24 DEBUG } 05:32:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723545" 05:32:25 INFO [http_cmd]: http status code OK 05:32:25 DEBUG --- status code --- 05:32:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:25 DEBUG --- http response --- 05:32:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:32:25 DEBUG ----- output ----- 05:32:25 DEBUG { 05:32:25 DEBUG "status": "success", 05:32:25 DEBUG "data": { 05:32:25 DEBUG "resultType": "vector", 05:32:25 DEBUG "result": [ 05:32:25 DEBUG { 05:32:25 DEBUG "metric": { 05:32:25 DEBUG "pod": "ds-idrepo-0" 05:32:25 DEBUG }, 05:32:25 DEBUG "value": [ 05:32:25 DEBUG 05:32:25 DEBUG 05:32:25 DEBUG 05:32:25 DEBUG [...] 05:32:25 DEBUG 05:32:25 DEBUG 05:32:25 DEBUG 05:32:25 DEBUG "pod": "ds-idrepo-1" 05:32:25 DEBUG }, 05:32:25 DEBUG "value": [ 05:32:25 DEBUG 1712723545, 05:32:25 DEBUG "0" 05:32:25 DEBUG ] 05:32:25 DEBUG } 05:32:25 DEBUG ] 05:32:25 DEBUG } 05:32:25 DEBUG } 05:32:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723545" 05:32:25 INFO [http_cmd]: http status code OK 05:32:25 DEBUG --- status code --- 05:32:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:25 DEBUG --- http response --- 05:32:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:32:25 DEBUG ----- output ----- 05:32:25 DEBUG { 05:32:25 DEBUG "status": "success", 05:32:25 DEBUG "data": { 05:32:25 DEBUG "resultType": "vector", 05:32:25 DEBUG "result": [ 05:32:25 DEBUG { 05:32:25 DEBUG "metric": { 05:32:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:32:25 DEBUG }, 05:32:25 DEBUG "value": [ 05:32:25 DEBUG 05:32:25 DEBUG 05:32:25 DEBUG 05:32:25 DEBUG [...] 05:32:25 DEBUG 05:32:25 DEBUG 05:32:25 DEBUG 05:32:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:32:25 DEBUG }, 05:32:25 DEBUG "value": [ 05:32:25 DEBUG 1712723545, 05:32:25 DEBUG "1271.4" 05:32:25 DEBUG ] 05:32:25 DEBUG } 05:32:25 DEBUG ] 05:32:25 DEBUG } 05:32:25 DEBUG } 05:32:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723545" 05:32:25 INFO [http_cmd]: http status code OK 05:32:25 DEBUG --- status code --- 05:32:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:25 DEBUG --- http response --- 05:32:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:32:25 DEBUG ----- output ----- 05:32:25 DEBUG { 05:32:25 DEBUG "status": "success", 05:32:25 DEBUG "data": { 05:32:25 DEBUG "resultType": "vector", 05:32:25 DEBUG "result": [ 05:32:25 DEBUG { 05:32:25 DEBUG "metric": { 05:32:25 DEBUG "instance": "10.142.0.25:9100" 05:32:25 DEBUG }, 05:32:25 DEBUG "value": [ 05:32:25 DEBUG 05:32:25 DEBUG 05:32:25 DEBUG 05:32:25 DEBUG [...] 05:32:25 DEBUG 05:32:25 DEBUG 05:32:25 DEBUG 05:32:25 DEBUG "instance": "10.142.0.77:9100" 05:32:25 DEBUG }, 05:32:25 DEBUG "value": [ 05:32:25 DEBUG 1712723545, 05:32:25 DEBUG "0.0359964003599643" 05:32:25 DEBUG ] 05:32:25 DEBUG } 05:32:25 DEBUG ] 05:32:25 DEBUG } 05:32:25 DEBUG } 05:32:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723546" 05:32:26 INFO [http_cmd]: http status code OK 05:32:26 DEBUG --- status code --- 05:32:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:26 DEBUG --- http response --- 05:32:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:32:26 DEBUG ----- output ----- 05:32:26 DEBUG { 05:32:26 DEBUG "status": "success", 05:32:26 DEBUG "data": { 05:32:26 DEBUG "resultType": "vector", 05:32:26 DEBUG "result": [ 05:32:26 DEBUG { 05:32:26 DEBUG "metric": { 05:32:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:32:26 DEBUG }, 05:32:26 DEBUG "value": [ 05:32:26 DEBUG 05:32:26 DEBUG 05:32:26 DEBUG 05:32:26 DEBUG [...] 05:32:26 DEBUG 05:32:26 DEBUG 05:32:26 DEBUG 05:32:26 DEBUG "pod": "ds-idrepo-2" 05:32:26 DEBUG }, 05:32:26 DEBUG "value": [ 05:32:26 DEBUG 1712723546, 05:32:26 DEBUG "0" 05:32:26 DEBUG ] 05:32:26 DEBUG } 05:32:26 DEBUG ] 05:32:26 DEBUG } 05:32:26 DEBUG } 05:32:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712723546" 05:32:26 INFO [http_cmd]: http status code OK 05:32:26 DEBUG --- status code --- 05:32:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:26 DEBUG --- http response --- 05:32:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:32:26 DEBUG ----- output ----- 05:32:26 DEBUG { 05:32:26 DEBUG "status": "success", 05:32:26 DEBUG "data": { 05:32:26 DEBUG "resultType": "vector", 05:32:26 DEBUG "result": [ 05:32:26 DEBUG { 05:32:26 DEBUG "metric": { 05:32:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:32:26 DEBUG }, 05:32:26 DEBUG "value": [ 05:32:26 DEBUG 05:32:26 DEBUG 05:32:26 DEBUG 05:32:26 DEBUG [...] 05:32:26 DEBUG 05:32:26 DEBUG 05:32:26 DEBUG 05:32:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:32:26 DEBUG }, 05:32:26 DEBUG "value": [ 05:32:26 DEBUG 1712723546, 05:32:26 DEBUG "3.633298710024277" 05:32:26 DEBUG ] 05:32:26 DEBUG } 05:32:26 DEBUG ] 05:32:26 DEBUG } 05:32:26 DEBUG } 05:32:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712723546" 05:32:26 INFO [http_cmd]: http status code OK 05:32:26 DEBUG --- status code --- 05:32:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:26 DEBUG --- http response --- 05:32:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:32:26 DEBUG ----- output ----- 05:32:26 DEBUG { 05:32:26 DEBUG "status": "success", 05:32:26 DEBUG "data": { 05:32:26 DEBUG "resultType": "vector", 05:32:26 DEBUG "result": [ 05:32:26 DEBUG { 05:32:26 DEBUG "metric": { 05:32:26 DEBUG "pod": "ds-idrepo-0" 05:32:26 DEBUG }, 05:32:26 DEBUG "value": [ 05:32:26 DEBUG 05:32:26 DEBUG 05:32:26 DEBUG 05:32:26 DEBUG [...] 05:32:26 DEBUG 05:32:26 DEBUG 05:32:26 DEBUG 05:32:26 DEBUG "pod": "ds-cts-2" 05:32:26 DEBUG }, 05:32:26 DEBUG "value": [ 05:32:26 DEBUG 1712723546, 05:32:26 DEBUG "0" 05:32:26 DEBUG ] 05:32:26 DEBUG } 05:32:26 DEBUG ] 05:32:26 DEBUG } 05:32:26 DEBUG } 05:32:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723546" 05:32:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723546" 05:32:26 INFO [http_cmd]: http status code OK 05:32:26 DEBUG --- status code --- 05:32:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:26 DEBUG --- http response --- 05:32:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:32:26 DEBUG ----- output ----- 05:32:26 DEBUG { 05:32:26 DEBUG "status": "success", 05:32:26 DEBUG "data": { 05:32:26 DEBUG "resultType": "vector", 05:32:26 DEBUG "result": [ 05:32:26 DEBUG { 05:32:26 DEBUG "metric": { 05:32:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:32:26 DEBUG }, 05:32:26 DEBUG "value": [ 05:32:26 DEBUG 05:32:26 DEBUG 05:32:26 DEBUG 05:32:26 DEBUG [...] 05:32:26 DEBUG 05:32:26 DEBUG 05:32:26 DEBUG 05:32:26 DEBUG "pod": "ds-idrepo-2" 05:32:26 DEBUG }, 05:32:26 DEBUG "value": [ 05:32:26 DEBUG 1712723546, 05:32:26 DEBUG "0" 05:32:26 DEBUG ] 05:32:26 DEBUG } 05:32:26 DEBUG ] 05:32:26 DEBUG } 05:32:26 DEBUG } 05:32:26 INFO [http_cmd]: http status code OK 05:32:26 DEBUG --- status code --- 05:32:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:26 DEBUG --- http response --- 05:32:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:32:26 DEBUG ----- output ----- 05:32:26 DEBUG { 05:32:26 DEBUG "status": "success", 05:32:26 DEBUG "data": { 05:32:26 DEBUG "resultType": "vector", 05:32:26 DEBUG "result": [ 05:32:26 DEBUG { 05:32:26 DEBUG "metric": { 05:32:26 DEBUG "pod": "ds-idrepo-2" 05:32:26 DEBUG }, 05:32:26 DEBUG "value": [ 05:32:26 DEBUG 05:32:26 DEBUG 05:32:26 DEBUG 05:32:26 DEBUG [...] 05:32:26 DEBUG 05:32:26 DEBUG 05:32:26 DEBUG 05:32:26 DEBUG "pod": "ds-cts-2" 05:32:26 DEBUG }, 05:32:26 DEBUG "value": [ 05:32:26 DEBUG 1712723546, 05:32:26 DEBUG "0.00045399666666666666" 05:32:26 DEBUG ] 05:32:26 DEBUG } 05:32:26 DEBUG ] 05:32:26 DEBUG } 05:32:26 DEBUG } 05:32:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723546" 05:32:26 INFO [http_cmd]: http status code OK 05:32:26 DEBUG --- status code --- 05:32:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:26 DEBUG --- http response --- 05:32:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:32:26 DEBUG ----- output ----- 05:32:26 DEBUG { 05:32:26 DEBUG "status": "success", 05:32:26 DEBUG "data": { 05:32:26 DEBUG "resultType": "vector", 05:32:26 DEBUG "result": [ 05:32:26 DEBUG { 05:32:26 DEBUG "metric": { 05:32:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:32:26 DEBUG }, 05:32:26 DEBUG "value": [ 05:32:26 DEBUG 05:32:26 DEBUG 05:32:26 DEBUG 05:32:26 DEBUG [...] 05:32:26 DEBUG 05:32:26 DEBUG 05:32:26 DEBUG 05:32:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:32:26 DEBUG }, 05:32:26 DEBUG "value": [ 05:32:26 DEBUG 1712723546, 05:32:26 DEBUG "1271.4333333333334" 05:32:26 DEBUG ] 05:32:26 DEBUG } 05:32:26 DEBUG ] 05:32:26 DEBUG } 05:32:26 DEBUG } 05:32:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712723547" 05:32:27 INFO [http_cmd]: http status code OK 05:32:27 DEBUG --- status code --- 05:32:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:27 DEBUG --- http response --- 05:32:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:32:27 DEBUG ----- output ----- 05:32:27 DEBUG { 05:32:27 DEBUG "status": "success", 05:32:27 DEBUG "data": { 05:32:27 DEBUG "resultType": "vector", 05:32:27 DEBUG "result": [ 05:32:27 DEBUG { 05:32:27 DEBUG "metric": { 05:32:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:32:27 DEBUG }, 05:32:27 DEBUG "value": [ 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG [...] 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:32:27 DEBUG }, 05:32:27 DEBUG "value": [ 05:32:27 DEBUG 1712723547, 05:32:27 DEBUG "0.042" 05:32:27 DEBUG ] 05:32:27 DEBUG } 05:32:27 DEBUG ] 05:32:27 DEBUG } 05:32:27 DEBUG } 05:32:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723547" 05:32:27 INFO [http_cmd]: http status code OK 05:32:27 DEBUG --- status code --- 05:32:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:27 DEBUG --- http response --- 05:32:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:32:27 DEBUG ----- output ----- 05:32:27 DEBUG { 05:32:27 DEBUG "status": "success", 05:32:27 DEBUG "data": { 05:32:27 DEBUG "resultType": "vector", 05:32:27 DEBUG "result": [ 05:32:27 DEBUG { 05:32:27 DEBUG "metric": { 05:32:27 DEBUG "pod": "ds-idrepo-0" 05:32:27 DEBUG }, 05:32:27 DEBUG "value": [ 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG [...] 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG "pod": "ds-idrepo-1" 05:32:27 DEBUG }, 05:32:27 DEBUG "value": [ 05:32:27 DEBUG 1712723547, 05:32:27 DEBUG "0" 05:32:27 DEBUG ] 05:32:27 DEBUG } 05:32:27 DEBUG ] 05:32:27 DEBUG } 05:32:27 DEBUG } 05:32:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723547" 05:32:27 INFO [http_cmd]: http status code OK 05:32:27 DEBUG --- status code --- 05:32:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:27 DEBUG --- http response --- 05:32:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:32:27 DEBUG ----- output ----- 05:32:27 DEBUG { 05:32:27 DEBUG "status": "success", 05:32:27 DEBUG "data": { 05:32:27 DEBUG "resultType": "vector", 05:32:27 DEBUG "result": [ 05:32:27 DEBUG { 05:32:27 DEBUG "metric": { 05:32:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:32:27 DEBUG }, 05:32:27 DEBUG "value": [ 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG [...] 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:32:27 DEBUG }, 05:32:27 DEBUG "value": [ 05:32:27 DEBUG 1712723547, 05:32:27 DEBUG "1260.9333333333334" 05:32:27 DEBUG ] 05:32:27 DEBUG } 05:32:27 DEBUG ] 05:32:27 DEBUG } 05:32:27 DEBUG } 05:32:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723547" 05:32:27 INFO [http_cmd]: http status code OK 05:32:27 DEBUG --- status code --- 05:32:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:27 DEBUG --- http response --- 05:32:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:32:27 DEBUG ----- output ----- 05:32:27 DEBUG { 05:32:27 DEBUG "status": "success", 05:32:27 DEBUG "data": { 05:32:27 DEBUG "resultType": "vector", 05:32:27 DEBUG "result": [ 05:32:27 DEBUG { 05:32:27 DEBUG "metric": { 05:32:27 DEBUG "pod": "ds-idrepo-0" 05:32:27 DEBUG }, 05:32:27 DEBUG "value": [ 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG [...] 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG "pod": "ds-idrepo-1" 05:32:27 DEBUG }, 05:32:27 DEBUG "value": [ 05:32:27 DEBUG 1712723547, 05:32:27 DEBUG "0" 05:32:27 DEBUG ] 05:32:27 DEBUG } 05:32:27 DEBUG ] 05:32:27 DEBUG } 05:32:27 DEBUG } 05:32:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712723547" 05:32:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723547" 05:32:27 INFO [http_cmd]: http status code OK 05:32:27 INFO [http_cmd]: http status code OK 05:32:27 DEBUG --- status code --- 05:32:27 DEBUG --- status code --- 05:32:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:27 DEBUG --- http response --- 05:32:27 DEBUG --- http response --- 05:32:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:32:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 05:32:27 DEBUG ----- output ----- 05:32:27 DEBUG ----- output ----- 05:32:27 DEBUG { 05:32:27 DEBUG { 05:32:27 DEBUG "status": "success", 05:32:27 DEBUG "status": "success", 05:32:27 DEBUG "data": { 05:32:27 DEBUG "data": { 05:32:27 DEBUG "resultType": "vector", 05:32:27 DEBUG "resultType": "vector", 05:32:27 DEBUG "result": [ 05:32:27 DEBUG "result": [ 05:32:27 DEBUG { 05:32:27 DEBUG { 05:32:27 DEBUG "metric": { 05:32:27 DEBUG "metric": { 05:32:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:32:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:32:27 DEBUG }, 05:32:27 DEBUG }, 05:32:27 DEBUG "value": [ 05:32:27 DEBUG "value": [ 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG [...] 05:32:27 DEBUG [...] 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 05:32:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:32:27 DEBUG }, 05:32:27 DEBUG }, 05:32:27 DEBUG "value": [ 05:32:27 DEBUG "value": [ 05:32:27 DEBUG 1712723547, 05:32:27 DEBUG 1712723547, 05:32:27 DEBUG "0" 05:32:27 DEBUG "2490404864" 05:32:27 DEBUG ] 05:32:27 DEBUG ] 05:32:27 DEBUG } 05:32:27 DEBUG } 05:32:27 DEBUG ] 05:32:27 DEBUG ] 05:32:27 DEBUG } 05:32:27 DEBUG } 05:32:27 DEBUG } 05:32:27 DEBUG } 05:32:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723547" 05:32:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723547" 05:32:27 INFO [http_cmd]: http status code OK 05:32:27 INFO [http_cmd]: http status code OK 05:32:27 DEBUG --- status code --- 05:32:27 DEBUG --- status code --- 05:32:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:32:27 DEBUG --- http response --- 05:32:27 DEBUG --- http response --- 05:32:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:32:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:32:27 DEBUG ----- output ----- 05:32:27 DEBUG ----- output ----- 05:32:27 DEBUG { 05:32:27 DEBUG { 05:32:27 DEBUG "status": "success", 05:32:27 DEBUG "status": "success", 05:32:27 DEBUG "data": { 05:32:27 DEBUG "data": { 05:32:27 DEBUG "resultType": "vector", 05:32:27 DEBUG "resultType": "vector", 05:32:27 DEBUG "result": [ 05:32:27 DEBUG "result": [ 05:32:27 DEBUG { 05:32:27 DEBUG { 05:32:27 DEBUG "metric": { 05:32:27 DEBUG "metric": { 05:32:27 DEBUG "instance": "10.142.0.25:9100" 05:32:27 DEBUG "instance": "10.142.0.25:9100" 05:32:27 DEBUG }, 05:32:27 DEBUG }, 05:32:27 DEBUG "value": [ 05:32:27 DEBUG "value": [ 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG [...] 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG 05:32:27 DEBUG "instance": "10.142.0.77:9100" 05:32:27 DEBUG }, 05:32:27 DEBUG "value": [ 05:32:27 DEBUG [...] 05:32:27 DEBUG 1712723547, 05:32:27 DEBUG 05:32:27 DEBUG "0.01399860013998606" 05:32:27 DEBUG 05:32:27 DEBUG ] 05:32:27 DEBUG 05:32:27 DEBUG } 05:32:27 DEBUG "instance": "10.142.0.77:9100" 05:32:27 DEBUG ] 05:32:27 DEBUG }, 05:32:27 DEBUG } 05:32:27 DEBUG "value": [ 05:32:27 DEBUG } 05:32:27 DEBUG 1712723547, 05:32:27 DEBUG "37635600.70659601" 05:32:27 DEBUG ] 05:32:27 DEBUG } 05:32:27 DEBUG ] 05:32:27 DEBUG } 05:32:27 DEBUG } 05:32:42 INFO 05:32:42 INFO [loop_until]: kubectl --namespace=xlou top pods 05:32:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:32:42 INFO [loop_until]: OK (rc = 0) 05:32:42 DEBUG --- stdout --- 05:32:42 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 8829m 8806Mi am-654c4fcbcf-9scrj 8557m 8421Mi am-654c4fcbcf-v5qb5 7362m 8401Mi ds-cts-0 7170m 13308Mi ds-cts-1 7497m 13210Mi ds-cts-2 6821m 12653Mi ds-idrepo-0 9579m 5621Mi ds-idrepo-1 321m 5669Mi ds-idrepo-2 325m 5698Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2212Mi lodemon-b776b7d88-thrqf 9m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3609m 1666Mi 05:32:42 DEBUG --- stderr --- 05:32:42 DEBUG 05:32:49 INFO 05:32:49 INFO [loop_until]: kubectl --namespace=xlou top node 05:32:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:32:49 INFO [loop_until]: OK (rc = 0) 05:32:49 DEBUG --- stdout --- 05:32:49 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 95m 0% 2307Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8219m 51% 9279Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 82m 0% 3260Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10778m 67% 6521Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9516m 59% 9310Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9905m 62% 9647Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 76m 0% 854Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9371m 58% 6281Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 7408m 46% 13878Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 7158m 45% 13293Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 385m 2% 6367Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7643m 48% 13898Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 390m 2% 6330Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3578m 22% 5346Mi 9% 05:32:49 DEBUG --- stderr --- 05:32:49 DEBUG 05:33:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712723604" 05:33:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723604" 05:33:24 ERROR [http_cmd]: ERROR 05:33:24 INFO [http_cmd]: http status code OK 05:33:24 ERROR --- status code --- 05:33:24 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:33:24 DEBUG --- status code --- 05:33:24 ERROR --- http response --- 05:33:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:24 DEBUG --- http response --- 05:33:24 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:33:24 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:33:24 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:33:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723604" 05:33:24 INFO [http_cmd]: http status code OK 05:33:24 DEBUG --- status code --- 05:33:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:24 DEBUG --- http response --- 05:33:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:33:24 DEBUG ----- output ----- 05:33:24 DEBUG { 05:33:24 DEBUG "status": "success", 05:33:24 DEBUG "data": { 05:33:24 DEBUG "resultType": "vector", 05:33:24 DEBUG "result": [ 05:33:24 DEBUG { 05:33:24 DEBUG "metric": { 05:33:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:33:24 DEBUG }, 05:33:24 DEBUG "value": [ 05:33:24 DEBUG 05:33:24 DEBUG 05:33:24 DEBUG 05:33:24 DEBUG [...] 05:33:24 DEBUG 05:33:24 DEBUG 05:33:24 DEBUG 05:33:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:33:24 DEBUG }, 05:33:24 DEBUG "value": [ 05:33:24 DEBUG 1712723604, 05:33:24 DEBUG "0" 05:33:24 DEBUG ] 05:33:24 DEBUG } 05:33:24 DEBUG ] 05:33:24 DEBUG } 05:33:24 DEBUG } 05:33:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723604" 05:33:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723604" 05:33:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723604" 05:33:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723604" 05:33:24 INFO [http_cmd]: http status code OK 05:33:24 DEBUG --- status code --- 05:33:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:24 DEBUG --- http response --- 05:33:24 INFO [http_cmd]: http status code OK 05:33:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:33:24 INFO [http_cmd]: http status code OK 05:33:24 DEBUG --- status code --- 05:33:24 INFO [http_cmd]: http status code OK 05:33:24 DEBUG ----- output ----- 05:33:24 DEBUG --- status code --- 05:33:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:24 DEBUG --- status code --- 05:33:24 DEBUG { 05:33:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:24 DEBUG --- http response --- 05:33:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:24 DEBUG "status": "success", 05:33:24 DEBUG --- http response --- 05:33:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:33:24 DEBUG --- http response --- 05:33:24 DEBUG "data": { 05:33:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:33:24 DEBUG ----- output ----- 05:33:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:33:24 DEBUG "resultType": "vector", 05:33:24 DEBUG ----- output ----- 05:33:24 DEBUG { 05:33:24 DEBUG ----- output ----- 05:33:24 DEBUG "result": [ 05:33:24 DEBUG { 05:33:24 DEBUG "status": "success", 05:33:24 DEBUG { 05:33:24 DEBUG { 05:33:24 DEBUG "status": "success", 05:33:24 DEBUG "data": { 05:33:24 DEBUG "status": "success", 05:33:24 DEBUG "metric": { 05:33:24 DEBUG "data": { 05:33:24 DEBUG "resultType": "vector", 05:33:24 DEBUG "data": { 05:33:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:33:24 DEBUG "resultType": "vector", 05:33:24 DEBUG "result": [ 05:33:24 DEBUG "resultType": "vector", 05:33:24 DEBUG }, 05:33:24 DEBUG "result": [ 05:33:24 DEBUG { 05:33:24 DEBUG "result": [ 05:33:24 DEBUG { 05:33:24 DEBUG { 05:33:24 DEBUG "metric": { 05:33:24 DEBUG "value": [ 05:33:24 DEBUG "metric": { 05:33:24 DEBUG "metric": { 05:33:24 DEBUG "pod": "ds-idrepo-0" 05:33:24 DEBUG 05:33:24 DEBUG "pod": "ds-idrepo-0" 05:33:24 DEBUG "instance": "10.142.0.25:9100" 05:33:24 DEBUG }, 05:33:24 DEBUG 05:33:24 DEBUG }, 05:33:24 DEBUG }, 05:33:24 DEBUG "value": [ 05:33:24 DEBUG 05:33:24 DEBUG "value": [ 05:33:24 DEBUG "value": [ 05:33:24 DEBUG 05:33:24 DEBUG [...] 05:33:24 DEBUG 05:33:24 DEBUG 05:33:24 DEBUG 05:33:24 DEBUG 05:33:24 DEBUG 05:33:24 DEBUG 05:33:24 DEBUG 05:33:24 DEBUG 05:33:24 DEBUG [...] 05:33:24 DEBUG 05:33:24 DEBUG [...] 05:33:24 DEBUG 05:33:24 DEBUG 05:33:24 DEBUG 05:33:24 DEBUG 05:33:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:33:24 DEBUG 05:33:24 DEBUG [...] 05:33:24 DEBUG 05:33:24 DEBUG }, 05:33:24 DEBUG 05:33:24 DEBUG 05:33:24 DEBUG 05:33:24 DEBUG "value": [ 05:33:24 DEBUG "pod": "ds-cts-2" 05:33:24 DEBUG 05:33:24 DEBUG "pod": "ds-cts-2" 05:33:24 DEBUG 1712723604, 05:33:24 DEBUG }, 05:33:24 DEBUG 05:33:24 DEBUG }, 05:33:24 DEBUG "1258.1666666666667" 05:33:24 DEBUG "value": [ 05:33:24 DEBUG "instance": "10.142.0.77:9100" 05:33:24 DEBUG "value": [ 05:33:24 DEBUG ] 05:33:24 DEBUG 1712723604, 05:33:24 DEBUG }, 05:33:24 DEBUG 1712723604, 05:33:24 DEBUG } 05:33:24 DEBUG "29.9" 05:33:24 DEBUG "value": [ 05:33:24 DEBUG "0" 05:33:24 DEBUG ] 05:33:24 DEBUG ] 05:33:24 DEBUG 1712723604, 05:33:24 DEBUG ] 05:33:24 DEBUG } 05:33:24 DEBUG } 05:33:24 DEBUG "294268984320" 05:33:24 DEBUG } 05:33:24 DEBUG } 05:33:24 DEBUG ] 05:33:24 DEBUG ] 05:33:24 DEBUG ] 05:33:24 DEBUG } 05:33:24 DEBUG } 05:33:24 DEBUG } 05:33:24 DEBUG } 05:33:24 DEBUG ] 05:33:24 DEBUG } 05:33:24 DEBUG } 05:33:24 DEBUG } 05:33:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723604" 05:33:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723604" 05:33:24 INFO [http_cmd]: http status code OK 05:33:24 DEBUG --- status code --- 05:33:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:24 DEBUG --- http response --- 05:33:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:33:24 DEBUG ----- output ----- 05:33:24 DEBUG { 05:33:24 DEBUG "status": "success", 05:33:24 DEBUG "data": { 05:33:24 INFO [http_cmd]: http status code OK 05:33:24 DEBUG "resultType": "vector", 05:33:24 DEBUG --- status code --- 05:33:24 DEBUG "result": [ 05:33:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:24 DEBUG { 05:33:24 DEBUG --- http response --- 05:33:24 DEBUG "metric": { 05:33:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:33:24 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:33:24 DEBUG ----- output ----- 05:33:24 DEBUG }, 05:33:24 DEBUG { 05:33:24 DEBUG "value": [ 05:33:24 DEBUG "status": "success", 05:33:24 DEBUG 05:33:24 DEBUG "data": { 05:33:24 DEBUG 05:33:24 DEBUG "resultType": "vector", 05:33:24 DEBUG 05:33:24 DEBUG "result": [ 05:33:24 DEBUG [...] 05:33:24 DEBUG { 05:33:24 DEBUG 05:33:24 DEBUG "metric": { 05:33:24 DEBUG 05:33:24 DEBUG "instance": "10.142.0.25:9100" 05:33:24 DEBUG 05:33:24 DEBUG }, 05:33:24 DEBUG "pod": "ds-idrepo-1" 05:33:24 DEBUG "value": [ 05:33:24 DEBUG }, 05:33:24 DEBUG 05:33:24 DEBUG "value": [ 05:33:24 DEBUG 05:33:24 DEBUG 1712723604, 05:33:24 DEBUG 05:33:24 DEBUG "0" 05:33:24 DEBUG [...] 05:33:24 DEBUG ] 05:33:24 DEBUG 05:33:24 DEBUG } 05:33:24 DEBUG 05:33:24 DEBUG ] 05:33:24 DEBUG 05:33:24 DEBUG } 05:33:24 DEBUG "instance": "10.142.0.77:9100" 05:33:24 DEBUG } 05:33:24 DEBUG }, 05:33:24 DEBUG "value": [ 05:33:24 DEBUG 1712723604, 05:33:24 DEBUG "37652791.64583125" 05:33:24 DEBUG ] 05:33:24 DEBUG } 05:33:24 DEBUG ] 05:33:24 DEBUG } 05:33:24 DEBUG } 05:33:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723604" 05:33:24 INFO [http_cmd]: http status code OK 05:33:24 DEBUG --- status code --- 05:33:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:24 DEBUG --- http response --- 05:33:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:33:24 DEBUG ----- output ----- 05:33:24 DEBUG { 05:33:24 DEBUG "status": "success", 05:33:24 DEBUG "data": { 05:33:24 DEBUG "resultType": "vector", 05:33:24 DEBUG "result": [ 05:33:24 DEBUG { 05:33:24 DEBUG "metric": { 05:33:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:33:24 DEBUG }, 05:33:24 DEBUG "value": [ 05:33:24 DEBUG 05:33:24 DEBUG 05:33:24 DEBUG 05:33:24 DEBUG [...] 05:33:24 DEBUG 05:33:24 DEBUG 05:33:24 DEBUG 05:33:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:33:24 DEBUG }, 05:33:24 DEBUG "value": [ 05:33:24 DEBUG 1712723604, 05:33:24 DEBUG "227590" 05:33:24 DEBUG ] 05:33:24 DEBUG } 05:33:24 DEBUG ] 05:33:24 DEBUG } 05:33:24 DEBUG } 05:33:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723605" 05:33:25 INFO [http_cmd]: http status code OK 05:33:25 DEBUG --- status code --- 05:33:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:25 DEBUG --- http response --- 05:33:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:33:25 DEBUG ----- output ----- 05:33:25 DEBUG { 05:33:25 DEBUG "status": "success", 05:33:25 DEBUG "data": { 05:33:25 DEBUG "resultType": "vector", 05:33:25 DEBUG "result": [ 05:33:25 DEBUG { 05:33:25 DEBUG "metric": { 05:33:25 DEBUG "pod": "ds-idrepo-0" 05:33:25 DEBUG }, 05:33:25 DEBUG "value": [ 05:33:25 DEBUG 05:33:25 DEBUG 05:33:25 DEBUG 05:33:25 DEBUG [...] 05:33:25 DEBUG 05:33:25 DEBUG 05:33:25 DEBUG 05:33:25 DEBUG "pod": "ds-idrepo-1" 05:33:25 DEBUG }, 05:33:25 DEBUG "value": [ 05:33:25 DEBUG 1712723605, 05:33:25 DEBUG "0" 05:33:25 DEBUG ] 05:33:25 DEBUG } 05:33:25 DEBUG ] 05:33:25 DEBUG } 05:33:25 DEBUG } 05:33:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723605" 05:33:25 INFO [http_cmd]: http status code OK 05:33:25 DEBUG --- status code --- 05:33:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:25 DEBUG --- http response --- 05:33:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:33:25 DEBUG ----- output ----- 05:33:25 DEBUG { 05:33:25 DEBUG "status": "success", 05:33:25 DEBUG "data": { 05:33:25 DEBUG "resultType": "vector", 05:33:25 DEBUG "result": [ 05:33:25 DEBUG { 05:33:25 DEBUG "metric": { 05:33:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:33:25 DEBUG }, 05:33:25 DEBUG "value": [ 05:33:25 DEBUG 05:33:25 DEBUG 05:33:25 DEBUG 05:33:25 DEBUG [...] 05:33:25 DEBUG 05:33:25 DEBUG 05:33:25 DEBUG 05:33:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:33:25 DEBUG }, 05:33:25 DEBUG "value": [ 05:33:25 DEBUG 1712723605, 05:33:25 DEBUG "1263.4666666666667" 05:33:25 DEBUG ] 05:33:25 DEBUG } 05:33:25 DEBUG ] 05:33:25 DEBUG } 05:33:25 DEBUG } 05:33:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723605" 05:33:25 INFO [http_cmd]: http status code OK 05:33:25 DEBUG --- status code --- 05:33:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:25 DEBUG --- http response --- 05:33:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:33:25 DEBUG ----- output ----- 05:33:25 DEBUG { 05:33:25 DEBUG "status": "success", 05:33:25 DEBUG "data": { 05:33:25 DEBUG "resultType": "vector", 05:33:25 DEBUG "result": [ 05:33:25 DEBUG { 05:33:25 DEBUG "metric": { 05:33:25 DEBUG "instance": "10.142.0.25:9100" 05:33:25 DEBUG }, 05:33:25 DEBUG "value": [ 05:33:25 DEBUG 05:33:25 DEBUG 05:33:25 DEBUG 05:33:25 DEBUG [...] 05:33:25 DEBUG 05:33:25 DEBUG 05:33:25 DEBUG 05:33:25 DEBUG "instance": "10.142.0.77:9100" 05:33:25 DEBUG }, 05:33:25 DEBUG "value": [ 05:33:25 DEBUG 1712723605, 05:33:25 DEBUG "0.1531819848651536" 05:33:25 DEBUG ] 05:33:25 DEBUG } 05:33:25 DEBUG ] 05:33:25 DEBUG } 05:33:25 DEBUG } 05:33:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723606" 05:33:26 INFO [http_cmd]: http status code OK 05:33:26 DEBUG --- status code --- 05:33:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:26 DEBUG --- http response --- 05:33:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:33:26 DEBUG ----- output ----- 05:33:26 DEBUG { 05:33:26 DEBUG "status": "success", 05:33:26 DEBUG "data": { 05:33:26 DEBUG "resultType": "vector", 05:33:26 DEBUG "result": [ 05:33:26 DEBUG { 05:33:26 DEBUG "metric": { 05:33:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:33:26 DEBUG }, 05:33:26 DEBUG "value": [ 05:33:26 DEBUG 05:33:26 DEBUG 05:33:26 DEBUG 05:33:26 DEBUG [...] 05:33:26 DEBUG 05:33:26 DEBUG 05:33:26 DEBUG 05:33:26 DEBUG "pod": "ds-idrepo-1" 05:33:26 DEBUG }, 05:33:26 DEBUG "value": [ 05:33:26 DEBUG 1712723606, 05:33:26 DEBUG "0" 05:33:26 DEBUG ] 05:33:26 DEBUG } 05:33:26 DEBUG ] 05:33:26 DEBUG } 05:33:26 DEBUG } 05:33:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712723606" 05:33:26 INFO [http_cmd]: http status code OK 05:33:26 DEBUG --- status code --- 05:33:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:26 DEBUG --- http response --- 05:33:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:33:26 DEBUG ----- output ----- 05:33:26 DEBUG { 05:33:26 DEBUG "status": "success", 05:33:26 DEBUG "data": { 05:33:26 DEBUG "resultType": "vector", 05:33:26 DEBUG "result": [ 05:33:26 DEBUG { 05:33:26 DEBUG "metric": { 05:33:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:33:26 DEBUG }, 05:33:26 DEBUG "value": [ 05:33:26 DEBUG 05:33:26 DEBUG 05:33:26 DEBUG 05:33:26 DEBUG [...] 05:33:26 DEBUG 05:33:26 DEBUG 05:33:26 DEBUG 05:33:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:33:26 DEBUG }, 05:33:26 DEBUG "value": [ 05:33:26 DEBUG 1712723606, 05:33:26 DEBUG "3.6296658392060612" 05:33:26 DEBUG ] 05:33:26 DEBUG } 05:33:26 DEBUG ] 05:33:26 DEBUG } 05:33:26 DEBUG } 05:33:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712723606" 05:33:26 INFO [http_cmd]: http status code OK 05:33:26 DEBUG --- status code --- 05:33:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:26 DEBUG --- http response --- 05:33:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:33:26 DEBUG ----- output ----- 05:33:26 DEBUG { 05:33:26 DEBUG "status": "success", 05:33:26 DEBUG "data": { 05:33:26 DEBUG "resultType": "vector", 05:33:26 DEBUG "result": [ 05:33:26 DEBUG { 05:33:26 DEBUG "metric": { 05:33:26 DEBUG "pod": "ds-idrepo-0" 05:33:26 DEBUG }, 05:33:26 DEBUG "value": [ 05:33:26 DEBUG 05:33:26 DEBUG 05:33:26 DEBUG 05:33:26 DEBUG [...] 05:33:26 DEBUG 05:33:26 DEBUG 05:33:26 DEBUG 05:33:26 DEBUG "pod": "ds-cts-2" 05:33:26 DEBUG }, 05:33:26 DEBUG "value": [ 05:33:26 DEBUG 1712723606, 05:33:26 DEBUG "0" 05:33:26 DEBUG ] 05:33:26 DEBUG } 05:33:26 DEBUG ] 05:33:26 DEBUG } 05:33:26 DEBUG } 05:33:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723606" 05:33:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723606" 05:33:26 INFO [http_cmd]: http status code OK 05:33:26 DEBUG --- status code --- 05:33:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:26 DEBUG --- http response --- 05:33:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:33:26 DEBUG ----- output ----- 05:33:26 DEBUG { 05:33:26 DEBUG "status": "success", 05:33:26 DEBUG "data": { 05:33:26 DEBUG "resultType": "vector", 05:33:26 DEBUG "result": [ 05:33:26 DEBUG { 05:33:26 DEBUG "metric": { 05:33:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:33:26 DEBUG }, 05:33:26 DEBUG "value": [ 05:33:26 DEBUG 05:33:26 DEBUG 05:33:26 DEBUG 05:33:26 DEBUG [...] 05:33:26 DEBUG 05:33:26 DEBUG 05:33:26 DEBUG 05:33:26 DEBUG "pod": "ds-idrepo-1" 05:33:26 DEBUG }, 05:33:26 DEBUG "value": [ 05:33:26 DEBUG 1712723606, 05:33:26 DEBUG "0" 05:33:26 DEBUG ] 05:33:26 DEBUG } 05:33:26 DEBUG ] 05:33:26 DEBUG } 05:33:26 DEBUG } 05:33:26 INFO [http_cmd]: http status code OK 05:33:26 DEBUG --- status code --- 05:33:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:26 DEBUG --- http response --- 05:33:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:33:26 DEBUG ----- output ----- 05:33:26 DEBUG { 05:33:26 DEBUG "status": "success", 05:33:26 DEBUG "data": { 05:33:26 DEBUG "resultType": "vector", 05:33:26 DEBUG "result": [ 05:33:26 DEBUG { 05:33:26 DEBUG "metric": { 05:33:26 DEBUG "pod": "ds-idrepo-2" 05:33:26 DEBUG }, 05:33:26 DEBUG "value": [ 05:33:26 DEBUG 05:33:26 DEBUG 05:33:26 DEBUG 05:33:26 DEBUG [...] 05:33:26 DEBUG 05:33:26 DEBUG 05:33:26 DEBUG 05:33:26 DEBUG "pod": "ds-cts-2" 05:33:26 DEBUG }, 05:33:26 DEBUG "value": [ 05:33:26 DEBUG 1712723606, 05:33:26 DEBUG "0" 05:33:26 DEBUG ] 05:33:26 DEBUG } 05:33:26 DEBUG ] 05:33:26 DEBUG } 05:33:26 DEBUG } 05:33:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723607" 05:33:27 INFO [http_cmd]: http status code OK 05:33:27 DEBUG --- status code --- 05:33:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:27 DEBUG --- http response --- 05:33:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:33:27 DEBUG ----- output ----- 05:33:27 DEBUG { 05:33:27 DEBUG "status": "success", 05:33:27 DEBUG "data": { 05:33:27 DEBUG "resultType": "vector", 05:33:27 DEBUG "result": [ 05:33:27 DEBUG { 05:33:27 DEBUG "metric": { 05:33:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:33:27 DEBUG }, 05:33:27 DEBUG "value": [ 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG [...] 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:33:27 DEBUG }, 05:33:27 DEBUG "value": [ 05:33:27 DEBUG 1712723607, 05:33:27 DEBUG "1263.3666666666666" 05:33:27 DEBUG ] 05:33:27 DEBUG } 05:33:27 DEBUG ] 05:33:27 DEBUG } 05:33:27 DEBUG } 05:33:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712723607" 05:33:27 INFO [http_cmd]: http status code OK 05:33:27 DEBUG --- status code --- 05:33:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:27 DEBUG --- http response --- 05:33:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:33:27 DEBUG ----- output ----- 05:33:27 DEBUG { 05:33:27 DEBUG "status": "success", 05:33:27 DEBUG "data": { 05:33:27 DEBUG "resultType": "vector", 05:33:27 DEBUG "result": [ 05:33:27 DEBUG { 05:33:27 DEBUG "metric": { 05:33:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:33:27 DEBUG }, 05:33:27 DEBUG "value": [ 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG [...] 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:33:27 DEBUG }, 05:33:27 DEBUG "value": [ 05:33:27 DEBUG 1712723607, 05:33:27 DEBUG "0.042" 05:33:27 DEBUG ] 05:33:27 DEBUG } 05:33:27 DEBUG ] 05:33:27 DEBUG } 05:33:27 DEBUG } 05:33:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723607" 05:33:27 INFO [http_cmd]: http status code OK 05:33:27 DEBUG --- status code --- 05:33:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:27 DEBUG --- http response --- 05:33:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:33:27 DEBUG ----- output ----- 05:33:27 DEBUG { 05:33:27 DEBUG "status": "success", 05:33:27 DEBUG "data": { 05:33:27 DEBUG "resultType": "vector", 05:33:27 DEBUG "result": [ 05:33:27 DEBUG { 05:33:27 DEBUG "metric": { 05:33:27 DEBUG "pod": "ds-idrepo-0" 05:33:27 DEBUG }, 05:33:27 DEBUG "value": [ 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG [...] 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG "pod": "ds-idrepo-1" 05:33:27 DEBUG }, 05:33:27 DEBUG "value": [ 05:33:27 DEBUG 1712723607, 05:33:27 DEBUG "0" 05:33:27 DEBUG ] 05:33:27 DEBUG } 05:33:27 DEBUG ] 05:33:27 DEBUG } 05:33:27 DEBUG } 05:33:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723607" 05:33:27 INFO [http_cmd]: http status code OK 05:33:27 DEBUG --- status code --- 05:33:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:27 DEBUG --- http response --- 05:33:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:33:27 DEBUG ----- output ----- 05:33:27 DEBUG { 05:33:27 DEBUG "status": "success", 05:33:27 DEBUG "data": { 05:33:27 DEBUG "resultType": "vector", 05:33:27 DEBUG "result": [ 05:33:27 DEBUG { 05:33:27 DEBUG "metric": { 05:33:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:33:27 DEBUG }, 05:33:27 DEBUG "value": [ 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG [...] 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:33:27 DEBUG }, 05:33:27 DEBUG "value": [ 05:33:27 DEBUG 1712723607, 05:33:27 DEBUG "1263.9666666666667" 05:33:27 DEBUG ] 05:33:27 DEBUG } 05:33:27 DEBUG ] 05:33:27 DEBUG } 05:33:27 DEBUG } 05:33:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723607" 05:33:27 INFO [http_cmd]: http status code OK 05:33:27 DEBUG --- status code --- 05:33:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:27 DEBUG --- http response --- 05:33:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:33:27 DEBUG ----- output ----- 05:33:27 DEBUG { 05:33:27 DEBUG "status": "success", 05:33:27 DEBUG "data": { 05:33:27 DEBUG "resultType": "vector", 05:33:27 DEBUG "result": [ 05:33:27 DEBUG { 05:33:27 DEBUG "metric": { 05:33:27 DEBUG "pod": "ds-idrepo-0" 05:33:27 DEBUG }, 05:33:27 DEBUG "value": [ 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG [...] 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG "pod": "ds-idrepo-1" 05:33:27 DEBUG }, 05:33:27 DEBUG "value": [ 05:33:27 DEBUG 1712723607, 05:33:27 DEBUG "0" 05:33:27 DEBUG ] 05:33:27 DEBUG } 05:33:27 DEBUG ] 05:33:27 DEBUG } 05:33:27 DEBUG } 05:33:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712723607" 05:33:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723607" 05:33:27 INFO [http_cmd]: http status code OK 05:33:27 DEBUG --- status code --- 05:33:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:27 DEBUG --- http response --- 05:33:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:33:27 INFO [http_cmd]: http status code OK 05:33:27 DEBUG ----- output ----- 05:33:27 DEBUG --- status code --- 05:33:27 DEBUG { 05:33:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:27 DEBUG "status": "success", 05:33:27 DEBUG --- http response --- 05:33:27 DEBUG "data": { 05:33:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:33:27 DEBUG "resultType": "vector", 05:33:27 DEBUG ----- output ----- 05:33:27 DEBUG "result": [ 05:33:27 DEBUG { 05:33:27 DEBUG { 05:33:27 DEBUG "status": "success", 05:33:27 DEBUG "metric": { 05:33:27 DEBUG "data": { 05:33:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:33:27 DEBUG "resultType": "vector", 05:33:27 DEBUG }, 05:33:27 DEBUG "result": [ 05:33:27 DEBUG "value": [ 05:33:27 DEBUG { 05:33:27 DEBUG 05:33:27 DEBUG "metric": { 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG [...] 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:33:27 DEBUG }, 05:33:27 DEBUG "value": [ 05:33:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:33:27 DEBUG }, 05:33:27 DEBUG 1712723607, 05:33:27 DEBUG "value": [ 05:33:27 DEBUG "0" 05:33:27 DEBUG 05:33:27 DEBUG ] 05:33:27 DEBUG 05:33:27 DEBUG } 05:33:27 DEBUG 05:33:27 DEBUG ] 05:33:27 DEBUG [...] 05:33:27 DEBUG } 05:33:27 DEBUG 05:33:27 DEBUG } 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:33:27 DEBUG }, 05:33:27 DEBUG "value": [ 05:33:27 DEBUG 1712723607, 05:33:27 DEBUG "2487386112" 05:33:27 DEBUG ] 05:33:27 DEBUG } 05:33:27 DEBUG ] 05:33:27 DEBUG } 05:33:27 DEBUG } 05:33:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723607" 05:33:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723607" 05:33:27 INFO [http_cmd]: http status code OK 05:33:27 DEBUG --- status code --- 05:33:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:27 DEBUG --- http response --- 05:33:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:33:27 DEBUG ----- output ----- 05:33:27 DEBUG { 05:33:27 DEBUG "status": "success", 05:33:27 DEBUG "data": { 05:33:27 DEBUG "resultType": "vector", 05:33:27 DEBUG "result": [ 05:33:27 DEBUG { 05:33:27 DEBUG "metric": { 05:33:27 DEBUG "instance": "10.142.0.25:9100" 05:33:27 DEBUG }, 05:33:27 DEBUG "value": [ 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG [...] 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG "instance": "10.142.0.77:9100" 05:33:27 DEBUG }, 05:33:27 DEBUG "value": [ 05:33:27 INFO [http_cmd]: http status code OK 05:33:27 DEBUG 1712723607, 05:33:27 DEBUG --- status code --- 05:33:27 DEBUG "37657261.42614261" 05:33:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:27 DEBUG ] 05:33:27 DEBUG --- http response --- 05:33:27 DEBUG } 05:33:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:33:27 DEBUG ] 05:33:27 DEBUG ----- output ----- 05:33:27 DEBUG } 05:33:27 DEBUG { 05:33:27 DEBUG } 05:33:27 DEBUG "status": "success", 05:33:27 DEBUG "data": { 05:33:27 DEBUG "resultType": "vector", 05:33:27 DEBUG "result": [ 05:33:27 DEBUG { 05:33:27 DEBUG "metric": { 05:33:27 DEBUG "instance": "10.142.0.25:9100" 05:33:27 DEBUG }, 05:33:27 DEBUG "value": [ 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG [...] 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG 05:33:27 DEBUG "instance": "10.142.0.77:9100" 05:33:27 DEBUG }, 05:33:27 DEBUG "value": [ 05:33:27 DEBUG 1712723607, 05:33:27 DEBUG "0.05300530053005223" 05:33:27 DEBUG ] 05:33:27 DEBUG } 05:33:27 DEBUG ] 05:33:27 DEBUG } 05:33:27 DEBUG } 05:33:42 INFO 05:33:42 INFO [loop_until]: kubectl --namespace=xlou top pods 05:33:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:42 INFO [loop_until]: OK (rc = 0) 05:33:42 DEBUG --- stdout --- 05:33:42 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 8748m 8808Mi am-654c4fcbcf-9scrj 8525m 8423Mi am-654c4fcbcf-v5qb5 7260m 8401Mi ds-cts-0 7182m 13319Mi ds-cts-1 7202m 13320Mi ds-cts-2 6885m 13296Mi ds-idrepo-0 9622m 5621Mi ds-idrepo-1 313m 5669Mi ds-idrepo-2 333m 5698Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 5m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3709m 1666Mi 05:33:42 DEBUG --- stderr --- 05:33:42 DEBUG 05:33:49 INFO 05:33:49 INFO [loop_until]: kubectl --namespace=xlou top node 05:33:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:49 INFO [loop_until]: OK (rc = 0) 05:33:49 DEBUG --- stdout --- 05:33:49 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 94m 0% 2303Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8150m 51% 9281Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 82m 0% 3257Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10865m 68% 6519Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9313m 58% 9311Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9929m 62% 9649Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 73m 0% 856Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9794m 61% 6281Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 7452m 46% 13942Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 7019m 44% 13939Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 390m 2% 6365Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7451m 46% 13942Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 427m 2% 6331Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3734m 23% 5343Mi 9% 05:33:49 DEBUG --- stderr --- 05:33:49 DEBUG 05:34:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712723664" 05:34:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723664" 05:34:24 ERROR [http_cmd]: ERROR 05:34:24 ERROR --- status code --- 05:34:24 INFO [http_cmd]: http status code OK 05:34:24 DEBUG --- status code --- 05:34:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:24 DEBUG --- http response --- 05:34:24 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:34:24 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:34:24 ERROR --- http response --- 05:34:24 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:34:24 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:34:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723664" 05:34:24 INFO [http_cmd]: http status code OK 05:34:24 DEBUG --- status code --- 05:34:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:24 DEBUG --- http response --- 05:34:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:34:24 DEBUG ----- output ----- 05:34:24 DEBUG { 05:34:24 DEBUG "status": "success", 05:34:24 DEBUG "data": { 05:34:24 DEBUG "resultType": "vector", 05:34:24 DEBUG "result": [ 05:34:24 DEBUG { 05:34:24 DEBUG "metric": { 05:34:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:34:24 DEBUG }, 05:34:24 DEBUG "value": [ 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG [...] 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:34:24 DEBUG }, 05:34:24 DEBUG "value": [ 05:34:24 DEBUG 1712723664, 05:34:24 DEBUG "0" 05:34:24 DEBUG ] 05:34:24 DEBUG } 05:34:24 DEBUG ] 05:34:24 DEBUG } 05:34:24 DEBUG } 05:34:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723664" 05:34:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723664" 05:34:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723664" 05:34:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723664" 05:34:24 INFO [http_cmd]: http status code OK 05:34:24 DEBUG --- status code --- 05:34:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:24 DEBUG --- http response --- 05:34:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:34:24 DEBUG ----- output ----- 05:34:24 DEBUG { 05:34:24 DEBUG "status": "success", 05:34:24 DEBUG "data": { 05:34:24 DEBUG "resultType": "vector", 05:34:24 DEBUG "result": [ 05:34:24 DEBUG { 05:34:24 DEBUG "metric": { 05:34:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:34:24 DEBUG }, 05:34:24 DEBUG "value": [ 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG [...] 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:34:24 DEBUG }, 05:34:24 DEBUG "value": [ 05:34:24 DEBUG 1712723664, 05:34:24 DEBUG "1265.3" 05:34:24 DEBUG ] 05:34:24 DEBUG } 05:34:24 DEBUG ] 05:34:24 DEBUG } 05:34:24 DEBUG } 05:34:24 INFO [http_cmd]: http status code OK 05:34:24 INFO [http_cmd]: http status code OK 05:34:24 INFO [http_cmd]: http status code OK 05:34:24 DEBUG --- status code --- 05:34:24 DEBUG --- status code --- 05:34:24 DEBUG --- status code --- 05:34:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:24 DEBUG --- http response --- 05:34:24 DEBUG --- http response --- 05:34:24 DEBUG --- http response --- 05:34:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:34:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:34:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:34:24 DEBUG ----- output ----- 05:34:24 DEBUG ----- output ----- 05:34:24 DEBUG ----- output ----- 05:34:24 DEBUG { 05:34:24 DEBUG { 05:34:24 DEBUG "status": "success", 05:34:24 DEBUG "status": "success", 05:34:24 DEBUG "data": { 05:34:24 DEBUG "data": { 05:34:24 DEBUG "resultType": "vector", 05:34:24 DEBUG "resultType": "vector", 05:34:24 DEBUG { 05:34:24 DEBUG "result": [ 05:34:24 DEBUG { 05:34:24 DEBUG "status": "success", 05:34:24 DEBUG "data": { 05:34:24 DEBUG "metric": { 05:34:24 DEBUG "pod": "ds-idrepo-0" 05:34:24 DEBUG "resultType": "vector", 05:34:24 DEBUG "result": [ 05:34:24 DEBUG }, 05:34:24 DEBUG "result": [ 05:34:24 DEBUG { 05:34:24 DEBUG "value": [ 05:34:24 DEBUG { 05:34:24 DEBUG "metric": { 05:34:24 DEBUG 05:34:24 DEBUG "metric": { 05:34:24 DEBUG "pod": "ds-idrepo-0" 05:34:24 DEBUG 05:34:24 DEBUG "instance": "10.142.0.25:9100" 05:34:24 DEBUG }, 05:34:24 DEBUG 05:34:24 DEBUG }, 05:34:24 DEBUG "value": [ 05:34:24 DEBUG [...] 05:34:24 DEBUG "value": [ 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG [...] 05:34:24 DEBUG "pod": "ds-cts-2" 05:34:24 DEBUG [...] 05:34:24 DEBUG 05:34:24 DEBUG }, 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG "value": [ 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG 1712723664, 05:34:24 DEBUG 05:34:24 DEBUG "instance": "10.142.0.77:9100" 05:34:24 DEBUG "29.966666666666665" 05:34:24 DEBUG "pod": "ds-cts-2" 05:34:24 DEBUG }, 05:34:24 DEBUG ] 05:34:24 DEBUG }, 05:34:24 DEBUG "value": [ 05:34:24 DEBUG } 05:34:24 DEBUG "value": [ 05:34:24 DEBUG 1712723664, 05:34:24 DEBUG ] 05:34:24 DEBUG 1712723664, 05:34:24 DEBUG "295991413760" 05:34:24 DEBUG } 05:34:24 DEBUG "0" 05:34:24 DEBUG ] 05:34:24 DEBUG } 05:34:24 DEBUG ] 05:34:24 DEBUG } 05:34:24 DEBUG } 05:34:24 DEBUG ] 05:34:24 DEBUG ] 05:34:24 DEBUG } 05:34:24 DEBUG } 05:34:24 DEBUG } 05:34:24 DEBUG } 05:34:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723664" 05:34:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723664" 05:34:24 INFO [http_cmd]: http status code OK 05:34:24 INFO [http_cmd]: http status code OK 05:34:24 DEBUG --- status code --- 05:34:24 DEBUG --- status code --- 05:34:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:24 DEBUG --- http response --- 05:34:24 DEBUG --- http response --- 05:34:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:34:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:34:24 DEBUG ----- output ----- 05:34:24 DEBUG ----- output ----- 05:34:24 DEBUG { 05:34:24 DEBUG { 05:34:24 DEBUG "status": "success", 05:34:24 DEBUG "status": "success", 05:34:24 DEBUG "data": { 05:34:24 DEBUG "data": { 05:34:24 DEBUG "resultType": "vector", 05:34:24 DEBUG "resultType": "vector", 05:34:24 DEBUG "result": [ 05:34:24 DEBUG "result": [ 05:34:24 DEBUG { 05:34:24 DEBUG { 05:34:24 DEBUG "metric": { 05:34:24 DEBUG "metric": { 05:34:24 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:34:24 DEBUG "instance": "10.142.0.25:9100" 05:34:24 DEBUG }, 05:34:24 DEBUG }, 05:34:24 DEBUG "value": [ 05:34:24 DEBUG "value": [ 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG [...] 05:34:24 DEBUG [...] 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG "pod": "ds-idrepo-2" 05:34:24 DEBUG "instance": "10.142.0.77:9100" 05:34:24 DEBUG }, 05:34:24 DEBUG }, 05:34:24 DEBUG "value": [ 05:34:24 DEBUG "value": [ 05:34:24 DEBUG 1712723664, 05:34:24 DEBUG 1712723664, 05:34:24 DEBUG "0" 05:34:24 DEBUG "37580512.4" 05:34:24 DEBUG ] 05:34:24 DEBUG ] 05:34:24 DEBUG } 05:34:24 DEBUG } 05:34:24 DEBUG ] 05:34:24 DEBUG ] 05:34:24 DEBUG } 05:34:24 DEBUG } 05:34:24 DEBUG } 05:34:24 DEBUG } 05:34:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723664" 05:34:24 INFO [http_cmd]: http status code OK 05:34:24 DEBUG --- status code --- 05:34:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:24 DEBUG --- http response --- 05:34:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:34:24 DEBUG ----- output ----- 05:34:24 DEBUG { 05:34:24 DEBUG "status": "success", 05:34:24 DEBUG "data": { 05:34:24 DEBUG "resultType": "vector", 05:34:24 DEBUG "result": [ 05:34:24 DEBUG { 05:34:24 DEBUG "metric": { 05:34:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:34:24 DEBUG }, 05:34:24 DEBUG "value": [ 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG [...] 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG 05:34:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:34:24 DEBUG }, 05:34:24 DEBUG "value": [ 05:34:24 DEBUG 1712723664, 05:34:24 DEBUG "227570.5" 05:34:24 DEBUG ] 05:34:24 DEBUG } 05:34:24 DEBUG ] 05:34:24 DEBUG } 05:34:24 DEBUG } 05:34:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723665" 05:34:25 INFO [http_cmd]: http status code OK 05:34:25 DEBUG --- status code --- 05:34:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:25 DEBUG --- http response --- 05:34:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:34:25 DEBUG ----- output ----- 05:34:25 DEBUG { 05:34:25 DEBUG "status": "success", 05:34:25 DEBUG "data": { 05:34:25 DEBUG "resultType": "vector", 05:34:25 DEBUG "result": [ 05:34:25 DEBUG { 05:34:25 DEBUG "metric": { 05:34:25 DEBUG "pod": "ds-idrepo-0" 05:34:25 DEBUG }, 05:34:25 DEBUG "value": [ 05:34:25 DEBUG 05:34:25 DEBUG 05:34:25 DEBUG 05:34:25 DEBUG [...] 05:34:25 DEBUG 05:34:25 DEBUG 05:34:25 DEBUG 05:34:25 DEBUG "pod": "ds-idrepo-1" 05:34:25 DEBUG }, 05:34:25 DEBUG "value": [ 05:34:25 DEBUG 1712723665, 05:34:25 DEBUG "0" 05:34:25 DEBUG ] 05:34:25 DEBUG } 05:34:25 DEBUG ] 05:34:25 DEBUG } 05:34:25 DEBUG } 05:34:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723665" 05:34:25 INFO [http_cmd]: http status code OK 05:34:25 DEBUG --- status code --- 05:34:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:25 DEBUG --- http response --- 05:34:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:34:25 DEBUG ----- output ----- 05:34:25 DEBUG { 05:34:25 DEBUG "status": "success", 05:34:25 DEBUG "data": { 05:34:25 DEBUG "resultType": "vector", 05:34:25 DEBUG "result": [ 05:34:25 DEBUG { 05:34:25 DEBUG "metric": { 05:34:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:34:25 DEBUG }, 05:34:25 DEBUG "value": [ 05:34:25 DEBUG 05:34:25 DEBUG 05:34:25 DEBUG 05:34:25 DEBUG [...] 05:34:25 DEBUG 05:34:25 DEBUG 05:34:25 DEBUG 05:34:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:34:25 DEBUG }, 05:34:25 DEBUG "value": [ 05:34:25 DEBUG 1712723665, 05:34:25 DEBUG "1263.8" 05:34:25 DEBUG ] 05:34:25 DEBUG } 05:34:25 DEBUG ] 05:34:25 DEBUG } 05:34:25 DEBUG } 05:34:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723665" 05:34:25 INFO [http_cmd]: http status code OK 05:34:25 DEBUG --- status code --- 05:34:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:25 DEBUG --- http response --- 05:34:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:34:25 DEBUG ----- output ----- 05:34:25 DEBUG { 05:34:25 DEBUG "status": "success", 05:34:25 DEBUG "data": { 05:34:25 DEBUG "resultType": "vector", 05:34:25 DEBUG "result": [ 05:34:25 DEBUG { 05:34:25 DEBUG "metric": { 05:34:25 DEBUG "instance": "10.142.0.25:9100" 05:34:25 DEBUG }, 05:34:25 DEBUG "value": [ 05:34:25 DEBUG 05:34:25 DEBUG 05:34:25 DEBUG 05:34:25 DEBUG [...] 05:34:25 DEBUG 05:34:25 DEBUG 05:34:25 DEBUG 05:34:25 DEBUG "instance": "10.142.0.77:9100" 05:34:25 DEBUG }, 05:34:25 DEBUG "value": [ 05:34:25 DEBUG 1712723665, 05:34:25 DEBUG "0.14813333333332843" 05:34:25 DEBUG ] 05:34:25 DEBUG } 05:34:25 DEBUG ] 05:34:25 DEBUG } 05:34:25 DEBUG } 05:34:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723666" 05:34:26 INFO [http_cmd]: http status code OK 05:34:26 DEBUG --- status code --- 05:34:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:26 DEBUG --- http response --- 05:34:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:34:26 DEBUG ----- output ----- 05:34:26 DEBUG { 05:34:26 DEBUG "status": "success", 05:34:26 DEBUG "data": { 05:34:26 DEBUG "resultType": "vector", 05:34:26 DEBUG "result": [ 05:34:26 DEBUG { 05:34:26 DEBUG "metric": { 05:34:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:34:26 DEBUG }, 05:34:26 DEBUG "value": [ 05:34:26 DEBUG 05:34:26 DEBUG 05:34:26 DEBUG 05:34:26 DEBUG [...] 05:34:26 DEBUG 05:34:26 DEBUG 05:34:26 DEBUG 05:34:26 DEBUG "pod": "ds-idrepo-2" 05:34:26 DEBUG }, 05:34:26 DEBUG "value": [ 05:34:26 DEBUG 1712723666, 05:34:26 DEBUG "1779.3703002504797" 05:34:26 DEBUG ] 05:34:26 DEBUG } 05:34:26 DEBUG ] 05:34:26 DEBUG } 05:34:26 DEBUG } 05:34:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712723666" 05:34:26 INFO [http_cmd]: http status code OK 05:34:26 DEBUG --- status code --- 05:34:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:26 DEBUG --- http response --- 05:34:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:34:26 DEBUG ----- output ----- 05:34:26 DEBUG { 05:34:26 DEBUG "status": "success", 05:34:26 DEBUG "data": { 05:34:26 DEBUG "resultType": "vector", 05:34:26 DEBUG "result": [ 05:34:26 DEBUG { 05:34:26 DEBUG "metric": { 05:34:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:34:26 DEBUG }, 05:34:26 DEBUG "value": [ 05:34:26 DEBUG 05:34:26 DEBUG 05:34:26 DEBUG 05:34:26 DEBUG [...] 05:34:26 DEBUG 05:34:26 DEBUG 05:34:26 DEBUG 05:34:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:34:26 DEBUG }, 05:34:26 DEBUG "value": [ 05:34:26 DEBUG 1712723666, 05:34:26 DEBUG "3.6850590521194913" 05:34:26 DEBUG ] 05:34:26 DEBUG } 05:34:26 DEBUG ] 05:34:26 DEBUG } 05:34:26 DEBUG } 05:34:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712723666" 05:34:26 INFO [http_cmd]: http status code OK 05:34:26 DEBUG --- status code --- 05:34:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:26 DEBUG --- http response --- 05:34:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:34:26 DEBUG ----- output ----- 05:34:26 DEBUG { 05:34:26 DEBUG "status": "success", 05:34:26 DEBUG "data": { 05:34:26 DEBUG "resultType": "vector", 05:34:26 DEBUG "result": [ 05:34:26 DEBUG { 05:34:26 DEBUG "metric": { 05:34:26 DEBUG "pod": "ds-idrepo-0" 05:34:26 DEBUG }, 05:34:26 DEBUG "value": [ 05:34:26 DEBUG 05:34:26 DEBUG 05:34:26 DEBUG 05:34:26 DEBUG [...] 05:34:26 DEBUG 05:34:26 DEBUG 05:34:26 DEBUG 05:34:26 DEBUG "pod": "ds-cts-2" 05:34:26 DEBUG }, 05:34:26 DEBUG "value": [ 05:34:26 DEBUG 1712723666, 05:34:26 DEBUG "0" 05:34:26 DEBUG ] 05:34:26 DEBUG } 05:34:26 DEBUG ] 05:34:26 DEBUG } 05:34:26 DEBUG } 05:34:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723666" 05:34:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723666" 05:34:26 INFO [http_cmd]: http status code OK 05:34:26 DEBUG --- status code --- 05:34:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:26 DEBUG --- http response --- 05:34:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:34:26 DEBUG ----- output ----- 05:34:26 DEBUG { 05:34:26 DEBUG "status": "success", 05:34:26 DEBUG "data": { 05:34:26 DEBUG "resultType": "vector", 05:34:26 DEBUG "result": [ 05:34:26 DEBUG { 05:34:26 DEBUG "metric": { 05:34:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:34:26 DEBUG }, 05:34:26 DEBUG "value": [ 05:34:26 DEBUG 05:34:26 DEBUG 05:34:26 DEBUG 05:34:26 DEBUG [...] 05:34:26 DEBUG 05:34:26 DEBUG 05:34:26 DEBUG 05:34:26 DEBUG "pod": "ds-idrepo-2" 05:34:26 DEBUG }, 05:34:26 DEBUG "value": [ 05:34:26 DEBUG 1712723666, 05:34:26 DEBUG "0.22620455145255985" 05:34:26 DEBUG ] 05:34:26 DEBUG } 05:34:26 DEBUG ] 05:34:26 DEBUG } 05:34:26 DEBUG } 05:34:26 INFO [http_cmd]: http status code OK 05:34:26 DEBUG --- status code --- 05:34:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:26 DEBUG --- http response --- 05:34:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:34:26 DEBUG ----- output ----- 05:34:26 DEBUG { 05:34:26 DEBUG "status": "success", 05:34:26 DEBUG "data": { 05:34:26 DEBUG "resultType": "vector", 05:34:26 DEBUG "result": [ 05:34:26 DEBUG { 05:34:26 DEBUG "metric": { 05:34:26 DEBUG "pod": "ds-idrepo-2" 05:34:26 DEBUG }, 05:34:26 DEBUG "value": [ 05:34:26 DEBUG 05:34:26 DEBUG 05:34:26 DEBUG 05:34:26 DEBUG [...] 05:34:26 DEBUG 05:34:26 DEBUG 05:34:26 DEBUG 05:34:26 DEBUG "pod": "ds-cts-2" 05:34:26 DEBUG }, 05:34:26 DEBUG "value": [ 05:34:26 DEBUG 1712723666, 05:34:26 DEBUG "0.000041326666666666665" 05:34:26 DEBUG ] 05:34:26 DEBUG } 05:34:26 DEBUG ] 05:34:26 DEBUG } 05:34:26 DEBUG } 05:34:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723667" 05:34:27 INFO [http_cmd]: http status code OK 05:34:27 DEBUG --- status code --- 05:34:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:27 DEBUG --- http response --- 05:34:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:34:27 DEBUG ----- output ----- 05:34:27 DEBUG { 05:34:27 DEBUG "status": "success", 05:34:27 DEBUG "data": { 05:34:27 DEBUG "resultType": "vector", 05:34:27 DEBUG "result": [ 05:34:27 DEBUG { 05:34:27 DEBUG "metric": { 05:34:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:34:27 DEBUG }, 05:34:27 DEBUG "value": [ 05:34:27 DEBUG 05:34:27 DEBUG 05:34:27 DEBUG 05:34:27 DEBUG [...] 05:34:27 DEBUG 05:34:27 DEBUG 05:34:27 DEBUG 05:34:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:34:27 DEBUG }, 05:34:27 DEBUG "value": [ 05:34:27 DEBUG 1712723667, 05:34:27 DEBUG "1263.3666666666666" 05:34:27 DEBUG ] 05:34:27 DEBUG } 05:34:27 DEBUG ] 05:34:27 DEBUG } 05:34:27 DEBUG } 05:34:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712723667" 05:34:27 INFO [http_cmd]: http status code OK 05:34:27 DEBUG --- status code --- 05:34:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:27 DEBUG --- http response --- 05:34:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:34:27 DEBUG ----- output ----- 05:34:27 DEBUG { 05:34:27 DEBUG "status": "success", 05:34:27 DEBUG "data": { 05:34:27 DEBUG "resultType": "vector", 05:34:27 DEBUG "result": [ 05:34:27 DEBUG { 05:34:27 DEBUG "metric": { 05:34:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:34:27 DEBUG }, 05:34:27 DEBUG "value": [ 05:34:27 DEBUG 05:34:27 DEBUG 05:34:27 DEBUG 05:34:27 DEBUG [...] 05:34:27 DEBUG 05:34:27 DEBUG 05:34:27 DEBUG 05:34:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:34:27 DEBUG }, 05:34:27 DEBUG "value": [ 05:34:27 DEBUG 1712723667, 05:34:27 DEBUG "0.042" 05:34:27 DEBUG ] 05:34:27 DEBUG } 05:34:27 DEBUG ] 05:34:27 DEBUG } 05:34:27 DEBUG } 05:34:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723667" 05:34:27 INFO [http_cmd]: http status code OK 05:34:27 DEBUG --- status code --- 05:34:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:27 DEBUG --- http response --- 05:34:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:34:27 DEBUG ----- output ----- 05:34:27 DEBUG { 05:34:27 DEBUG "status": "success", 05:34:27 DEBUG "data": { 05:34:27 DEBUG "resultType": "vector", 05:34:27 DEBUG "result": [ 05:34:27 DEBUG { 05:34:27 DEBUG "metric": { 05:34:27 DEBUG "pod": "ds-idrepo-0" 05:34:27 DEBUG }, 05:34:27 DEBUG "value": [ 05:34:27 DEBUG 05:34:27 DEBUG 05:34:27 DEBUG 05:34:27 DEBUG [...] 05:34:27 DEBUG 05:34:27 DEBUG 05:34:27 DEBUG 05:34:27 DEBUG "pod": "ds-idrepo-1" 05:34:27 DEBUG }, 05:34:27 DEBUG "value": [ 05:34:27 DEBUG 1712723667, 05:34:27 DEBUG "0" 05:34:27 DEBUG ] 05:34:27 DEBUG } 05:34:27 DEBUG ] 05:34:27 DEBUG } 05:34:27 DEBUG } 05:34:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723667" 05:34:27 INFO [http_cmd]: http status code OK 05:34:27 DEBUG --- status code --- 05:34:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:27 DEBUG --- http response --- 05:34:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:34:27 DEBUG ----- output ----- 05:34:27 DEBUG { 05:34:27 DEBUG "status": "success", 05:34:27 DEBUG "data": { 05:34:27 DEBUG "resultType": "vector", 05:34:27 DEBUG "result": [ 05:34:27 DEBUG { 05:34:27 DEBUG "metric": { 05:34:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:34:27 DEBUG }, 05:34:27 DEBUG "value": [ 05:34:27 DEBUG 05:34:27 DEBUG 05:34:27 DEBUG 05:34:27 DEBUG [...] 05:34:27 DEBUG 05:34:27 DEBUG 05:34:27 DEBUG 05:34:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:34:27 DEBUG }, 05:34:27 DEBUG "value": [ 05:34:27 DEBUG 1712723667, 05:34:27 DEBUG "1266.5333333333333" 05:34:27 DEBUG ] 05:34:27 DEBUG } 05:34:27 DEBUG ] 05:34:27 DEBUG } 05:34:27 DEBUG } 05:34:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723667" 05:34:27 INFO [http_cmd]: http status code OK 05:34:27 DEBUG --- status code --- 05:34:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:27 DEBUG --- http response --- 05:34:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:34:27 DEBUG ----- output ----- 05:34:27 DEBUG { 05:34:27 DEBUG "status": "success", 05:34:27 DEBUG "data": { 05:34:27 DEBUG "resultType": "vector", 05:34:27 DEBUG "result": [ 05:34:27 DEBUG { 05:34:27 DEBUG "metric": { 05:34:27 DEBUG "pod": "ds-idrepo-0" 05:34:27 DEBUG }, 05:34:27 DEBUG "value": [ 05:34:27 DEBUG 05:34:27 DEBUG 05:34:27 DEBUG 05:34:27 DEBUG [...] 05:34:27 DEBUG 05:34:27 DEBUG 05:34:27 DEBUG 05:34:27 DEBUG "pod": "ds-idrepo-1" 05:34:27 DEBUG }, 05:34:27 DEBUG "value": [ 05:34:27 DEBUG 1712723667, 05:34:27 DEBUG "0" 05:34:27 DEBUG ] 05:34:27 DEBUG } 05:34:27 DEBUG ] 05:34:27 DEBUG } 05:34:27 DEBUG } 05:34:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723667" 05:34:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712723667" 05:34:27 INFO [http_cmd]: http status code OK 05:34:27 DEBUG --- status code --- 05:34:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:27 DEBUG --- http response --- 05:34:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:34:27 DEBUG ----- output ----- 05:34:27 DEBUG { 05:34:27 DEBUG "status": "success", 05:34:27 INFO [http_cmd]: http status code OK 05:34:27 DEBUG "data": { 05:34:27 DEBUG --- status code --- 05:34:27 DEBUG "resultType": "vector", 05:34:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:27 DEBUG "result": [ 05:34:27 DEBUG --- http response --- 05:34:27 DEBUG { 05:34:27 DEBUG "metric": { 05:34:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:34:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:34:27 DEBUG }, 05:34:27 DEBUG ----- output ----- 05:34:27 DEBUG "value": [ 05:34:27 DEBUG { 05:34:27 DEBUG 05:34:27 DEBUG "status": "success", 05:34:27 DEBUG 05:34:27 DEBUG "data": { 05:34:27 DEBUG 05:34:27 DEBUG "resultType": "vector", 05:34:27 DEBUG [...] 05:34:27 DEBUG "result": [ 05:34:27 DEBUG 05:34:27 DEBUG { 05:34:27 DEBUG 05:34:27 DEBUG "metric": { 05:34:27 DEBUG 05:34:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:34:27 DEBUG "pod": "ds-idrepo-2" 05:34:27 DEBUG }, 05:34:27 DEBUG }, 05:34:27 DEBUG "value": [ 05:34:27 DEBUG "value": [ 05:34:27 DEBUG 05:34:27 DEBUG 1712723667, 05:34:27 DEBUG 05:34:27 DEBUG "2049.7121629094695" 05:34:27 DEBUG 05:34:27 DEBUG ] 05:34:27 DEBUG [...] 05:34:27 DEBUG } 05:34:27 DEBUG 05:34:27 DEBUG ] 05:34:27 DEBUG 05:34:27 DEBUG } 05:34:27 DEBUG 05:34:27 DEBUG } 05:34:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:34:27 DEBUG }, 05:34:27 DEBUG "value": [ 05:34:27 DEBUG 1712723667, 05:34:27 DEBUG "2487705600" 05:34:27 DEBUG ] 05:34:27 DEBUG } 05:34:27 DEBUG ] 05:34:27 DEBUG } 05:34:27 DEBUG } 05:34:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723667" 05:34:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723667" 05:34:27 INFO [http_cmd]: http status code OK 05:34:27 DEBUG --- status code --- 05:34:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:27 DEBUG --- http response --- 05:34:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:34:27 DEBUG ----- output ----- 05:34:27 DEBUG { 05:34:27 DEBUG "status": "success", 05:34:27 DEBUG "data": { 05:34:27 DEBUG "resultType": "vector", 05:34:27 DEBUG "result": [ 05:34:27 DEBUG { 05:34:27 DEBUG "metric": { 05:34:27 DEBUG "instance": "10.142.0.25:9100" 05:34:27 DEBUG }, 05:34:27 DEBUG "value": [ 05:34:27 DEBUG 05:34:27 DEBUG 05:34:27 DEBUG 05:34:27 INFO [http_cmd]: http status code OK 05:34:27 DEBUG [...] 05:34:27 DEBUG --- status code --- 05:34:27 DEBUG 05:34:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:27 DEBUG 05:34:27 DEBUG --- http response --- 05:34:27 DEBUG 05:34:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:34:27 DEBUG "instance": "10.142.0.77:9100" 05:34:27 DEBUG ----- output ----- 05:34:27 DEBUG }, 05:34:27 DEBUG { 05:34:27 DEBUG "value": [ 05:34:27 DEBUG "status": "success", 05:34:27 DEBUG 1712723667, 05:34:27 DEBUG "data": { 05:34:27 DEBUG "37581930.66666667" 05:34:27 DEBUG "resultType": "vector", 05:34:27 DEBUG "result": [ 05:34:27 DEBUG ] 05:34:27 DEBUG { 05:34:27 DEBUG } 05:34:27 DEBUG "metric": { 05:34:27 DEBUG ] 05:34:27 DEBUG "instance": "10.142.0.25:9100" 05:34:27 DEBUG }, 05:34:27 DEBUG } 05:34:27 DEBUG } 05:34:27 DEBUG "value": [ 05:34:27 DEBUG 05:34:27 DEBUG 05:34:27 DEBUG 05:34:27 DEBUG [...] 05:34:27 DEBUG 05:34:27 DEBUG 05:34:27 DEBUG 05:34:27 DEBUG "instance": "10.142.0.77:9100" 05:34:27 DEBUG }, 05:34:27 DEBUG "value": [ 05:34:27 DEBUG 1712723667, 05:34:27 DEBUG "0.0563333333333335" 05:34:27 DEBUG ] 05:34:27 DEBUG } 05:34:27 DEBUG ] 05:34:27 DEBUG } 05:34:27 DEBUG } 05:34:42 INFO 05:34:42 INFO [loop_until]: kubectl --namespace=xlou top pods 05:34:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:34:43 INFO [loop_until]: OK (rc = 0) 05:34:43 DEBUG --- stdout --- 05:34:43 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 9240m 8806Mi am-654c4fcbcf-9scrj 8706m 8424Mi am-654c4fcbcf-v5qb5 7381m 8397Mi ds-cts-0 7121m 12876Mi ds-cts-1 7243m 12879Mi ds-cts-2 6836m 12394Mi ds-idrepo-0 9474m 5622Mi ds-idrepo-1 316m 5669Mi ds-idrepo-2 332m 5698Mi end-user-ui-6d9fd5487b-2s47b 1m 5Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2212Mi lodemon-b776b7d88-thrqf 8m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3534m 1667Mi 05:34:43 DEBUG --- stderr --- 05:34:43 DEBUG 05:34:49 INFO 05:34:49 INFO [loop_until]: kubectl --namespace=xlou top node 05:34:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:34:49 INFO [loop_until]: OK (rc = 0) 05:34:49 DEBUG --- stdout --- 05:34:49 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 99m 0% 2308Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8230m 51% 9281Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 88m 0% 3258Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10860m 68% 6536Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9669m 60% 9311Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9945m 62% 9647Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 76m 0% 853Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9761m 61% 6282Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 7226m 45% 13437Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 7035m 44% 13039Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 388m 2% 6368Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7438m 46% 13498Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 381m 2% 6333Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3723m 23% 5345Mi 9% 05:34:49 DEBUG --- stderr --- 05:34:49 DEBUG 05:35:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712723724" 05:35:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723724" 05:35:24 INFO [http_cmd]: http status code OK 05:35:24 ERROR [http_cmd]: ERROR 05:35:24 DEBUG --- status code --- 05:35:24 ERROR --- status code --- 05:35:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:24 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:35:24 DEBUG --- http response --- 05:35:24 ERROR --- http response --- 05:35:24 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:35:24 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:35:24 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:35:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723724" 05:35:24 INFO [http_cmd]: http status code OK 05:35:24 DEBUG --- status code --- 05:35:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:24 DEBUG --- http response --- 05:35:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:35:24 DEBUG ----- output ----- 05:35:24 DEBUG { 05:35:24 DEBUG "status": "success", 05:35:24 DEBUG "data": { 05:35:24 DEBUG "resultType": "vector", 05:35:24 DEBUG "result": [ 05:35:24 DEBUG { 05:35:24 DEBUG "metric": { 05:35:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:35:24 DEBUG }, 05:35:24 DEBUG "value": [ 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG [...] 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:35:24 DEBUG }, 05:35:24 DEBUG "value": [ 05:35:24 DEBUG 1712723724, 05:35:24 DEBUG "0" 05:35:24 DEBUG ] 05:35:24 DEBUG } 05:35:24 DEBUG ] 05:35:24 DEBUG } 05:35:24 DEBUG } 05:35:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723724" 05:35:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723724" 05:35:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723724" 05:35:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723724" 05:35:24 INFO [http_cmd]: http status code OK 05:35:24 DEBUG --- status code --- 05:35:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:24 DEBUG --- http response --- 05:35:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:35:24 DEBUG ----- output ----- 05:35:24 DEBUG { 05:35:24 DEBUG "status": "success", 05:35:24 DEBUG "data": { 05:35:24 DEBUG "resultType": "vector", 05:35:24 DEBUG "result": [ 05:35:24 DEBUG { 05:35:24 DEBUG "metric": { 05:35:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:35:24 DEBUG }, 05:35:24 DEBUG "value": [ 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG [...] 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:35:24 DEBUG }, 05:35:24 DEBUG "value": [ 05:35:24 DEBUG 1712723724, 05:35:24 DEBUG "1255.6" 05:35:24 DEBUG ] 05:35:24 DEBUG } 05:35:24 DEBUG ] 05:35:24 DEBUG } 05:35:24 DEBUG } 05:35:24 INFO [http_cmd]: http status code OK 05:35:24 INFO [http_cmd]: http status code OK 05:35:24 DEBUG --- status code --- 05:35:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:24 DEBUG --- http response --- 05:35:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:35:24 DEBUG ----- output ----- 05:35:24 DEBUG { 05:35:24 DEBUG "status": "success", 05:35:24 DEBUG "data": { 05:35:24 DEBUG "resultType": "vector", 05:35:24 DEBUG "result": [ 05:35:24 DEBUG { 05:35:24 DEBUG "metric": { 05:35:24 DEBUG "instance": "10.142.0.25:9100" 05:35:24 DEBUG }, 05:35:24 DEBUG "value": [ 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG [...] 05:35:24 DEBUG --- status code --- 05:35:24 INFO [http_cmd]: http status code OK 05:35:24 DEBUG 05:35:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:24 DEBUG --- status code --- 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:24 DEBUG "instance": "10.142.0.77:9100" 05:35:24 DEBUG }, 05:35:24 DEBUG --- http response --- 05:35:24 DEBUG "value": [ 05:35:24 DEBUG --- http response --- 05:35:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:35:24 DEBUG 1712723724, 05:35:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:35:24 DEBUG ----- output ----- 05:35:24 DEBUG "297772817408" 05:35:24 DEBUG ----- output ----- 05:35:24 DEBUG { 05:35:24 DEBUG ] 05:35:24 DEBUG { 05:35:24 DEBUG "status": "success", 05:35:24 DEBUG } 05:35:24 DEBUG "status": "success", 05:35:24 DEBUG "data": { 05:35:24 DEBUG ] 05:35:24 DEBUG "data": { 05:35:24 DEBUG "resultType": "vector", 05:35:24 DEBUG } 05:35:24 DEBUG "resultType": "vector", 05:35:24 DEBUG "result": [ 05:35:24 DEBUG } 05:35:24 DEBUG "result": [ 05:35:24 DEBUG { 05:35:24 DEBUG { 05:35:24 DEBUG "metric": { 05:35:24 DEBUG "metric": { 05:35:24 DEBUG "pod": "ds-idrepo-0" 05:35:24 DEBUG "pod": "ds-idrepo-0" 05:35:24 DEBUG }, 05:35:24 DEBUG }, 05:35:24 DEBUG "value": [ 05:35:24 DEBUG "value": [ 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG [...] 05:35:24 DEBUG 05:35:24 DEBUG [...] 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG "pod": "ds-cts-2" 05:35:24 DEBUG 05:35:24 DEBUG }, 05:35:24 DEBUG "pod": "ds-cts-2" 05:35:24 DEBUG "value": [ 05:35:24 DEBUG }, 05:35:24 DEBUG 1712723724, 05:35:24 DEBUG "29.633333333333333" 05:35:24 DEBUG "value": [ 05:35:24 DEBUG ] 05:35:24 DEBUG 1712723724, 05:35:24 DEBUG } 05:35:24 DEBUG "0" 05:35:24 DEBUG ] 05:35:24 DEBUG ] 05:35:24 DEBUG } 05:35:24 DEBUG } 05:35:24 DEBUG } 05:35:24 DEBUG ] 05:35:24 DEBUG } 05:35:24 DEBUG } 05:35:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723724" 05:35:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723724" 05:35:24 INFO [http_cmd]: http status code OK 05:35:24 INFO [http_cmd]: http status code OK 05:35:24 DEBUG --- status code --- 05:35:24 DEBUG --- status code --- 05:35:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:24 DEBUG --- http response --- 05:35:24 DEBUG --- http response --- 05:35:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:35:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:35:24 DEBUG ----- output ----- 05:35:24 DEBUG ----- output ----- 05:35:24 DEBUG { 05:35:24 DEBUG { 05:35:24 DEBUG "status": "success", 05:35:24 DEBUG "status": "success", 05:35:24 DEBUG "data": { 05:35:24 DEBUG "data": { 05:35:24 DEBUG "resultType": "vector", 05:35:24 DEBUG "resultType": "vector", 05:35:24 DEBUG "result": [ 05:35:24 DEBUG "result": [ 05:35:24 DEBUG { 05:35:24 DEBUG { 05:35:24 DEBUG "metric": { 05:35:24 DEBUG "metric": { 05:35:24 DEBUG "instance": "10.142.0.25:9100" 05:35:24 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:35:24 DEBUG }, 05:35:24 DEBUG }, 05:35:24 DEBUG "value": [ 05:35:24 DEBUG "value": [ 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG [...] 05:35:24 DEBUG [...] 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG "instance": "10.142.0.77:9100" 05:35:24 DEBUG "pod": "ds-idrepo-1" 05:35:24 DEBUG }, 05:35:24 DEBUG "value": [ 05:35:24 DEBUG }, 05:35:24 DEBUG 1712723724, 05:35:24 DEBUG "value": [ 05:35:24 DEBUG "37530396.66666667" 05:35:24 DEBUG 1712723724, 05:35:24 DEBUG ] 05:35:24 DEBUG "0" 05:35:24 DEBUG } 05:35:24 DEBUG ] 05:35:24 DEBUG ] 05:35:24 DEBUG } 05:35:24 DEBUG } 05:35:24 DEBUG } 05:35:24 DEBUG ] 05:35:24 DEBUG } 05:35:24 DEBUG } 05:35:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723724" 05:35:24 INFO [http_cmd]: http status code OK 05:35:24 DEBUG --- status code --- 05:35:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:24 DEBUG --- http response --- 05:35:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:35:24 DEBUG ----- output ----- 05:35:24 DEBUG { 05:35:24 DEBUG "status": "success", 05:35:24 DEBUG "data": { 05:35:24 DEBUG "resultType": "vector", 05:35:24 DEBUG "result": [ 05:35:24 DEBUG { 05:35:24 DEBUG "metric": { 05:35:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:35:24 DEBUG }, 05:35:24 DEBUG "value": [ 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG [...] 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG 05:35:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:35:24 DEBUG }, 05:35:24 DEBUG "value": [ 05:35:24 DEBUG 1712723724, 05:35:24 DEBUG "227538" 05:35:24 DEBUG ] 05:35:24 DEBUG } 05:35:24 DEBUG ] 05:35:24 DEBUG } 05:35:24 DEBUG } 05:35:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723725" 05:35:25 INFO [http_cmd]: http status code OK 05:35:25 DEBUG --- status code --- 05:35:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:25 DEBUG --- http response --- 05:35:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:35:25 DEBUG ----- output ----- 05:35:25 DEBUG { 05:35:25 DEBUG "status": "success", 05:35:25 DEBUG "data": { 05:35:25 DEBUG "resultType": "vector", 05:35:25 DEBUG "result": [ 05:35:25 DEBUG { 05:35:25 DEBUG "metric": { 05:35:25 DEBUG "pod": "ds-idrepo-0" 05:35:25 DEBUG }, 05:35:25 DEBUG "value": [ 05:35:25 DEBUG 05:35:25 DEBUG 05:35:25 DEBUG 05:35:25 DEBUG [...] 05:35:25 DEBUG 05:35:25 DEBUG 05:35:25 DEBUG 05:35:25 DEBUG "pod": "ds-idrepo-1" 05:35:25 DEBUG }, 05:35:25 DEBUG "value": [ 05:35:25 DEBUG 1712723725, 05:35:25 DEBUG "0" 05:35:25 DEBUG ] 05:35:25 DEBUG } 05:35:25 DEBUG ] 05:35:25 DEBUG } 05:35:25 DEBUG } 05:35:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723725" 05:35:25 INFO [http_cmd]: http status code OK 05:35:25 DEBUG --- status code --- 05:35:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:25 DEBUG --- http response --- 05:35:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:35:25 DEBUG ----- output ----- 05:35:25 DEBUG { 05:35:25 DEBUG "status": "success", 05:35:25 DEBUG "data": { 05:35:25 DEBUG "resultType": "vector", 05:35:25 DEBUG "result": [ 05:35:25 DEBUG { 05:35:25 DEBUG "metric": { 05:35:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:35:25 DEBUG }, 05:35:25 DEBUG "value": [ 05:35:25 DEBUG 05:35:25 DEBUG 05:35:25 DEBUG 05:35:25 DEBUG [...] 05:35:25 DEBUG 05:35:25 DEBUG 05:35:25 DEBUG 05:35:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:35:25 DEBUG }, 05:35:25 DEBUG "value": [ 05:35:25 DEBUG 1712723725, 05:35:25 DEBUG "1254.4666666666667" 05:35:25 DEBUG ] 05:35:25 DEBUG } 05:35:25 DEBUG ] 05:35:25 DEBUG } 05:35:25 DEBUG } 05:35:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723725" 05:35:25 INFO [http_cmd]: http status code OK 05:35:25 DEBUG --- status code --- 05:35:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:25 DEBUG --- http response --- 05:35:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:35:25 DEBUG ----- output ----- 05:35:25 DEBUG { 05:35:25 DEBUG "status": "success", 05:35:25 DEBUG "data": { 05:35:25 DEBUG "resultType": "vector", 05:35:25 DEBUG "result": [ 05:35:25 DEBUG { 05:35:25 DEBUG "metric": { 05:35:25 DEBUG "instance": "10.142.0.25:9100" 05:35:25 DEBUG }, 05:35:25 DEBUG "value": [ 05:35:25 DEBUG 05:35:25 DEBUG 05:35:25 DEBUG 05:35:25 DEBUG [...] 05:35:25 DEBUG 05:35:25 DEBUG 05:35:25 DEBUG 05:35:25 DEBUG "instance": "10.142.0.77:9100" 05:35:25 DEBUG }, 05:35:25 DEBUG "value": [ 05:35:25 DEBUG 1712723725, 05:35:25 DEBUG "0.10930000000000605" 05:35:25 DEBUG ] 05:35:25 DEBUG } 05:35:25 DEBUG ] 05:35:25 DEBUG } 05:35:25 DEBUG } 05:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723726" 05:35:26 INFO [http_cmd]: http status code OK 05:35:26 DEBUG --- status code --- 05:35:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:26 DEBUG --- http response --- 05:35:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:35:26 DEBUG ----- output ----- 05:35:26 DEBUG { 05:35:26 DEBUG "status": "success", 05:35:26 DEBUG "data": { 05:35:26 DEBUG "resultType": "vector", 05:35:26 DEBUG "result": [ 05:35:26 DEBUG { 05:35:26 DEBUG "metric": { 05:35:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:35:26 DEBUG }, 05:35:26 DEBUG "value": [ 05:35:26 DEBUG 05:35:26 DEBUG 05:35:26 DEBUG 05:35:26 DEBUG [...] 05:35:26 DEBUG 05:35:26 DEBUG 05:35:26 DEBUG 05:35:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:35:26 DEBUG }, 05:35:26 DEBUG "value": [ 05:35:26 DEBUG 1712723726, 05:35:26 DEBUG "326554675.91184074" 05:35:26 DEBUG ] 05:35:26 DEBUG } 05:35:26 DEBUG ] 05:35:26 DEBUG } 05:35:26 DEBUG } 05:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712723726" 05:35:26 INFO [http_cmd]: http status code OK 05:35:26 DEBUG --- status code --- 05:35:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:26 DEBUG --- http response --- 05:35:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:35:26 DEBUG ----- output ----- 05:35:26 DEBUG { 05:35:26 DEBUG "status": "success", 05:35:26 DEBUG "data": { 05:35:26 DEBUG "resultType": "vector", 05:35:26 DEBUG "result": [ 05:35:26 DEBUG { 05:35:26 DEBUG "metric": { 05:35:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:35:26 DEBUG }, 05:35:26 DEBUG "value": [ 05:35:26 DEBUG 05:35:26 DEBUG 05:35:26 DEBUG 05:35:26 DEBUG [...] 05:35:26 DEBUG 05:35:26 DEBUG 05:35:26 DEBUG 05:35:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:35:26 DEBUG }, 05:35:26 DEBUG "value": [ 05:35:26 DEBUG 1712723726, 05:35:26 DEBUG "3.6138176905787813" 05:35:26 DEBUG ] 05:35:26 DEBUG } 05:35:26 DEBUG ] 05:35:26 DEBUG } 05:35:26 DEBUG } 05:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712723726" 05:35:26 INFO [http_cmd]: http status code OK 05:35:26 DEBUG --- status code --- 05:35:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:26 DEBUG --- http response --- 05:35:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:35:26 DEBUG ----- output ----- 05:35:26 DEBUG { 05:35:26 DEBUG "status": "success", 05:35:26 DEBUG "data": { 05:35:26 DEBUG "resultType": "vector", 05:35:26 DEBUG "result": [ 05:35:26 DEBUG { 05:35:26 DEBUG "metric": { 05:35:26 DEBUG "pod": "ds-idrepo-0" 05:35:26 DEBUG }, 05:35:26 DEBUG "value": [ 05:35:26 DEBUG 05:35:26 DEBUG 05:35:26 DEBUG 05:35:26 DEBUG [...] 05:35:26 DEBUG 05:35:26 DEBUG 05:35:26 DEBUG 05:35:26 DEBUG "pod": "ds-cts-2" 05:35:26 DEBUG }, 05:35:26 DEBUG "value": [ 05:35:26 DEBUG 1712723726, 05:35:26 DEBUG "0" 05:35:26 DEBUG ] 05:35:26 DEBUG } 05:35:26 DEBUG ] 05:35:26 DEBUG } 05:35:26 DEBUG } 05:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723726" 05:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723726" 05:35:26 INFO [http_cmd]: http status code OK 05:35:26 INFO [http_cmd]: http status code OK 05:35:26 DEBUG --- status code --- 05:35:26 DEBUG --- status code --- 05:35:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:26 DEBUG --- http response --- 05:35:26 DEBUG --- http response --- 05:35:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:35:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:35:26 DEBUG ----- output ----- 05:35:26 DEBUG ----- output ----- 05:35:26 DEBUG { 05:35:26 DEBUG { 05:35:26 DEBUG "status": "success", 05:35:26 DEBUG "status": "success", 05:35:26 DEBUG "data": { 05:35:26 DEBUG "data": { 05:35:26 DEBUG "resultType": "vector", 05:35:26 DEBUG "resultType": "vector", 05:35:26 DEBUG "result": [ 05:35:26 DEBUG "result": [ 05:35:26 DEBUG { 05:35:26 DEBUG { 05:35:26 DEBUG "metric": { 05:35:26 DEBUG "metric": { 05:35:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:35:26 DEBUG "pod": "ds-idrepo-2" 05:35:26 DEBUG }, 05:35:26 DEBUG }, 05:35:26 DEBUG "value": [ 05:35:26 DEBUG "value": [ 05:35:26 DEBUG 05:35:26 DEBUG 05:35:26 DEBUG 05:35:26 DEBUG 05:35:26 DEBUG 05:35:26 DEBUG 05:35:26 DEBUG [...] 05:35:26 DEBUG [...] 05:35:26 DEBUG 05:35:26 DEBUG 05:35:26 DEBUG 05:35:26 DEBUG 05:35:26 DEBUG 05:35:26 DEBUG 05:35:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:35:26 DEBUG "pod": "ds-cts-2" 05:35:26 DEBUG }, 05:35:26 DEBUG }, 05:35:26 DEBUG "value": [ 05:35:26 DEBUG "value": [ 05:35:26 DEBUG 1712723726, 05:35:26 DEBUG 1712723726, 05:35:26 DEBUG "0" 05:35:26 DEBUG "8.974060351788456" 05:35:26 DEBUG ] 05:35:26 DEBUG ] 05:35:26 DEBUG } 05:35:26 DEBUG } 05:35:26 DEBUG ] 05:35:26 DEBUG ] 05:35:26 DEBUG } 05:35:26 DEBUG } 05:35:26 DEBUG } 05:35:26 DEBUG } 05:35:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723727" 05:35:27 INFO [http_cmd]: http status code OK 05:35:27 DEBUG --- status code --- 05:35:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:27 DEBUG --- http response --- 05:35:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:35:27 DEBUG ----- output ----- 05:35:27 DEBUG { 05:35:27 DEBUG "status": "success", 05:35:27 DEBUG "data": { 05:35:27 DEBUG "resultType": "vector", 05:35:27 DEBUG "result": [ 05:35:27 DEBUG { 05:35:27 DEBUG "metric": { 05:35:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:35:27 DEBUG }, 05:35:27 DEBUG "value": [ 05:35:27 DEBUG 05:35:27 DEBUG 05:35:27 DEBUG 05:35:27 DEBUG [...] 05:35:27 DEBUG 05:35:27 DEBUG 05:35:27 DEBUG 05:35:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:35:27 DEBUG }, 05:35:27 DEBUG "value": [ 05:35:27 DEBUG 1712723727, 05:35:27 DEBUG "1254.1" 05:35:27 DEBUG ] 05:35:27 DEBUG } 05:35:27 DEBUG ] 05:35:27 DEBUG } 05:35:27 DEBUG } 05:35:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712723727" 05:35:27 INFO [http_cmd]: http status code OK 05:35:27 DEBUG --- status code --- 05:35:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:27 DEBUG --- http response --- 05:35:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:35:27 DEBUG ----- output ----- 05:35:27 DEBUG { 05:35:27 DEBUG "status": "success", 05:35:27 DEBUG "data": { 05:35:27 DEBUG "resultType": "vector", 05:35:27 DEBUG "result": [ 05:35:27 DEBUG { 05:35:27 DEBUG "metric": { 05:35:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:35:27 DEBUG }, 05:35:27 DEBUG "value": [ 05:35:27 DEBUG 05:35:27 DEBUG 05:35:27 DEBUG 05:35:27 DEBUG [...] 05:35:27 DEBUG 05:35:27 DEBUG 05:35:27 DEBUG 05:35:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:35:27 DEBUG }, 05:35:27 DEBUG "value": [ 05:35:27 DEBUG 1712723727, 05:35:27 DEBUG "0.042" 05:35:27 DEBUG ] 05:35:27 DEBUG } 05:35:27 DEBUG ] 05:35:27 DEBUG } 05:35:27 DEBUG } 05:35:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723727" 05:35:27 INFO [http_cmd]: http status code OK 05:35:27 DEBUG --- status code --- 05:35:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:27 DEBUG --- http response --- 05:35:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:35:27 DEBUG ----- output ----- 05:35:27 DEBUG { 05:35:27 DEBUG "status": "success", 05:35:27 DEBUG "data": { 05:35:27 DEBUG "resultType": "vector", 05:35:27 DEBUG "result": [ 05:35:27 DEBUG { 05:35:27 DEBUG "metric": { 05:35:27 DEBUG "pod": "ds-idrepo-0" 05:35:27 DEBUG }, 05:35:27 DEBUG "value": [ 05:35:27 DEBUG 05:35:27 DEBUG 05:35:27 DEBUG 05:35:27 DEBUG [...] 05:35:27 DEBUG 05:35:27 DEBUG 05:35:27 DEBUG 05:35:27 DEBUG "pod": "ds-idrepo-1" 05:35:27 DEBUG }, 05:35:27 DEBUG "value": [ 05:35:27 DEBUG 1712723727, 05:35:27 DEBUG "0" 05:35:27 DEBUG ] 05:35:27 DEBUG } 05:35:27 DEBUG ] 05:35:27 DEBUG } 05:35:27 DEBUG } 05:35:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723727" 05:35:27 INFO [http_cmd]: http status code OK 05:35:27 DEBUG --- status code --- 05:35:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:27 DEBUG --- http response --- 05:35:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:35:27 DEBUG ----- output ----- 05:35:27 DEBUG { 05:35:27 DEBUG "status": "success", 05:35:27 DEBUG "data": { 05:35:27 DEBUG "resultType": "vector", 05:35:27 DEBUG "result": [ 05:35:27 DEBUG { 05:35:27 DEBUG "metric": { 05:35:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:35:27 DEBUG }, 05:35:27 DEBUG "value": [ 05:35:27 DEBUG 05:35:27 DEBUG 05:35:27 DEBUG 05:35:27 DEBUG [...] 05:35:27 DEBUG 05:35:27 DEBUG 05:35:27 DEBUG 05:35:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:35:27 DEBUG }, 05:35:27 DEBUG "value": [ 05:35:27 DEBUG 1712723727, 05:35:27 DEBUG "1271.7333333333333" 05:35:27 DEBUG ] 05:35:27 DEBUG } 05:35:27 DEBUG ] 05:35:27 DEBUG } 05:35:27 DEBUG } 05:35:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723727" 05:35:27 INFO [http_cmd]: http status code OK 05:35:27 DEBUG --- status code --- 05:35:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:27 DEBUG --- http response --- 05:35:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:35:27 DEBUG ----- output ----- 05:35:27 DEBUG { 05:35:27 DEBUG "status": "success", 05:35:27 DEBUG "data": { 05:35:27 DEBUG "resultType": "vector", 05:35:27 DEBUG "result": [ 05:35:27 DEBUG { 05:35:27 DEBUG "metric": { 05:35:27 DEBUG "pod": "ds-idrepo-0" 05:35:27 DEBUG }, 05:35:27 DEBUG "value": [ 05:35:27 DEBUG 05:35:27 DEBUG 05:35:27 DEBUG 05:35:27 DEBUG [...] 05:35:27 DEBUG 05:35:27 DEBUG 05:35:27 DEBUG 05:35:27 DEBUG "pod": "ds-idrepo-1" 05:35:27 DEBUG }, 05:35:27 DEBUG "value": [ 05:35:27 DEBUG 1712723727, 05:35:27 DEBUG "0" 05:35:27 DEBUG ] 05:35:27 DEBUG } 05:35:27 DEBUG ] 05:35:27 DEBUG } 05:35:27 DEBUG } 05:35:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723727" 05:35:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712723727" 05:35:27 INFO [http_cmd]: http status code OK 05:35:27 DEBUG --- status code --- 05:35:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:27 DEBUG --- http response --- 05:35:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:35:27 DEBUG ----- output ----- 05:35:27 INFO [http_cmd]: http status code OK 05:35:27 DEBUG { 05:35:27 DEBUG --- status code --- 05:35:27 DEBUG "status": "success", 05:35:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:27 DEBUG "data": { 05:35:27 DEBUG --- http response --- 05:35:27 DEBUG "resultType": "vector", 05:35:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:35:27 DEBUG "result": [ 05:35:27 DEBUG ----- output ----- 05:35:27 DEBUG { 05:35:27 DEBUG { 05:35:27 DEBUG "metric": { 05:35:27 DEBUG "status": "success", 05:35:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:35:27 DEBUG "data": { 05:35:27 DEBUG }, 05:35:27 DEBUG "resultType": "vector", 05:35:27 DEBUG "value": [ 05:35:27 DEBUG "result": [ 05:35:27 DEBUG 05:35:27 DEBUG { 05:35:27 DEBUG 05:35:27 DEBUG "metric": { 05:35:27 DEBUG 05:35:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:35:27 DEBUG [...] 05:35:27 DEBUG }, 05:35:27 DEBUG 05:35:27 DEBUG "value": [ 05:35:27 DEBUG 05:35:27 DEBUG 05:35:27 DEBUG 05:35:27 DEBUG 05:35:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:35:27 DEBUG 05:35:27 DEBUG }, 05:35:27 DEBUG [...] 05:35:27 DEBUG "value": [ 05:35:27 DEBUG 1712723727, 05:35:27 DEBUG 05:35:27 DEBUG "0" 05:35:27 DEBUG 05:35:27 DEBUG ] 05:35:27 DEBUG 05:35:27 DEBUG } 05:35:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:35:27 DEBUG ] 05:35:27 DEBUG }, 05:35:27 DEBUG } 05:35:27 DEBUG "value": [ 05:35:27 DEBUG } 05:35:27 DEBUG 1712723727, 05:35:27 DEBUG "2488905728" 05:35:27 DEBUG ] 05:35:27 DEBUG } 05:35:27 DEBUG ] 05:35:27 DEBUG } 05:35:27 DEBUG } 05:35:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723728" 05:35:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723728" 05:35:28 INFO [http_cmd]: http status code OK 05:35:28 DEBUG --- status code --- 05:35:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:28 DEBUG --- http response --- 05:35:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:35:28 INFO [http_cmd]: http status code OK 05:35:28 DEBUG ----- output ----- 05:35:28 DEBUG --- status code --- 05:35:28 DEBUG { 05:35:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:28 DEBUG "status": "success", 05:35:28 DEBUG "data": { 05:35:28 DEBUG "resultType": "vector", 05:35:28 DEBUG --- http response --- 05:35:28 DEBUG "result": [ 05:35:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:35:28 DEBUG { 05:35:28 DEBUG ----- output ----- 05:35:28 DEBUG "metric": { 05:35:28 DEBUG "instance": "10.142.0.25:9100" 05:35:28 DEBUG }, 05:35:28 DEBUG "value": [ 05:35:28 DEBUG 05:35:28 DEBUG 05:35:28 DEBUG 05:35:28 DEBUG [...] 05:35:28 DEBUG 05:35:28 DEBUG 05:35:28 DEBUG 05:35:28 DEBUG "instance": "10.142.0.77:9100" 05:35:28 DEBUG }, 05:35:28 DEBUG "value": [ 05:35:28 DEBUG 1712723728, 05:35:28 DEBUG "37533124.06666666" 05:35:28 DEBUG ] 05:35:28 DEBUG } 05:35:28 DEBUG ] 05:35:28 DEBUG } 05:35:28 DEBUG } 05:35:28 DEBUG { 05:35:28 DEBUG "status": "success", 05:35:28 DEBUG "data": { 05:35:28 DEBUG "resultType": "vector", 05:35:28 DEBUG "result": [ 05:35:28 DEBUG { 05:35:28 DEBUG "metric": { 05:35:28 DEBUG "instance": "10.142.0.25:9100" 05:35:28 DEBUG }, 05:35:28 DEBUG "value": [ 05:35:28 DEBUG 05:35:28 DEBUG 05:35:28 DEBUG 05:35:28 DEBUG [...] 05:35:28 DEBUG 05:35:28 DEBUG 05:35:28 DEBUG 05:35:28 DEBUG "instance": "10.142.0.77:9100" 05:35:28 DEBUG }, 05:35:28 DEBUG "value": [ 05:35:28 DEBUG 1712723728, 05:35:28 DEBUG "0.03733333333333331" 05:35:28 DEBUG ] 05:35:28 DEBUG } 05:35:28 DEBUG ] 05:35:28 DEBUG } 05:35:28 DEBUG } 05:35:43 INFO 05:35:43 INFO [loop_until]: kubectl --namespace=xlou top pods 05:35:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:35:43 INFO [loop_until]: OK (rc = 0) 05:35:43 DEBUG --- stdout --- 05:35:43 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 8986m 8807Mi am-654c4fcbcf-9scrj 8499m 8422Mi am-654c4fcbcf-v5qb5 7499m 8397Mi ds-cts-0 7167m 12406Mi ds-cts-1 7490m 12837Mi ds-cts-2 6995m 12595Mi ds-idrepo-0 9685m 5622Mi ds-idrepo-1 358m 5670Mi ds-idrepo-2 387m 5698Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 2m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3604m 1668Mi 05:35:43 DEBUG --- stderr --- 05:35:43 DEBUG 05:35:49 INFO 05:35:49 INFO [loop_until]: kubectl --namespace=xlou top node 05:35:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:35:49 INFO [loop_until]: OK (rc = 0) 05:35:49 DEBUG --- stdout --- 05:35:49 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 96m 0% 2306Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 7905m 49% 9283Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 83m 0% 3258Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10879m 68% 6522Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9591m 60% 9322Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9983m 62% 9645Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 77m 0% 857Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9802m 61% 6282Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 7337m 46% 13565Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 7240m 45% 13170Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 394m 2% 6366Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7423m 46% 13774Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 433m 2% 6336Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3712m 23% 5347Mi 9% 05:35:49 DEBUG --- stderr --- 05:35:49 DEBUG 05:36:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723784" 05:36:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712723784" 05:36:24 ERROR [http_cmd]: ERROR 05:36:24 ERROR --- status code --- 05:36:24 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:36:24 ERROR --- http response --- 05:36:24 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:36:24 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:36:24 INFO [http_cmd]: http status code OK 05:36:24 DEBUG --- status code --- 05:36:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:24 DEBUG --- http response --- 05:36:24 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:36:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723784" 05:36:24 INFO [http_cmd]: http status code OK 05:36:24 DEBUG --- status code --- 05:36:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:24 DEBUG --- http response --- 05:36:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:36:24 DEBUG ----- output ----- 05:36:24 DEBUG { 05:36:24 DEBUG "status": "success", 05:36:24 DEBUG "data": { 05:36:24 DEBUG "resultType": "vector", 05:36:24 DEBUG "result": [ 05:36:24 DEBUG { 05:36:24 DEBUG "metric": { 05:36:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:36:24 DEBUG }, 05:36:24 DEBUG "value": [ 05:36:24 DEBUG 05:36:24 DEBUG 05:36:24 DEBUG 05:36:24 DEBUG [...] 05:36:24 DEBUG 05:36:24 DEBUG 05:36:24 DEBUG 05:36:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:36:24 DEBUG }, 05:36:24 DEBUG "value": [ 05:36:24 DEBUG 1712723784, 05:36:24 DEBUG "0" 05:36:24 DEBUG ] 05:36:24 DEBUG } 05:36:24 DEBUG ] 05:36:24 DEBUG } 05:36:24 DEBUG } 05:36:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723784" 05:36:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723784" 05:36:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723784" 05:36:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723784" 05:36:24 INFO [http_cmd]: http status code OK 05:36:24 DEBUG --- status code --- 05:36:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:24 DEBUG --- http response --- 05:36:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:36:24 DEBUG ----- output ----- 05:36:24 DEBUG { 05:36:24 DEBUG "status": "success", 05:36:24 DEBUG "data": { 05:36:24 DEBUG "resultType": "vector", 05:36:24 DEBUG "result": [ 05:36:24 DEBUG { 05:36:24 DEBUG "metric": { 05:36:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:36:24 DEBUG }, 05:36:24 DEBUG "value": [ 05:36:24 DEBUG 05:36:24 DEBUG 05:36:24 DEBUG 05:36:24 DEBUG [...] 05:36:24 DEBUG 05:36:24 DEBUG 05:36:24 DEBUG 05:36:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:36:24 DEBUG }, 05:36:24 DEBUG "value": [ 05:36:24 DEBUG 1712723784, 05:36:24 DEBUG "1262" 05:36:24 DEBUG ] 05:36:24 DEBUG } 05:36:24 DEBUG ] 05:36:24 DEBUG } 05:36:24 DEBUG } 05:36:24 INFO [http_cmd]: http status code OK 05:36:24 DEBUG --- status code --- 05:36:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:24 DEBUG --- http response --- 05:36:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:36:24 DEBUG ----- output ----- 05:36:24 DEBUG { 05:36:24 DEBUG "status": "success", 05:36:24 DEBUG "data": { 05:36:24 DEBUG "resultType": "vector", 05:36:24 DEBUG "result": [ 05:36:24 DEBUG { 05:36:24 DEBUG "metric": { 05:36:24 DEBUG "instance": "10.142.0.25:9100" 05:36:24 DEBUG }, 05:36:24 DEBUG "value": [ 05:36:24 DEBUG 05:36:24 DEBUG 05:36:24 DEBUG 05:36:24 DEBUG [...] 05:36:24 DEBUG 05:36:24 DEBUG 05:36:24 DEBUG 05:36:24 DEBUG "instance": "10.142.0.77:9100" 05:36:24 DEBUG }, 05:36:24 DEBUG "value": [ 05:36:24 DEBUG 1712723784, 05:36:24 DEBUG "299722941440" 05:36:24 DEBUG ] 05:36:24 DEBUG } 05:36:24 DEBUG ] 05:36:24 DEBUG } 05:36:24 DEBUG } 05:36:24 INFO [http_cmd]: http status code OK 05:36:24 DEBUG --- status code --- 05:36:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:24 DEBUG --- http response --- 05:36:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:36:24 INFO [http_cmd]: http status code OK 05:36:24 DEBUG ----- output ----- 05:36:24 DEBUG --- status code --- 05:36:24 DEBUG { 05:36:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:24 DEBUG "status": "success", 05:36:24 DEBUG --- http response --- 05:36:24 DEBUG "data": { 05:36:24 DEBUG "resultType": "vector", 05:36:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:36:24 DEBUG "result": [ 05:36:24 DEBUG { 05:36:24 DEBUG "metric": { 05:36:24 DEBUG ----- output ----- 05:36:24 DEBUG "pod": "ds-idrepo-0" 05:36:24 DEBUG { 05:36:24 DEBUG }, 05:36:24 DEBUG "status": "success", 05:36:24 DEBUG "value": [ 05:36:24 DEBUG "data": { 05:36:24 DEBUG 05:36:24 DEBUG "resultType": "vector", 05:36:24 DEBUG 05:36:24 DEBUG "result": [ 05:36:24 DEBUG 05:36:24 DEBUG { 05:36:24 DEBUG [...] 05:36:24 DEBUG "metric": { 05:36:24 DEBUG 05:36:24 DEBUG "pod": "ds-idrepo-0" 05:36:24 DEBUG 05:36:24 DEBUG }, 05:36:24 DEBUG 05:36:24 DEBUG "value": [ 05:36:24 DEBUG "pod": "ds-cts-2" 05:36:24 DEBUG 05:36:24 DEBUG }, 05:36:24 DEBUG 05:36:24 DEBUG "value": [ 05:36:24 DEBUG 05:36:24 DEBUG 1712723784, 05:36:24 DEBUG [...] 05:36:24 DEBUG "0" 05:36:24 DEBUG 05:36:24 DEBUG ] 05:36:24 DEBUG 05:36:24 DEBUG } 05:36:24 DEBUG ] 05:36:24 DEBUG 05:36:24 DEBUG } 05:36:24 DEBUG "pod": "ds-cts-2" 05:36:24 DEBUG } 05:36:24 DEBUG }, 05:36:24 DEBUG "value": [ 05:36:24 DEBUG 1712723784, 05:36:24 DEBUG "29.9" 05:36:24 DEBUG ] 05:36:24 DEBUG } 05:36:24 DEBUG ] 05:36:24 DEBUG } 05:36:24 DEBUG } 05:36:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723784" 05:36:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723784" 05:36:24 INFO [http_cmd]: http status code OK 05:36:24 INFO [http_cmd]: http status code OK 05:36:24 DEBUG --- status code --- 05:36:24 DEBUG --- status code --- 05:36:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:24 DEBUG --- http response --- 05:36:24 DEBUG --- http response --- 05:36:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:36:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:36:24 DEBUG ----- output ----- 05:36:24 DEBUG ----- output ----- 05:36:24 DEBUG { 05:36:24 DEBUG { 05:36:24 DEBUG "status": "success", 05:36:24 DEBUG "status": "success", 05:36:24 DEBUG "data": { 05:36:24 DEBUG "data": { 05:36:24 DEBUG "resultType": "vector", 05:36:24 DEBUG "resultType": "vector", 05:36:24 DEBUG "result": [ 05:36:24 DEBUG { 05:36:24 DEBUG "metric": { 05:36:24 DEBUG "instance": "10.142.0.25:9100" 05:36:24 DEBUG }, 05:36:24 DEBUG "value": [ 05:36:24 DEBUG 05:36:24 DEBUG 05:36:24 DEBUG 05:36:24 DEBUG [...] 05:36:24 DEBUG 05:36:24 DEBUG 05:36:24 DEBUG 05:36:24 DEBUG "instance": "10.142.0.77:9100" 05:36:24 DEBUG }, 05:36:24 DEBUG "value": [ 05:36:24 DEBUG 1712723784, 05:36:24 DEBUG "37513085.9" 05:36:24 DEBUG ] 05:36:24 DEBUG } 05:36:24 DEBUG ] 05:36:24 DEBUG } 05:36:24 DEBUG } 05:36:24 DEBUG "result": [ 05:36:24 DEBUG { 05:36:24 DEBUG "metric": { 05:36:24 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:36:24 DEBUG }, 05:36:24 DEBUG "value": [ 05:36:24 DEBUG 05:36:24 DEBUG 05:36:24 DEBUG 05:36:24 DEBUG [...] 05:36:24 DEBUG 05:36:24 DEBUG 05:36:24 DEBUG 05:36:24 DEBUG "pod": "ds-idrepo-1" 05:36:24 DEBUG }, 05:36:24 DEBUG "value": [ 05:36:24 DEBUG 1712723784, 05:36:24 DEBUG "0" 05:36:24 DEBUG ] 05:36:24 DEBUG } 05:36:24 DEBUG ] 05:36:24 DEBUG } 05:36:24 DEBUG } 05:36:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723784" 05:36:24 INFO [http_cmd]: http status code OK 05:36:24 DEBUG --- status code --- 05:36:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:24 DEBUG --- http response --- 05:36:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:36:24 DEBUG ----- output ----- 05:36:24 DEBUG { 05:36:24 DEBUG "status": "success", 05:36:24 DEBUG "data": { 05:36:24 DEBUG "resultType": "vector", 05:36:24 DEBUG "result": [ 05:36:24 DEBUG { 05:36:24 DEBUG "metric": { 05:36:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:36:24 DEBUG }, 05:36:24 DEBUG "value": [ 05:36:24 DEBUG 05:36:24 DEBUG 05:36:24 DEBUG 05:36:24 DEBUG [...] 05:36:24 DEBUG 05:36:24 DEBUG 05:36:24 DEBUG 05:36:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:36:24 DEBUG }, 05:36:24 DEBUG "value": [ 05:36:24 DEBUG 1712723784, 05:36:24 DEBUG "227443.5" 05:36:24 DEBUG ] 05:36:24 DEBUG } 05:36:24 DEBUG ] 05:36:24 DEBUG } 05:36:24 DEBUG } 05:36:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723785" 05:36:25 INFO [http_cmd]: http status code OK 05:36:25 DEBUG --- status code --- 05:36:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:25 DEBUG --- http response --- 05:36:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:36:25 DEBUG ----- output ----- 05:36:25 DEBUG { 05:36:25 DEBUG "status": "success", 05:36:25 DEBUG "data": { 05:36:25 DEBUG "resultType": "vector", 05:36:25 DEBUG "result": [ 05:36:25 DEBUG { 05:36:25 DEBUG "metric": { 05:36:25 DEBUG "pod": "ds-idrepo-0" 05:36:25 DEBUG }, 05:36:25 DEBUG "value": [ 05:36:25 DEBUG 05:36:25 DEBUG 05:36:25 DEBUG 05:36:25 DEBUG [...] 05:36:25 DEBUG 05:36:25 DEBUG 05:36:25 DEBUG 05:36:25 DEBUG "pod": "ds-idrepo-1" 05:36:25 DEBUG }, 05:36:25 DEBUG "value": [ 05:36:25 DEBUG 1712723785, 05:36:25 DEBUG "0" 05:36:25 DEBUG ] 05:36:25 DEBUG } 05:36:25 DEBUG ] 05:36:25 DEBUG } 05:36:25 DEBUG } 05:36:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723785" 05:36:25 INFO [http_cmd]: http status code OK 05:36:25 DEBUG --- status code --- 05:36:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:25 DEBUG --- http response --- 05:36:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:36:25 DEBUG ----- output ----- 05:36:25 DEBUG { 05:36:25 DEBUG "status": "success", 05:36:25 DEBUG "data": { 05:36:25 DEBUG "resultType": "vector", 05:36:25 DEBUG "result": [ 05:36:25 DEBUG { 05:36:25 DEBUG "metric": { 05:36:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:36:25 DEBUG }, 05:36:25 DEBUG "value": [ 05:36:25 DEBUG 05:36:25 DEBUG 05:36:25 DEBUG 05:36:25 DEBUG [...] 05:36:25 DEBUG 05:36:25 DEBUG 05:36:25 DEBUG 05:36:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:36:25 DEBUG }, 05:36:25 DEBUG "value": [ 05:36:25 DEBUG 1712723785, 05:36:25 DEBUG "1261.9" 05:36:25 DEBUG ] 05:36:25 DEBUG } 05:36:25 DEBUG ] 05:36:25 DEBUG } 05:36:25 DEBUG } 05:36:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723785" 05:36:25 INFO [http_cmd]: http status code OK 05:36:25 DEBUG --- status code --- 05:36:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:25 DEBUG --- http response --- 05:36:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:36:25 DEBUG ----- output ----- 05:36:25 DEBUG { 05:36:25 DEBUG "status": "success", 05:36:25 DEBUG "data": { 05:36:25 DEBUG "resultType": "vector", 05:36:25 DEBUG "result": [ 05:36:25 DEBUG { 05:36:25 DEBUG "metric": { 05:36:25 DEBUG "instance": "10.142.0.25:9100" 05:36:25 DEBUG }, 05:36:25 DEBUG "value": [ 05:36:25 DEBUG 05:36:25 DEBUG 05:36:25 DEBUG 05:36:25 DEBUG [...] 05:36:25 DEBUG 05:36:25 DEBUG 05:36:25 DEBUG 05:36:25 DEBUG "instance": "10.142.0.77:9100" 05:36:25 DEBUG }, 05:36:25 DEBUG "value": [ 05:36:25 DEBUG 1712723785, 05:36:25 DEBUG "0.04933333333333855" 05:36:25 DEBUG ] 05:36:25 DEBUG } 05:36:25 DEBUG ] 05:36:25 DEBUG } 05:36:25 DEBUG } 05:36:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723786" 05:36:26 INFO [http_cmd]: http status code OK 05:36:26 DEBUG --- status code --- 05:36:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:26 DEBUG --- http response --- 05:36:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:36:26 DEBUG ----- output ----- 05:36:26 DEBUG { 05:36:26 DEBUG "status": "success", 05:36:26 DEBUG "data": { 05:36:26 DEBUG "resultType": "vector", 05:36:26 DEBUG "result": [ 05:36:26 DEBUG { 05:36:26 DEBUG "metric": { 05:36:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:36:26 DEBUG }, 05:36:26 DEBUG "value": [ 05:36:26 DEBUG 05:36:26 DEBUG 05:36:26 DEBUG 05:36:26 DEBUG [...] 05:36:26 DEBUG 05:36:26 DEBUG 05:36:26 DEBUG 05:36:26 DEBUG "pod": "ds-idrepo-1" 05:36:26 DEBUG }, 05:36:26 DEBUG "value": [ 05:36:26 DEBUG 1712723786, 05:36:26 DEBUG "5323.282864383651" 05:36:26 DEBUG ] 05:36:26 DEBUG } 05:36:26 DEBUG ] 05:36:26 DEBUG } 05:36:26 DEBUG } 05:36:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712723786" 05:36:26 INFO [http_cmd]: http status code OK 05:36:26 DEBUG --- status code --- 05:36:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:26 DEBUG --- http response --- 05:36:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:36:26 DEBUG ----- output ----- 05:36:26 DEBUG { 05:36:26 DEBUG "status": "success", 05:36:26 DEBUG "data": { 05:36:26 DEBUG "resultType": "vector", 05:36:26 DEBUG "result": [ 05:36:26 DEBUG { 05:36:26 DEBUG "metric": { 05:36:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:36:26 DEBUG }, 05:36:26 DEBUG "value": [ 05:36:26 DEBUG 05:36:26 DEBUG 05:36:26 DEBUG 05:36:26 DEBUG [...] 05:36:26 DEBUG 05:36:26 DEBUG 05:36:26 DEBUG 05:36:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:36:26 DEBUG }, 05:36:26 DEBUG "value": [ 05:36:26 DEBUG 1712723786, 05:36:26 DEBUG "3.6140285528802547" 05:36:26 DEBUG ] 05:36:26 DEBUG } 05:36:26 DEBUG ] 05:36:26 DEBUG } 05:36:26 DEBUG } 05:36:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712723786" 05:36:26 INFO [http_cmd]: http status code OK 05:36:26 DEBUG --- status code --- 05:36:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:26 DEBUG --- http response --- 05:36:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:36:26 DEBUG ----- output ----- 05:36:26 DEBUG { 05:36:26 DEBUG "status": "success", 05:36:26 DEBUG "data": { 05:36:26 DEBUG "resultType": "vector", 05:36:26 DEBUG "result": [ 05:36:26 DEBUG { 05:36:26 DEBUG "metric": { 05:36:26 DEBUG "pod": "ds-idrepo-0" 05:36:26 DEBUG }, 05:36:26 DEBUG "value": [ 05:36:26 DEBUG 05:36:26 DEBUG 05:36:26 DEBUG 05:36:26 DEBUG [...] 05:36:26 DEBUG 05:36:26 DEBUG 05:36:26 DEBUG 05:36:26 DEBUG "pod": "ds-cts-2" 05:36:26 DEBUG }, 05:36:26 DEBUG "value": [ 05:36:26 DEBUG 1712723786, 05:36:26 DEBUG "0" 05:36:26 DEBUG ] 05:36:26 DEBUG } 05:36:26 DEBUG ] 05:36:26 DEBUG } 05:36:26 DEBUG } 05:36:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723787" 05:36:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723787" 05:36:27 INFO [http_cmd]: http status code OK 05:36:27 DEBUG --- status code --- 05:36:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:27 DEBUG --- http response --- 05:36:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:36:27 DEBUG ----- output ----- 05:36:27 DEBUG { 05:36:27 DEBUG "status": "success", 05:36:27 DEBUG "data": { 05:36:27 DEBUG "resultType": "vector", 05:36:27 DEBUG "result": [ 05:36:27 DEBUG { 05:36:27 DEBUG "metric": { 05:36:27 DEBUG "pod": "ds-idrepo-2" 05:36:27 DEBUG }, 05:36:27 DEBUG "value": [ 05:36:27 DEBUG 05:36:27 DEBUG 05:36:27 DEBUG 05:36:27 DEBUG [...] 05:36:27 DEBUG 05:36:27 DEBUG 05:36:27 DEBUG 05:36:27 DEBUG "pod": "ds-cts-2" 05:36:27 DEBUG }, 05:36:27 DEBUG "value": [ 05:36:27 DEBUG 1712723787, 05:36:27 DEBUG "0.000033333333333333335" 05:36:27 DEBUG ] 05:36:27 DEBUG } 05:36:27 DEBUG ] 05:36:27 DEBUG } 05:36:27 DEBUG } 05:36:27 INFO [http_cmd]: http status code OK 05:36:27 DEBUG --- status code --- 05:36:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:27 DEBUG --- http response --- 05:36:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:36:27 DEBUG ----- output ----- 05:36:27 DEBUG { 05:36:27 DEBUG "status": "success", 05:36:27 DEBUG "data": { 05:36:27 DEBUG "resultType": "vector", 05:36:27 DEBUG "result": [ 05:36:27 DEBUG { 05:36:27 DEBUG "metric": { 05:36:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:36:27 DEBUG }, 05:36:27 DEBUG "value": [ 05:36:27 DEBUG 05:36:27 DEBUG 05:36:27 DEBUG 05:36:27 DEBUG [...] 05:36:27 DEBUG 05:36:27 DEBUG 05:36:27 DEBUG 05:36:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:36:27 DEBUG }, 05:36:27 DEBUG "value": [ 05:36:27 DEBUG 1712723787, 05:36:27 DEBUG "7.373084492968281" 05:36:27 DEBUG ] 05:36:27 DEBUG } 05:36:27 DEBUG ] 05:36:27 DEBUG } 05:36:27 DEBUG } 05:36:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723787" 05:36:27 INFO [http_cmd]: http status code OK 05:36:27 DEBUG --- status code --- 05:36:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:27 DEBUG --- http response --- 05:36:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:36:27 DEBUG ----- output ----- 05:36:27 DEBUG { 05:36:27 DEBUG "status": "success", 05:36:27 DEBUG "data": { 05:36:27 DEBUG "resultType": "vector", 05:36:27 DEBUG "result": [ 05:36:27 DEBUG { 05:36:27 DEBUG "metric": { 05:36:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:36:27 DEBUG }, 05:36:27 DEBUG "value": [ 05:36:27 DEBUG 05:36:27 DEBUG 05:36:27 DEBUG 05:36:27 DEBUG [...] 05:36:27 DEBUG 05:36:27 DEBUG 05:36:27 DEBUG 05:36:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:36:27 DEBUG }, 05:36:27 DEBUG "value": [ 05:36:27 DEBUG 1712723787, 05:36:27 DEBUG "1262.2666666666667" 05:36:27 DEBUG ] 05:36:27 DEBUG } 05:36:27 DEBUG ] 05:36:27 DEBUG } 05:36:27 DEBUG } 05:36:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712723787" 05:36:27 INFO [http_cmd]: http status code OK 05:36:27 DEBUG --- status code --- 05:36:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:27 DEBUG --- http response --- 05:36:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:36:27 DEBUG ----- output ----- 05:36:27 DEBUG { 05:36:27 DEBUG "status": "success", 05:36:27 DEBUG "data": { 05:36:27 DEBUG "resultType": "vector", 05:36:27 DEBUG "result": [ 05:36:27 DEBUG { 05:36:27 DEBUG "metric": { 05:36:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:36:27 DEBUG }, 05:36:27 DEBUG "value": [ 05:36:27 DEBUG 05:36:27 DEBUG 05:36:27 DEBUG 05:36:27 DEBUG [...] 05:36:27 DEBUG 05:36:27 DEBUG 05:36:27 DEBUG 05:36:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:36:27 DEBUG }, 05:36:27 DEBUG "value": [ 05:36:27 DEBUG 1712723787, 05:36:27 DEBUG "0.042" 05:36:27 DEBUG ] 05:36:27 DEBUG } 05:36:27 DEBUG ] 05:36:27 DEBUG } 05:36:27 DEBUG } 05:36:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723787" 05:36:27 INFO [http_cmd]: http status code OK 05:36:27 DEBUG --- status code --- 05:36:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:27 DEBUG --- http response --- 05:36:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:36:27 DEBUG ----- output ----- 05:36:27 DEBUG { 05:36:27 DEBUG "status": "success", 05:36:27 DEBUG "data": { 05:36:27 DEBUG "resultType": "vector", 05:36:27 DEBUG "result": [ 05:36:27 DEBUG { 05:36:27 DEBUG "metric": { 05:36:27 DEBUG "pod": "ds-idrepo-0" 05:36:27 DEBUG }, 05:36:27 DEBUG "value": [ 05:36:27 DEBUG 05:36:27 DEBUG 05:36:27 DEBUG 05:36:27 DEBUG [...] 05:36:27 DEBUG 05:36:27 DEBUG 05:36:27 DEBUG 05:36:27 DEBUG "pod": "ds-idrepo-1" 05:36:27 DEBUG }, 05:36:27 DEBUG "value": [ 05:36:27 DEBUG 1712723787, 05:36:27 DEBUG "0" 05:36:27 DEBUG ] 05:36:27 DEBUG } 05:36:27 DEBUG ] 05:36:27 DEBUG } 05:36:27 DEBUG } 05:36:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723787" 05:36:27 INFO [http_cmd]: http status code OK 05:36:27 DEBUG --- status code --- 05:36:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:27 DEBUG --- http response --- 05:36:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:36:27 DEBUG ----- output ----- 05:36:27 DEBUG { 05:36:27 DEBUG "status": "success", 05:36:27 DEBUG "data": { 05:36:27 DEBUG "resultType": "vector", 05:36:27 DEBUG "result": [ 05:36:27 DEBUG { 05:36:27 DEBUG "metric": { 05:36:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:36:27 DEBUG }, 05:36:27 DEBUG "value": [ 05:36:27 DEBUG 05:36:27 DEBUG 05:36:27 DEBUG 05:36:27 DEBUG [...] 05:36:27 DEBUG 05:36:27 DEBUG 05:36:27 DEBUG 05:36:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:36:27 DEBUG }, 05:36:27 DEBUG "value": [ 05:36:27 DEBUG 1712723787, 05:36:27 DEBUG "1263.6333333333332" 05:36:27 DEBUG ] 05:36:27 DEBUG } 05:36:27 DEBUG ] 05:36:27 DEBUG } 05:36:27 DEBUG } 05:36:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723787" 05:36:27 INFO [http_cmd]: http status code OK 05:36:27 DEBUG --- status code --- 05:36:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:27 DEBUG --- http response --- 05:36:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:36:27 DEBUG ----- output ----- 05:36:27 DEBUG { 05:36:27 DEBUG "status": "success", 05:36:27 DEBUG "data": { 05:36:27 DEBUG "resultType": "vector", 05:36:27 DEBUG "result": [ 05:36:27 DEBUG { 05:36:27 DEBUG "metric": { 05:36:27 DEBUG "pod": "ds-idrepo-0" 05:36:27 DEBUG }, 05:36:27 DEBUG "value": [ 05:36:27 DEBUG 05:36:27 DEBUG 05:36:27 DEBUG 05:36:27 DEBUG [...] 05:36:27 DEBUG 05:36:27 DEBUG 05:36:27 DEBUG 05:36:27 DEBUG "pod": "ds-idrepo-1" 05:36:27 DEBUG }, 05:36:27 DEBUG "value": [ 05:36:27 DEBUG 1712723787, 05:36:27 DEBUG "0" 05:36:27 DEBUG ] 05:36:27 DEBUG } 05:36:27 DEBUG ] 05:36:27 DEBUG } 05:36:27 DEBUG } 05:36:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712723788" 05:36:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723788" 05:36:28 INFO [http_cmd]: http status code OK 05:36:28 DEBUG --- status code --- 05:36:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:28 DEBUG --- http response --- 05:36:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:36:28 DEBUG ----- output ----- 05:36:28 DEBUG { 05:36:28 DEBUG "status": "success", 05:36:28 DEBUG "data": { 05:36:28 DEBUG "resultType": "vector", 05:36:28 DEBUG "result": [ 05:36:28 DEBUG { 05:36:28 DEBUG "metric": { 05:36:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:36:28 DEBUG }, 05:36:28 DEBUG "value": [ 05:36:28 DEBUG 05:36:28 DEBUG 05:36:28 DEBUG 05:36:28 DEBUG [...] 05:36:28 DEBUG 05:36:28 DEBUG 05:36:28 DEBUG 05:36:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:36:28 DEBUG }, 05:36:28 DEBUG "value": [ 05:36:28 DEBUG 1712723788, 05:36:28 DEBUG "2486448128" 05:36:28 DEBUG ] 05:36:28 DEBUG } 05:36:28 DEBUG ] 05:36:28 DEBUG } 05:36:28 DEBUG } 05:36:28 INFO [http_cmd]: http status code OK 05:36:28 DEBUG --- status code --- 05:36:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:28 DEBUG --- http response --- 05:36:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 05:36:28 DEBUG ----- output ----- 05:36:28 DEBUG { 05:36:28 DEBUG "status": "success", 05:36:28 DEBUG "data": { 05:36:28 DEBUG "resultType": "vector", 05:36:28 DEBUG "result": [ 05:36:28 DEBUG { 05:36:28 DEBUG "metric": { 05:36:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:36:28 DEBUG }, 05:36:28 DEBUG "value": [ 05:36:28 DEBUG 05:36:28 DEBUG 05:36:28 DEBUG 05:36:28 DEBUG [...] 05:36:28 DEBUG 05:36:28 DEBUG 05:36:28 DEBUG 05:36:28 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 05:36:28 DEBUG }, 05:36:28 DEBUG "value": [ 05:36:28 DEBUG 1712723788, 05:36:28 DEBUG "0" 05:36:28 DEBUG ] 05:36:28 DEBUG } 05:36:28 DEBUG ] 05:36:28 DEBUG } 05:36:28 DEBUG } 05:36:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723788" 05:36:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723788" 05:36:28 INFO [http_cmd]: http status code OK 05:36:28 DEBUG --- status code --- 05:36:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:28 DEBUG --- http response --- 05:36:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:36:28 DEBUG ----- output ----- 05:36:28 DEBUG { 05:36:28 DEBUG "status": "success", 05:36:28 DEBUG "data": { 05:36:28 DEBUG "resultType": "vector", 05:36:28 DEBUG "result": [ 05:36:28 DEBUG { 05:36:28 DEBUG "metric": { 05:36:28 DEBUG "instance": "10.142.0.25:9100" 05:36:28 DEBUG }, 05:36:28 DEBUG "value": [ 05:36:28 DEBUG 05:36:28 DEBUG 05:36:28 DEBUG 05:36:28 DEBUG [...] 05:36:28 DEBUG 05:36:28 DEBUG 05:36:28 DEBUG 05:36:28 DEBUG "instance": "10.142.0.77:9100" 05:36:28 DEBUG }, 05:36:28 DEBUG "value": [ 05:36:28 DEBUG 1712723788, 05:36:28 DEBUG "37514356.03333333" 05:36:28 DEBUG ] 05:36:28 DEBUG } 05:36:28 DEBUG ] 05:36:28 DEBUG } 05:36:28 DEBUG } 05:36:28 INFO [http_cmd]: http status code OK 05:36:28 DEBUG --- status code --- 05:36:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:28 DEBUG --- http response --- 05:36:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:36:28 DEBUG ----- output ----- 05:36:28 DEBUG { 05:36:28 DEBUG "status": "success", 05:36:28 DEBUG "data": { 05:36:28 DEBUG "resultType": "vector", 05:36:28 DEBUG "result": [ 05:36:28 DEBUG { 05:36:28 DEBUG "metric": { 05:36:28 DEBUG "instance": "10.142.0.25:9100" 05:36:28 DEBUG }, 05:36:28 DEBUG "value": [ 05:36:28 DEBUG 05:36:28 DEBUG 05:36:28 DEBUG 05:36:28 DEBUG [...] 05:36:28 DEBUG 05:36:28 DEBUG 05:36:28 DEBUG 05:36:28 DEBUG "instance": "10.142.0.77:9100" 05:36:28 DEBUG }, 05:36:28 DEBUG "value": [ 05:36:28 DEBUG 1712723788, 05:36:28 DEBUG "0.018666666666666738" 05:36:28 DEBUG ] 05:36:28 DEBUG } 05:36:28 DEBUG ] 05:36:28 DEBUG } 05:36:28 DEBUG } 05:36:43 INFO 05:36:43 INFO [loop_until]: kubectl --namespace=xlou top pods 05:36:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:36:43 INFO [loop_until]: OK (rc = 0) 05:36:43 DEBUG --- stdout --- 05:36:43 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 8994m 8810Mi am-654c4fcbcf-9scrj 8552m 8421Mi am-654c4fcbcf-v5qb5 7418m 8403Mi ds-cts-0 7662m 13159Mi ds-cts-1 7754m 13265Mi ds-cts-2 7228m 13233Mi ds-idrepo-0 9775m 5622Mi ds-idrepo-1 327m 5669Mi ds-idrepo-2 346m 5698Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 5m 2213Mi lodemon-b776b7d88-thrqf 2m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3579m 1670Mi 05:36:43 DEBUG --- stderr --- 05:36:43 DEBUG 05:36:49 INFO 05:36:49 INFO [loop_until]: kubectl --namespace=xlou top node 05:36:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:36:50 INFO [loop_until]: OK (rc = 0) 05:36:50 DEBUG --- stdout --- 05:36:50 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 91m 0% 2305Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8152m 51% 9281Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 82m 0% 3258Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10786m 67% 6524Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9613m 60% 9312Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9953m 62% 9647Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 77m 0% 854Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9446m 59% 6282Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 7478m 47% 13902Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 7295m 45% 13811Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 406m 2% 6369Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7648m 48% 13948Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 374m 2% 6334Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3709m 23% 5342Mi 9% 05:36:50 DEBUG --- stderr --- 05:36:50 DEBUG 05:37:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712723844" 05:37:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723844" 05:37:24 ERROR [http_cmd]: ERROR 05:37:24 INFO [http_cmd]: http status code OK 05:37:24 ERROR --- status code --- 05:37:24 DEBUG --- status code --- 05:37:24 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:37:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:24 ERROR --- http response --- 05:37:24 DEBUG --- http response --- 05:37:24 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:37:24 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:37:24 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:37:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723844" 05:37:24 INFO [http_cmd]: http status code OK 05:37:24 DEBUG --- status code --- 05:37:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:24 DEBUG --- http response --- 05:37:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:37:24 DEBUG ----- output ----- 05:37:24 DEBUG { 05:37:24 DEBUG "status": "success", 05:37:24 DEBUG "data": { 05:37:24 DEBUG "resultType": "vector", 05:37:24 DEBUG "result": [ 05:37:24 DEBUG { 05:37:24 DEBUG "metric": { 05:37:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:37:24 DEBUG }, 05:37:24 DEBUG "value": [ 05:37:24 DEBUG 05:37:24 DEBUG 05:37:24 DEBUG 05:37:24 DEBUG [...] 05:37:24 DEBUG 05:37:24 DEBUG 05:37:24 DEBUG 05:37:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:37:24 DEBUG }, 05:37:24 DEBUG "value": [ 05:37:24 DEBUG 1712723844, 05:37:24 DEBUG "0" 05:37:24 DEBUG ] 05:37:24 DEBUG } 05:37:24 DEBUG ] 05:37:24 DEBUG } 05:37:24 DEBUG } 05:37:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723844" 05:37:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723844" 05:37:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723844" 05:37:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723844" 05:37:24 INFO [http_cmd]: http status code OK 05:37:24 INFO [http_cmd]: http status code OK 05:37:24 DEBUG --- status code --- 05:37:24 DEBUG --- status code --- 05:37:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:24 DEBUG --- http response --- 05:37:24 DEBUG --- http response --- 05:37:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:37:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:37:24 DEBUG ----- output ----- 05:37:24 DEBUG ----- output ----- 05:37:24 DEBUG { 05:37:24 DEBUG { 05:37:24 DEBUG "status": "success", 05:37:24 DEBUG "status": "success", 05:37:24 DEBUG "data": { 05:37:24 DEBUG "data": { 05:37:24 DEBUG "resultType": "vector", 05:37:24 DEBUG "resultType": "vector", 05:37:24 DEBUG "result": [ 05:37:24 DEBUG "result": [ 05:37:24 DEBUG { 05:37:24 DEBUG { 05:37:24 DEBUG "metric": { 05:37:24 DEBUG "metric": { 05:37:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:37:24 DEBUG "instance": "10.142.0.25:9100" 05:37:24 DEBUG }, 05:37:24 DEBUG }, 05:37:24 DEBUG "value": [ 05:37:24 DEBUG "value": [ 05:37:24 DEBUG 05:37:24 DEBUG 05:37:24 DEBUG 05:37:24 DEBUG 05:37:24 DEBUG 05:37:24 DEBUG 05:37:24 DEBUG [...] 05:37:24 DEBUG [...] 05:37:24 DEBUG 05:37:24 DEBUG 05:37:24 DEBUG 05:37:24 DEBUG 05:37:24 DEBUG 05:37:24 DEBUG 05:37:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:37:24 DEBUG "instance": "10.142.0.77:9100" 05:37:24 DEBUG }, 05:37:24 DEBUG }, 05:37:24 DEBUG "value": [ 05:37:24 DEBUG "value": [ 05:37:24 DEBUG 1712723844, 05:37:24 DEBUG 1712723844, 05:37:24 DEBUG "1255.4666666666667" 05:37:24 DEBUG "301615191040" 05:37:24 DEBUG ] 05:37:24 DEBUG ] 05:37:24 DEBUG } 05:37:24 DEBUG } 05:37:24 DEBUG ] 05:37:24 DEBUG ] 05:37:24 DEBUG } 05:37:24 DEBUG } 05:37:24 DEBUG } 05:37:24 DEBUG } 05:37:24 INFO [http_cmd]: http status code OK 05:37:24 INFO [http_cmd]: http status code OK 05:37:24 DEBUG --- status code --- 05:37:24 DEBUG --- status code --- 05:37:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:24 DEBUG --- http response --- 05:37:24 DEBUG --- http response --- 05:37:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:37:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:37:24 DEBUG ----- output ----- 05:37:24 DEBUG ----- output ----- 05:37:24 DEBUG { 05:37:24 DEBUG { 05:37:24 DEBUG "status": "success", 05:37:24 DEBUG "status": "success", 05:37:24 DEBUG "data": { 05:37:24 DEBUG "data": { 05:37:24 DEBUG "resultType": "vector", 05:37:24 DEBUG "resultType": "vector", 05:37:24 DEBUG "result": [ 05:37:24 DEBUG { 05:37:24 DEBUG "metric": { 05:37:24 DEBUG "pod": "ds-idrepo-0" 05:37:24 DEBUG }, 05:37:24 DEBUG "value": [ 05:37:24 DEBUG 05:37:24 DEBUG 05:37:24 DEBUG 05:37:24 DEBUG [...] 05:37:24 DEBUG 05:37:24 DEBUG 05:37:24 DEBUG 05:37:24 DEBUG "pod": "ds-cts-2" 05:37:24 DEBUG }, 05:37:24 DEBUG "value": [ 05:37:24 DEBUG 1712723844, 05:37:24 DEBUG "0" 05:37:24 DEBUG ] 05:37:24 DEBUG } 05:37:24 DEBUG ] 05:37:24 DEBUG } 05:37:24 DEBUG } 05:37:24 DEBUG "result": [ 05:37:24 DEBUG { 05:37:24 DEBUG "metric": { 05:37:24 DEBUG "pod": "ds-idrepo-0" 05:37:24 DEBUG }, 05:37:24 DEBUG "value": [ 05:37:24 DEBUG 05:37:24 DEBUG 05:37:24 DEBUG 05:37:24 DEBUG [...] 05:37:24 DEBUG 05:37:24 DEBUG 05:37:24 DEBUG 05:37:24 DEBUG "pod": "ds-cts-2" 05:37:24 DEBUG }, 05:37:24 DEBUG "value": [ 05:37:24 DEBUG 1712723844, 05:37:24 DEBUG "29.966666666666665" 05:37:24 DEBUG ] 05:37:24 DEBUG } 05:37:24 DEBUG ] 05:37:24 DEBUG } 05:37:24 DEBUG } 05:37:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723844" 05:37:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723844" 05:37:24 INFO [http_cmd]: http status code OK 05:37:24 INFO [http_cmd]: http status code OK 05:37:24 DEBUG --- status code --- 05:37:24 DEBUG --- status code --- 05:37:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:24 DEBUG --- http response --- 05:37:24 DEBUG --- http response --- 05:37:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:37:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:37:24 DEBUG ----- output ----- 05:37:24 DEBUG ----- output ----- 05:37:24 DEBUG { 05:37:24 DEBUG { 05:37:24 DEBUG "status": "success", 05:37:24 DEBUG "status": "success", 05:37:24 DEBUG "data": { 05:37:24 DEBUG "data": { 05:37:24 DEBUG "resultType": "vector", 05:37:24 DEBUG "resultType": "vector", 05:37:24 DEBUG "result": [ 05:37:24 DEBUG "result": [ 05:37:24 DEBUG { 05:37:24 DEBUG { 05:37:24 DEBUG "metric": { 05:37:24 DEBUG "metric": { 05:37:24 DEBUG "instance": "10.142.0.25:9100" 05:37:24 DEBUG }, 05:37:24 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:37:24 DEBUG "value": [ 05:37:24 DEBUG }, 05:37:24 DEBUG 05:37:24 DEBUG "value": [ 05:37:24 DEBUG 05:37:24 DEBUG 05:37:24 DEBUG 05:37:24 DEBUG 05:37:24 DEBUG [...] 05:37:24 DEBUG 05:37:24 DEBUG 05:37:24 DEBUG [...] 05:37:24 DEBUG 05:37:24 DEBUG 05:37:24 DEBUG 05:37:24 DEBUG "instance": "10.142.0.77:9100" 05:37:24 DEBUG 05:37:24 DEBUG }, 05:37:24 DEBUG 05:37:24 DEBUG "value": [ 05:37:24 DEBUG "pod": "ds-idrepo-2" 05:37:24 DEBUG 1712723844, 05:37:24 DEBUG }, 05:37:24 DEBUG "37446827.3" 05:37:24 DEBUG "value": [ 05:37:24 DEBUG ] 05:37:24 DEBUG 1712723844, 05:37:24 DEBUG } 05:37:24 DEBUG "0" 05:37:24 DEBUG ] 05:37:24 DEBUG ] 05:37:24 DEBUG } 05:37:24 DEBUG } 05:37:24 DEBUG } 05:37:24 DEBUG ] 05:37:24 DEBUG } 05:37:24 DEBUG } 05:37:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723845" 05:37:25 INFO [http_cmd]: http status code OK 05:37:25 DEBUG --- status code --- 05:37:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:25 DEBUG --- http response --- 05:37:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:37:25 DEBUG ----- output ----- 05:37:25 DEBUG { 05:37:25 DEBUG "status": "success", 05:37:25 DEBUG "data": { 05:37:25 DEBUG "resultType": "vector", 05:37:25 DEBUG "result": [ 05:37:25 DEBUG { 05:37:25 DEBUG "metric": { 05:37:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:37:25 DEBUG }, 05:37:25 DEBUG "value": [ 05:37:25 DEBUG 05:37:25 DEBUG 05:37:25 DEBUG 05:37:25 DEBUG [...] 05:37:25 DEBUG 05:37:25 DEBUG 05:37:25 DEBUG 05:37:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:37:25 DEBUG }, 05:37:25 DEBUG "value": [ 05:37:25 DEBUG 1712723845, 05:37:25 DEBUG "227235" 05:37:25 DEBUG ] 05:37:25 DEBUG } 05:37:25 DEBUG ] 05:37:25 DEBUG } 05:37:25 DEBUG } 05:37:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723845" 05:37:25 INFO [http_cmd]: http status code OK 05:37:25 DEBUG --- status code --- 05:37:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:25 DEBUG --- http response --- 05:37:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:37:25 DEBUG ----- output ----- 05:37:25 DEBUG { 05:37:25 DEBUG "status": "success", 05:37:25 DEBUG "data": { 05:37:25 DEBUG "resultType": "vector", 05:37:25 DEBUG "result": [ 05:37:25 DEBUG { 05:37:25 DEBUG "metric": { 05:37:25 DEBUG "pod": "ds-idrepo-0" 05:37:25 DEBUG }, 05:37:25 DEBUG "value": [ 05:37:25 DEBUG 05:37:25 DEBUG 05:37:25 DEBUG 05:37:25 DEBUG [...] 05:37:25 DEBUG 05:37:25 DEBUG 05:37:25 DEBUG 05:37:25 DEBUG "pod": "ds-idrepo-1" 05:37:25 DEBUG }, 05:37:25 DEBUG "value": [ 05:37:25 DEBUG 1712723845, 05:37:25 DEBUG "0" 05:37:25 DEBUG ] 05:37:25 DEBUG } 05:37:25 DEBUG ] 05:37:25 DEBUG } 05:37:25 DEBUG } 05:37:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723845" 05:37:25 INFO [http_cmd]: http status code OK 05:37:25 DEBUG --- status code --- 05:37:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:25 DEBUG --- http response --- 05:37:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:37:25 DEBUG ----- output ----- 05:37:25 DEBUG { 05:37:25 DEBUG "status": "success", 05:37:25 DEBUG "data": { 05:37:25 DEBUG "resultType": "vector", 05:37:25 DEBUG "result": [ 05:37:25 DEBUG { 05:37:25 DEBUG "metric": { 05:37:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:37:25 DEBUG }, 05:37:25 DEBUG "value": [ 05:37:25 DEBUG 05:37:25 DEBUG 05:37:25 DEBUG 05:37:25 DEBUG [...] 05:37:25 DEBUG 05:37:25 DEBUG 05:37:25 DEBUG 05:37:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:37:25 DEBUG }, 05:37:25 DEBUG "value": [ 05:37:25 DEBUG 1712723845, 05:37:25 DEBUG "1255.6333333333332" 05:37:25 DEBUG ] 05:37:25 DEBUG } 05:37:25 DEBUG ] 05:37:25 DEBUG } 05:37:25 DEBUG } 05:37:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723845" 05:37:25 INFO [http_cmd]: http status code OK 05:37:25 DEBUG --- status code --- 05:37:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:25 DEBUG --- http response --- 05:37:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:37:25 DEBUG ----- output ----- 05:37:25 DEBUG { 05:37:25 DEBUG "status": "success", 05:37:25 DEBUG "data": { 05:37:25 DEBUG "resultType": "vector", 05:37:25 DEBUG "result": [ 05:37:25 DEBUG { 05:37:25 DEBUG "metric": { 05:37:25 DEBUG "instance": "10.142.0.25:9100" 05:37:25 DEBUG }, 05:37:25 DEBUG "value": [ 05:37:25 DEBUG 05:37:25 DEBUG 05:37:25 DEBUG 05:37:25 DEBUG [...] 05:37:25 DEBUG 05:37:25 DEBUG 05:37:25 DEBUG 05:37:25 DEBUG "instance": "10.142.0.77:9100" 05:37:25 DEBUG }, 05:37:25 DEBUG "value": [ 05:37:25 DEBUG 1712723845, 05:37:25 DEBUG "0.14643333333333394" 05:37:25 DEBUG ] 05:37:25 DEBUG } 05:37:25 DEBUG ] 05:37:25 DEBUG } 05:37:25 DEBUG } 05:37:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723846" 05:37:26 INFO [http_cmd]: http status code OK 05:37:26 DEBUG --- status code --- 05:37:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:26 DEBUG --- http response --- 05:37:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:37:26 DEBUG ----- output ----- 05:37:26 DEBUG { 05:37:26 DEBUG "status": "success", 05:37:26 DEBUG "data": { 05:37:26 DEBUG "resultType": "vector", 05:37:26 DEBUG "result": [ 05:37:26 DEBUG { 05:37:26 DEBUG "metric": { 05:37:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:37:26 DEBUG }, 05:37:26 DEBUG "value": [ 05:37:26 DEBUG 05:37:26 DEBUG 05:37:26 DEBUG 05:37:26 DEBUG [...] 05:37:26 DEBUG 05:37:26 DEBUG 05:37:26 DEBUG 05:37:26 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 05:37:26 DEBUG }, 05:37:26 DEBUG "value": [ 05:37:26 DEBUG 1712723846, 05:37:26 DEBUG "0" 05:37:26 DEBUG ] 05:37:26 DEBUG } 05:37:26 DEBUG ] 05:37:26 DEBUG } 05:37:26 DEBUG } 05:37:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712723846" 05:37:26 INFO [http_cmd]: http status code OK 05:37:26 DEBUG --- status code --- 05:37:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:26 DEBUG --- http response --- 05:37:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:37:26 DEBUG ----- output ----- 05:37:26 DEBUG { 05:37:26 DEBUG "status": "success", 05:37:26 DEBUG "data": { 05:37:26 DEBUG "resultType": "vector", 05:37:26 DEBUG "result": [ 05:37:26 DEBUG { 05:37:26 DEBUG "metric": { 05:37:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:37:26 DEBUG }, 05:37:26 DEBUG "value": [ 05:37:26 DEBUG 05:37:26 DEBUG 05:37:26 DEBUG 05:37:26 DEBUG [...] 05:37:26 DEBUG 05:37:26 DEBUG 05:37:26 DEBUG 05:37:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:37:26 DEBUG }, 05:37:26 DEBUG "value": [ 05:37:26 DEBUG 1712723846, 05:37:26 DEBUG "3.6233582409121508" 05:37:26 DEBUG ] 05:37:26 DEBUG } 05:37:26 DEBUG ] 05:37:26 DEBUG } 05:37:26 DEBUG } 05:37:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712723846" 05:37:26 INFO [http_cmd]: http status code OK 05:37:26 DEBUG --- status code --- 05:37:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:26 DEBUG --- http response --- 05:37:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:37:26 DEBUG ----- output ----- 05:37:26 DEBUG { 05:37:26 DEBUG "status": "success", 05:37:26 DEBUG "data": { 05:37:26 DEBUG "resultType": "vector", 05:37:26 DEBUG "result": [ 05:37:26 DEBUG { 05:37:26 DEBUG "metric": { 05:37:26 DEBUG "pod": "ds-idrepo-0" 05:37:26 DEBUG }, 05:37:26 DEBUG "value": [ 05:37:26 DEBUG 05:37:26 DEBUG 05:37:26 DEBUG 05:37:26 DEBUG [...] 05:37:26 DEBUG 05:37:26 DEBUG 05:37:26 DEBUG 05:37:26 DEBUG "pod": "ds-cts-2" 05:37:26 DEBUG }, 05:37:26 DEBUG "value": [ 05:37:26 DEBUG 1712723846, 05:37:26 DEBUG "0" 05:37:26 DEBUG ] 05:37:26 DEBUG } 05:37:26 DEBUG ] 05:37:26 DEBUG } 05:37:26 DEBUG } 05:37:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723847" 05:37:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723847" 05:37:27 INFO [http_cmd]: http status code OK 05:37:27 DEBUG --- status code --- 05:37:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:27 DEBUG --- http response --- 05:37:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:37:27 DEBUG ----- output ----- 05:37:27 DEBUG { 05:37:27 DEBUG "status": "success", 05:37:27 DEBUG "data": { 05:37:27 DEBUG "resultType": "vector", 05:37:27 DEBUG "result": [ 05:37:27 DEBUG { 05:37:27 DEBUG "metric": { 05:37:27 DEBUG "pod": "ds-idrepo-2" 05:37:27 DEBUG }, 05:37:27 DEBUG "value": [ 05:37:27 DEBUG 05:37:27 DEBUG 05:37:27 DEBUG 05:37:27 DEBUG [...] 05:37:27 DEBUG 05:37:27 DEBUG 05:37:27 DEBUG 05:37:27 DEBUG "pod": "ds-cts-2" 05:37:27 DEBUG }, 05:37:27 DEBUG "value": [ 05:37:27 DEBUG 1712723847, 05:37:27 DEBUG "0.00006666666666666667" 05:37:27 DEBUG ] 05:37:27 DEBUG } 05:37:27 DEBUG ] 05:37:27 DEBUG } 05:37:27 DEBUG } 05:37:27 INFO [http_cmd]: http status code OK 05:37:27 DEBUG --- status code --- 05:37:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:27 DEBUG --- http response --- 05:37:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:37:27 DEBUG ----- output ----- 05:37:27 DEBUG { 05:37:27 DEBUG "status": "success", 05:37:27 DEBUG "data": { 05:37:27 DEBUG "resultType": "vector", 05:37:27 DEBUG "result": [ 05:37:27 DEBUG { 05:37:27 DEBUG "metric": { 05:37:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:37:27 DEBUG }, 05:37:27 DEBUG "value": [ 05:37:27 DEBUG 05:37:27 DEBUG 05:37:27 DEBUG 05:37:27 DEBUG [...] 05:37:27 DEBUG 05:37:27 DEBUG 05:37:27 DEBUG 05:37:27 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 05:37:27 DEBUG }, 05:37:27 DEBUG "value": [ 05:37:27 DEBUG 1712723847, 05:37:27 DEBUG "0" 05:37:27 DEBUG ] 05:37:27 DEBUG } 05:37:27 DEBUG ] 05:37:27 DEBUG } 05:37:27 DEBUG } 05:37:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723847" 05:37:27 INFO [http_cmd]: http status code OK 05:37:27 DEBUG --- status code --- 05:37:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:27 DEBUG --- http response --- 05:37:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:37:27 DEBUG ----- output ----- 05:37:27 DEBUG { 05:37:27 DEBUG "status": "success", 05:37:27 DEBUG "data": { 05:37:27 DEBUG "resultType": "vector", 05:37:27 DEBUG "result": [ 05:37:27 DEBUG { 05:37:27 DEBUG "metric": { 05:37:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:37:27 DEBUG }, 05:37:27 DEBUG "value": [ 05:37:27 DEBUG 05:37:27 DEBUG 05:37:27 DEBUG 05:37:27 DEBUG [...] 05:37:27 DEBUG 05:37:27 DEBUG 05:37:27 DEBUG 05:37:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:37:27 DEBUG }, 05:37:27 DEBUG "value": [ 05:37:27 DEBUG 1712723847, 05:37:27 DEBUG "1255.4" 05:37:27 DEBUG ] 05:37:27 DEBUG } 05:37:27 DEBUG ] 05:37:27 DEBUG } 05:37:27 DEBUG } 05:37:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712723847" 05:37:27 INFO [http_cmd]: http status code OK 05:37:27 DEBUG --- status code --- 05:37:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:27 DEBUG --- http response --- 05:37:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:37:27 DEBUG ----- output ----- 05:37:27 DEBUG { 05:37:27 DEBUG "status": "success", 05:37:27 DEBUG "data": { 05:37:27 DEBUG "resultType": "vector", 05:37:27 DEBUG "result": [ 05:37:27 DEBUG { 05:37:27 DEBUG "metric": { 05:37:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:37:27 DEBUG }, 05:37:27 DEBUG "value": [ 05:37:27 DEBUG 05:37:27 DEBUG 05:37:27 DEBUG 05:37:27 DEBUG [...] 05:37:27 DEBUG 05:37:27 DEBUG 05:37:27 DEBUG 05:37:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:37:27 DEBUG }, 05:37:27 DEBUG "value": [ 05:37:27 DEBUG 1712723847, 05:37:27 DEBUG "0.042" 05:37:27 DEBUG ] 05:37:27 DEBUG } 05:37:27 DEBUG ] 05:37:27 DEBUG } 05:37:27 DEBUG } 05:37:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723847" 05:37:27 INFO [http_cmd]: http status code OK 05:37:27 DEBUG --- status code --- 05:37:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:27 DEBUG --- http response --- 05:37:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:37:27 DEBUG ----- output ----- 05:37:27 DEBUG { 05:37:27 DEBUG "status": "success", 05:37:27 DEBUG "data": { 05:37:27 DEBUG "resultType": "vector", 05:37:27 DEBUG "result": [ 05:37:27 DEBUG { 05:37:27 DEBUG "metric": { 05:37:27 DEBUG "pod": "ds-idrepo-0" 05:37:27 DEBUG }, 05:37:27 DEBUG "value": [ 05:37:27 DEBUG 05:37:27 DEBUG 05:37:27 DEBUG 05:37:27 DEBUG [...] 05:37:27 DEBUG 05:37:27 DEBUG 05:37:27 DEBUG 05:37:27 DEBUG "pod": "ds-idrepo-1" 05:37:27 DEBUG }, 05:37:27 DEBUG "value": [ 05:37:27 DEBUG 1712723847, 05:37:27 DEBUG "0" 05:37:27 DEBUG ] 05:37:27 DEBUG } 05:37:27 DEBUG ] 05:37:27 DEBUG } 05:37:27 DEBUG } 05:37:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723847" 05:37:27 INFO [http_cmd]: http status code OK 05:37:27 DEBUG --- status code --- 05:37:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:27 DEBUG --- http response --- 05:37:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:37:27 DEBUG ----- output ----- 05:37:27 DEBUG { 05:37:27 DEBUG "status": "success", 05:37:27 DEBUG "data": { 05:37:27 DEBUG "resultType": "vector", 05:37:27 DEBUG "result": [ 05:37:27 DEBUG { 05:37:27 DEBUG "metric": { 05:37:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:37:27 DEBUG }, 05:37:27 DEBUG "value": [ 05:37:27 DEBUG 05:37:27 DEBUG 05:37:27 DEBUG 05:37:27 DEBUG [...] 05:37:27 DEBUG 05:37:27 DEBUG 05:37:27 DEBUG 05:37:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:37:27 DEBUG }, 05:37:27 DEBUG "value": [ 05:37:27 DEBUG 1712723847, 05:37:27 DEBUG "1263.1666666666667" 05:37:27 DEBUG ] 05:37:27 DEBUG } 05:37:27 DEBUG ] 05:37:27 DEBUG } 05:37:27 DEBUG } 05:37:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723848" 05:37:28 INFO [http_cmd]: http status code OK 05:37:28 DEBUG --- status code --- 05:37:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:28 DEBUG --- http response --- 05:37:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:37:28 DEBUG ----- output ----- 05:37:28 DEBUG { 05:37:28 DEBUG "status": "success", 05:37:28 DEBUG "data": { 05:37:28 DEBUG "resultType": "vector", 05:37:28 DEBUG "result": [ 05:37:28 DEBUG { 05:37:28 DEBUG "metric": { 05:37:28 DEBUG "pod": "ds-idrepo-0" 05:37:28 DEBUG }, 05:37:28 DEBUG "value": [ 05:37:28 DEBUG 05:37:28 DEBUG 05:37:28 DEBUG 05:37:28 DEBUG [...] 05:37:28 DEBUG 05:37:28 DEBUG 05:37:28 DEBUG 05:37:28 DEBUG "pod": "ds-idrepo-1" 05:37:28 DEBUG }, 05:37:28 DEBUG "value": [ 05:37:28 DEBUG 1712723848, 05:37:28 DEBUG "0" 05:37:28 DEBUG ] 05:37:28 DEBUG } 05:37:28 DEBUG ] 05:37:28 DEBUG } 05:37:28 DEBUG } 05:37:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712723848" 05:37:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723848" 05:37:28 INFO [http_cmd]: http status code OK 05:37:28 DEBUG --- status code --- 05:37:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:28 DEBUG --- http response --- 05:37:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 05:37:28 DEBUG ----- output ----- 05:37:28 DEBUG { 05:37:28 DEBUG "status": "success", 05:37:28 DEBUG "data": { 05:37:28 DEBUG "resultType": "vector", 05:37:28 DEBUG "result": [ 05:37:28 DEBUG { 05:37:28 DEBUG "metric": { 05:37:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:37:28 DEBUG }, 05:37:28 DEBUG "value": [ 05:37:28 DEBUG 05:37:28 DEBUG 05:37:28 DEBUG 05:37:28 DEBUG [...] 05:37:28 DEBUG 05:37:28 DEBUG 05:37:28 DEBUG 05:37:28 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 05:37:28 DEBUG }, 05:37:28 DEBUG "value": [ 05:37:28 DEBUG 1712723848, 05:37:28 DEBUG "0" 05:37:28 DEBUG ] 05:37:28 DEBUG } 05:37:28 DEBUG ] 05:37:28 DEBUG } 05:37:28 DEBUG } 05:37:28 INFO [http_cmd]: http status code OK 05:37:28 DEBUG --- status code --- 05:37:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:28 DEBUG --- http response --- 05:37:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:37:28 DEBUG ----- output ----- 05:37:28 DEBUG { 05:37:28 DEBUG "status": "success", 05:37:28 DEBUG "data": { 05:37:28 DEBUG "resultType": "vector", 05:37:28 DEBUG "result": [ 05:37:28 DEBUG { 05:37:28 DEBUG "metric": { 05:37:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:37:28 DEBUG }, 05:37:28 DEBUG "value": [ 05:37:28 DEBUG 05:37:28 DEBUG 05:37:28 DEBUG 05:37:28 DEBUG [...] 05:37:28 DEBUG 05:37:28 DEBUG 05:37:28 DEBUG 05:37:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:37:28 DEBUG }, 05:37:28 DEBUG "value": [ 05:37:28 DEBUG 1712723848, 05:37:28 DEBUG "2490343424" 05:37:28 DEBUG ] 05:37:28 DEBUG } 05:37:28 DEBUG ] 05:37:28 DEBUG } 05:37:28 DEBUG } 05:37:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723848" 05:37:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723848" 05:37:28 INFO [http_cmd]: http status code OK 05:37:28 DEBUG --- status code --- 05:37:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:28 DEBUG --- http response --- 05:37:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:37:28 DEBUG ----- output ----- 05:37:28 DEBUG { 05:37:28 DEBUG "status": "success", 05:37:28 DEBUG "data": { 05:37:28 DEBUG "resultType": "vector", 05:37:28 DEBUG "result": [ 05:37:28 DEBUG { 05:37:28 DEBUG "metric": { 05:37:28 DEBUG "instance": "10.142.0.25:9100" 05:37:28 DEBUG }, 05:37:28 DEBUG "value": [ 05:37:28 DEBUG 05:37:28 DEBUG 05:37:28 DEBUG 05:37:28 DEBUG [...] 05:37:28 DEBUG 05:37:28 DEBUG 05:37:28 DEBUG 05:37:28 DEBUG "instance": "10.142.0.77:9100" 05:37:28 DEBUG }, 05:37:28 DEBUG "value": [ 05:37:28 DEBUG 1712723848, 05:37:28 DEBUG "37448091.43333333" 05:37:28 DEBUG ] 05:37:28 DEBUG } 05:37:28 DEBUG ] 05:37:28 DEBUG } 05:37:28 DEBUG } 05:37:28 INFO [http_cmd]: http status code OK 05:37:28 DEBUG --- status code --- 05:37:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:28 DEBUG --- http response --- 05:37:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:37:28 DEBUG ----- output ----- 05:37:28 DEBUG { 05:37:28 DEBUG "status": "success", 05:37:28 DEBUG "data": { 05:37:28 DEBUG "resultType": "vector", 05:37:28 DEBUG "result": [ 05:37:28 DEBUG { 05:37:28 DEBUG "metric": { 05:37:28 DEBUG "instance": "10.142.0.25:9100" 05:37:28 DEBUG }, 05:37:28 DEBUG "value": [ 05:37:28 DEBUG 05:37:28 DEBUG 05:37:28 DEBUG 05:37:28 DEBUG [...] 05:37:28 DEBUG 05:37:28 DEBUG 05:37:28 DEBUG 05:37:28 DEBUG "instance": "10.142.0.77:9100" 05:37:28 DEBUG }, 05:37:28 DEBUG "value": [ 05:37:28 DEBUG 1712723848, 05:37:28 DEBUG "0.04866666666666651" 05:37:28 DEBUG ] 05:37:28 DEBUG } 05:37:28 DEBUG ] 05:37:28 DEBUG } 05:37:28 DEBUG } 05:37:43 INFO 05:37:43 INFO [loop_until]: kubectl --namespace=xlou top pods 05:37:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:37:43 INFO [loop_until]: OK (rc = 0) 05:37:43 DEBUG --- stdout --- 05:37:43 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 9075m 8806Mi am-654c4fcbcf-9scrj 8417m 8422Mi am-654c4fcbcf-v5qb5 7280m 8399Mi ds-cts-0 7151m 12708Mi ds-cts-1 7391m 12668Mi ds-cts-2 7052m 12382Mi ds-idrepo-0 9781m 5622Mi ds-idrepo-1 375m 5669Mi ds-idrepo-2 389m 5698Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 7m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3587m 1670Mi 05:37:43 DEBUG --- stderr --- 05:37:43 DEBUG 05:37:50 INFO 05:37:50 INFO [loop_until]: kubectl --namespace=xlou top node 05:37:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:37:50 INFO [loop_until]: OK (rc = 0) 05:37:50 DEBUG --- stdout --- 05:37:50 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 93m 0% 2307Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8217m 51% 9280Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 80m 0% 3258Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10445m 65% 6510Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9573m 60% 9311Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9591m 60% 9646Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 75m 0% 852Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9790m 61% 6284Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 7251m 45% 13363Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 6753m 42% 13948Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 425m 2% 6371Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7412m 46% 13379Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 378m 2% 6333Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3593m 22% 5345Mi 9% 05:37:50 DEBUG --- stderr --- 05:37:50 DEBUG 05:38:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723904" 05:38:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712723904" 05:38:24 INFO [http_cmd]: http status code OK 05:38:24 ERROR [http_cmd]: ERROR 05:38:24 DEBUG --- status code --- 05:38:24 ERROR --- status code --- 05:38:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:24 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:38:24 DEBUG --- http response --- 05:38:24 ERROR --- http response --- 05:38:24 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:38:24 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:38:24 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:38:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723904" 05:38:24 INFO [http_cmd]: http status code OK 05:38:24 DEBUG --- status code --- 05:38:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:24 DEBUG --- http response --- 05:38:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:38:24 DEBUG ----- output ----- 05:38:24 DEBUG { 05:38:24 DEBUG "status": "success", 05:38:24 DEBUG "data": { 05:38:24 DEBUG "resultType": "vector", 05:38:24 DEBUG "result": [ 05:38:24 DEBUG { 05:38:24 DEBUG "metric": { 05:38:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:38:24 DEBUG }, 05:38:24 DEBUG "value": [ 05:38:24 DEBUG 05:38:24 DEBUG 05:38:24 DEBUG 05:38:24 DEBUG [...] 05:38:24 DEBUG 05:38:24 DEBUG 05:38:24 DEBUG 05:38:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:38:24 DEBUG }, 05:38:24 DEBUG "value": [ 05:38:24 DEBUG 1712723904, 05:38:24 DEBUG "0" 05:38:24 DEBUG ] 05:38:24 DEBUG } 05:38:24 DEBUG ] 05:38:24 DEBUG } 05:38:24 DEBUG } 05:38:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723904" 05:38:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723904" 05:38:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723904" 05:38:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723904" 05:38:24 INFO [http_cmd]: http status code OK 05:38:24 INFO [http_cmd]: http status code OK 05:38:24 DEBUG --- status code --- 05:38:24 DEBUG --- status code --- 05:38:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:24 DEBUG --- http response --- 05:38:24 DEBUG --- http response --- 05:38:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:38:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:38:24 DEBUG ----- output ----- 05:38:24 INFO [http_cmd]: http status code OK 05:38:24 DEBUG --- status code --- 05:38:24 DEBUG { 05:38:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:24 DEBUG ----- output ----- 05:38:24 DEBUG "status": "success", 05:38:24 DEBUG --- http response --- 05:38:24 DEBUG { 05:38:24 DEBUG "data": { 05:38:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:38:24 INFO [http_cmd]: http status code OK 05:38:24 DEBUG "status": "success", 05:38:24 DEBUG "resultType": "vector", 05:38:24 DEBUG ----- output ----- 05:38:24 DEBUG --- status code --- 05:38:24 DEBUG "data": { 05:38:24 DEBUG "resultType": "vector", 05:38:24 DEBUG "result": [ 05:38:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:24 DEBUG { 05:38:24 DEBUG { 05:38:24 DEBUG "result": [ 05:38:24 DEBUG --- http response --- 05:38:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:38:24 DEBUG "metric": { 05:38:24 DEBUG { 05:38:24 DEBUG ----- output ----- 05:38:24 DEBUG { 05:38:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:38:24 DEBUG }, 05:38:24 DEBUG "status": "success", 05:38:24 DEBUG "data": { 05:38:24 DEBUG "value": [ 05:38:24 DEBUG 05:38:24 DEBUG "resultType": "vector", 05:38:24 DEBUG "result": [ 05:38:24 DEBUG "metric": { 05:38:24 DEBUG 05:38:24 DEBUG { 05:38:24 DEBUG "metric": { 05:38:24 DEBUG "pod": "ds-idrepo-0" 05:38:24 DEBUG 05:38:24 DEBUG "pod": "ds-idrepo-0" 05:38:24 DEBUG "status": "success", 05:38:24 DEBUG }, 05:38:24 DEBUG [...] 05:38:24 DEBUG }, 05:38:24 DEBUG "data": { 05:38:24 DEBUG "value": [ 05:38:24 DEBUG 05:38:24 DEBUG "value": [ 05:38:24 DEBUG "resultType": "vector", 05:38:24 DEBUG 05:38:24 DEBUG 05:38:24 DEBUG 05:38:24 DEBUG "result": [ 05:38:24 DEBUG 05:38:24 DEBUG 05:38:24 DEBUG 05:38:24 DEBUG { 05:38:24 DEBUG "metric": { 05:38:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:38:24 DEBUG 05:38:24 DEBUG 05:38:24 DEBUG "instance": "10.142.0.25:9100" 05:38:24 DEBUG }, 05:38:24 DEBUG [...] 05:38:24 DEBUG [...] 05:38:24 DEBUG }, 05:38:24 DEBUG "value": [ 05:38:24 DEBUG 1712723904, 05:38:24 DEBUG 05:38:24 DEBUG "value": [ 05:38:24 DEBUG 05:38:24 DEBUG "1251.8666666666666" 05:38:24 DEBUG 05:38:24 DEBUG 05:38:24 DEBUG 05:38:24 DEBUG ] 05:38:24 DEBUG 05:38:24 DEBUG 05:38:24 DEBUG 05:38:24 DEBUG } 05:38:24 DEBUG "pod": "ds-cts-2" 05:38:24 DEBUG 05:38:24 DEBUG [...] 05:38:24 DEBUG ] 05:38:24 DEBUG } 05:38:24 DEBUG } 05:38:24 DEBUG 05:38:24 DEBUG "pod": "ds-cts-2" 05:38:24 DEBUG }, 05:38:24 DEBUG "value": [ 05:38:24 DEBUG 1712723904, 05:38:24 DEBUG }, 05:38:24 DEBUG 05:38:24 DEBUG 05:38:24 DEBUG "value": [ 05:38:24 DEBUG "29.933333333333334" 05:38:24 DEBUG ] 05:38:24 DEBUG 1712723904, 05:38:24 DEBUG } 05:38:24 DEBUG ] 05:38:24 DEBUG "0" 05:38:24 DEBUG } 05:38:24 DEBUG } 05:38:24 DEBUG ] 05:38:24 DEBUG "instance": "10.142.0.77:9100" 05:38:24 DEBUG }, 05:38:24 DEBUG } 05:38:24 DEBUG "value": [ 05:38:24 DEBUG ] 05:38:24 DEBUG 1712723904, 05:38:24 DEBUG } 05:38:24 DEBUG "303314199552" 05:38:24 DEBUG } 05:38:24 DEBUG ] 05:38:24 DEBUG } 05:38:24 DEBUG ] 05:38:24 DEBUG } 05:38:24 DEBUG } 05:38:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723904" 05:38:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723904" 05:38:24 INFO [http_cmd]: http status code OK 05:38:24 DEBUG --- status code --- 05:38:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:24 DEBUG --- http response --- 05:38:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:38:24 DEBUG ----- output ----- 05:38:24 INFO [http_cmd]: http status code OK 05:38:24 DEBUG { 05:38:24 DEBUG --- status code --- 05:38:24 DEBUG "status": "success", 05:38:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:24 DEBUG "data": { 05:38:24 DEBUG --- http response --- 05:38:24 DEBUG "resultType": "vector", 05:38:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:38:24 DEBUG "result": [ 05:38:24 DEBUG ----- output ----- 05:38:24 DEBUG { 05:38:24 DEBUG { 05:38:24 DEBUG "metric": { 05:38:24 DEBUG "status": "success", 05:38:24 DEBUG "instance": "10.142.0.25:9100" 05:38:24 DEBUG "data": { 05:38:24 DEBUG }, 05:38:24 DEBUG "resultType": "vector", 05:38:24 DEBUG "value": [ 05:38:24 DEBUG "result": [ 05:38:24 DEBUG 05:38:24 DEBUG { 05:38:24 DEBUG 05:38:24 DEBUG "metric": { 05:38:24 DEBUG 05:38:24 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:38:24 DEBUG [...] 05:38:24 DEBUG }, 05:38:24 DEBUG 05:38:24 DEBUG "value": [ 05:38:24 DEBUG 05:38:24 DEBUG 05:38:24 DEBUG 05:38:24 DEBUG 05:38:24 DEBUG "instance": "10.142.0.77:9100" 05:38:24 DEBUG 05:38:24 DEBUG }, 05:38:24 DEBUG [...] 05:38:24 DEBUG "value": [ 05:38:24 DEBUG 05:38:24 DEBUG 1712723904, 05:38:24 DEBUG 05:38:24 DEBUG "37462137.56666666" 05:38:24 DEBUG 05:38:24 DEBUG ] 05:38:24 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:38:24 DEBUG } 05:38:24 DEBUG }, 05:38:24 DEBUG ] 05:38:24 DEBUG "value": [ 05:38:24 DEBUG } 05:38:24 DEBUG 1712723904, 05:38:24 DEBUG } 05:38:24 DEBUG "0" 05:38:24 DEBUG ] 05:38:24 DEBUG } 05:38:24 DEBUG ] 05:38:24 DEBUG } 05:38:24 DEBUG } 05:38:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723905" 05:38:25 INFO [http_cmd]: http status code OK 05:38:25 DEBUG --- status code --- 05:38:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:25 DEBUG --- http response --- 05:38:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:38:25 DEBUG ----- output ----- 05:38:25 DEBUG { 05:38:25 DEBUG "status": "success", 05:38:25 DEBUG "data": { 05:38:25 DEBUG "resultType": "vector", 05:38:25 DEBUG "result": [ 05:38:25 DEBUG { 05:38:25 DEBUG "metric": { 05:38:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:38:25 DEBUG }, 05:38:25 DEBUG "value": [ 05:38:25 DEBUG 05:38:25 DEBUG 05:38:25 DEBUG 05:38:25 DEBUG [...] 05:38:25 DEBUG 05:38:25 DEBUG 05:38:25 DEBUG 05:38:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:38:25 DEBUG }, 05:38:25 DEBUG "value": [ 05:38:25 DEBUG 1712723905, 05:38:25 DEBUG "227072.5" 05:38:25 DEBUG ] 05:38:25 DEBUG } 05:38:25 DEBUG ] 05:38:25 DEBUG } 05:38:25 DEBUG } 05:38:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723905" 05:38:25 INFO [http_cmd]: http status code OK 05:38:25 DEBUG --- status code --- 05:38:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:25 DEBUG --- http response --- 05:38:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:38:25 DEBUG ----- output ----- 05:38:25 DEBUG { 05:38:25 DEBUG "status": "success", 05:38:25 DEBUG "data": { 05:38:25 DEBUG "resultType": "vector", 05:38:25 DEBUG "result": [ 05:38:25 DEBUG { 05:38:25 DEBUG "metric": { 05:38:25 DEBUG "pod": "ds-idrepo-0" 05:38:25 DEBUG }, 05:38:25 DEBUG "value": [ 05:38:25 DEBUG 05:38:25 DEBUG 05:38:25 DEBUG 05:38:25 DEBUG [...] 05:38:25 DEBUG 05:38:25 DEBUG 05:38:25 DEBUG 05:38:25 DEBUG "pod": "ds-idrepo-1" 05:38:25 DEBUG }, 05:38:25 DEBUG "value": [ 05:38:25 DEBUG 1712723905, 05:38:25 DEBUG "0" 05:38:25 DEBUG ] 05:38:25 DEBUG } 05:38:25 DEBUG ] 05:38:25 DEBUG } 05:38:25 DEBUG } 05:38:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723905" 05:38:25 INFO [http_cmd]: http status code OK 05:38:25 DEBUG --- status code --- 05:38:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:25 DEBUG --- http response --- 05:38:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:38:25 DEBUG ----- output ----- 05:38:25 DEBUG { 05:38:25 DEBUG "status": "success", 05:38:25 DEBUG "data": { 05:38:25 DEBUG "resultType": "vector", 05:38:25 DEBUG "result": [ 05:38:25 DEBUG { 05:38:25 DEBUG "metric": { 05:38:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:38:25 DEBUG }, 05:38:25 DEBUG "value": [ 05:38:25 DEBUG 05:38:25 DEBUG 05:38:25 DEBUG 05:38:25 DEBUG [...] 05:38:25 DEBUG 05:38:25 DEBUG 05:38:25 DEBUG 05:38:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:38:25 DEBUG }, 05:38:25 DEBUG "value": [ 05:38:25 DEBUG 1712723905, 05:38:25 DEBUG "1251.8666666666666" 05:38:25 DEBUG ] 05:38:25 DEBUG } 05:38:25 DEBUG ] 05:38:25 DEBUG } 05:38:25 DEBUG } 05:38:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723905" 05:38:25 INFO [http_cmd]: http status code OK 05:38:25 DEBUG --- status code --- 05:38:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:25 DEBUG --- http response --- 05:38:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:38:25 DEBUG ----- output ----- 05:38:25 DEBUG { 05:38:25 DEBUG "status": "success", 05:38:25 DEBUG "data": { 05:38:25 DEBUG "resultType": "vector", 05:38:25 DEBUG "result": [ 05:38:25 DEBUG { 05:38:25 DEBUG "metric": { 05:38:25 DEBUG "instance": "10.142.0.25:9100" 05:38:25 DEBUG }, 05:38:25 DEBUG "value": [ 05:38:25 DEBUG 05:38:25 DEBUG 05:38:25 DEBUG 05:38:25 DEBUG [...] 05:38:25 DEBUG 05:38:25 DEBUG 05:38:25 DEBUG 05:38:25 DEBUG "instance": "10.142.0.77:9100" 05:38:25 DEBUG }, 05:38:25 DEBUG "value": [ 05:38:25 DEBUG 1712723905, 05:38:25 DEBUG "0.15443333333333767" 05:38:25 DEBUG ] 05:38:25 DEBUG } 05:38:25 DEBUG ] 05:38:25 DEBUG } 05:38:25 DEBUG } 05:38:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723906" 05:38:26 INFO [http_cmd]: http status code OK 05:38:26 DEBUG --- status code --- 05:38:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:26 DEBUG --- http response --- 05:38:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:38:26 DEBUG ----- output ----- 05:38:26 DEBUG { 05:38:26 DEBUG "status": "success", 05:38:26 DEBUG "data": { 05:38:26 DEBUG "resultType": "vector", 05:38:26 DEBUG "result": [ 05:38:26 DEBUG { 05:38:26 DEBUG "metric": { 05:38:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:38:26 DEBUG }, 05:38:26 DEBUG "value": [ 05:38:26 DEBUG 05:38:26 DEBUG 05:38:26 DEBUG 05:38:26 DEBUG [...] 05:38:26 DEBUG 05:38:26 DEBUG 05:38:26 DEBUG 05:38:26 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:38:26 DEBUG }, 05:38:26 DEBUG "value": [ 05:38:26 DEBUG 1712723906, 05:38:26 DEBUG "0" 05:38:26 DEBUG ] 05:38:26 DEBUG } 05:38:26 DEBUG ] 05:38:26 DEBUG } 05:38:26 DEBUG } 05:38:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712723906" 05:38:26 INFO [http_cmd]: http status code OK 05:38:26 DEBUG --- status code --- 05:38:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:26 DEBUG --- http response --- 05:38:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:38:26 DEBUG ----- output ----- 05:38:26 DEBUG { 05:38:26 DEBUG "status": "success", 05:38:26 DEBUG "data": { 05:38:26 DEBUG "resultType": "vector", 05:38:26 DEBUG "result": [ 05:38:26 DEBUG { 05:38:26 DEBUG "metric": { 05:38:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:38:26 DEBUG }, 05:38:26 DEBUG "value": [ 05:38:26 DEBUG 05:38:26 DEBUG 05:38:26 DEBUG 05:38:26 DEBUG [...] 05:38:26 DEBUG 05:38:26 DEBUG 05:38:26 DEBUG 05:38:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:38:26 DEBUG }, 05:38:26 DEBUG "value": [ 05:38:26 DEBUG 1712723906, 05:38:26 DEBUG "3.6085671997198907" 05:38:26 DEBUG ] 05:38:26 DEBUG } 05:38:26 DEBUG ] 05:38:26 DEBUG } 05:38:26 DEBUG } 05:38:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712723907" 05:38:27 INFO [http_cmd]: http status code OK 05:38:27 DEBUG --- status code --- 05:38:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:27 DEBUG --- http response --- 05:38:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:38:27 DEBUG ----- output ----- 05:38:27 DEBUG { 05:38:27 DEBUG "status": "success", 05:38:27 DEBUG "data": { 05:38:27 DEBUG "resultType": "vector", 05:38:27 DEBUG "result": [ 05:38:27 DEBUG { 05:38:27 DEBUG "metric": { 05:38:27 DEBUG "pod": "ds-idrepo-0" 05:38:27 DEBUG }, 05:38:27 DEBUG "value": [ 05:38:27 DEBUG 05:38:27 DEBUG 05:38:27 DEBUG 05:38:27 DEBUG [...] 05:38:27 DEBUG 05:38:27 DEBUG 05:38:27 DEBUG 05:38:27 DEBUG "pod": "ds-cts-2" 05:38:27 DEBUG }, 05:38:27 DEBUG "value": [ 05:38:27 DEBUG 1712723907, 05:38:27 DEBUG "0" 05:38:27 DEBUG ] 05:38:27 DEBUG } 05:38:27 DEBUG ] 05:38:27 DEBUG } 05:38:27 DEBUG } 05:38:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723907" 05:38:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723907" 05:38:27 INFO [http_cmd]: http status code OK 05:38:27 DEBUG --- status code --- 05:38:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:27 DEBUG --- http response --- 05:38:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:38:27 DEBUG ----- output ----- 05:38:27 DEBUG { 05:38:27 DEBUG "status": "success", 05:38:27 DEBUG "data": { 05:38:27 DEBUG "resultType": "vector", 05:38:27 DEBUG "result": [ 05:38:27 DEBUG { 05:38:27 DEBUG "metric": { 05:38:27 DEBUG "pod": "ds-idrepo-2" 05:38:27 DEBUG }, 05:38:27 DEBUG "value": [ 05:38:27 DEBUG 05:38:27 DEBUG 05:38:27 DEBUG 05:38:27 DEBUG [...] 05:38:27 DEBUG 05:38:27 DEBUG 05:38:27 DEBUG 05:38:27 DEBUG "pod": "ds-cts-2" 05:38:27 DEBUG }, 05:38:27 DEBUG "value": [ 05:38:27 DEBUG 1712723907, 05:38:27 DEBUG "0" 05:38:27 DEBUG ] 05:38:27 DEBUG } 05:38:27 DEBUG ] 05:38:27 DEBUG } 05:38:27 DEBUG } 05:38:27 INFO [http_cmd]: http status code OK 05:38:27 DEBUG --- status code --- 05:38:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:27 DEBUG --- http response --- 05:38:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:38:27 DEBUG ----- output ----- 05:38:27 DEBUG { 05:38:27 DEBUG "status": "success", 05:38:27 DEBUG "data": { 05:38:27 DEBUG "resultType": "vector", 05:38:27 DEBUG "result": [ 05:38:27 DEBUG { 05:38:27 DEBUG "metric": { 05:38:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:38:27 DEBUG }, 05:38:27 DEBUG "value": [ 05:38:27 DEBUG 05:38:27 DEBUG 05:38:27 DEBUG 05:38:27 DEBUG [...] 05:38:27 DEBUG 05:38:27 DEBUG 05:38:27 DEBUG 05:38:27 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 05:38:27 DEBUG }, 05:38:27 DEBUG "value": [ 05:38:27 DEBUG 1712723907, 05:38:27 DEBUG "0.005846395725768213" 05:38:27 DEBUG ] 05:38:27 DEBUG } 05:38:27 DEBUG ] 05:38:27 DEBUG } 05:38:27 DEBUG } 05:38:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723907" 05:38:27 INFO [http_cmd]: http status code OK 05:38:27 DEBUG --- status code --- 05:38:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:27 DEBUG --- http response --- 05:38:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:38:27 DEBUG ----- output ----- 05:38:27 DEBUG { 05:38:27 DEBUG "status": "success", 05:38:27 DEBUG "data": { 05:38:27 DEBUG "resultType": "vector", 05:38:27 DEBUG "result": [ 05:38:27 DEBUG { 05:38:27 DEBUG "metric": { 05:38:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:38:27 DEBUG }, 05:38:27 DEBUG "value": [ 05:38:27 DEBUG 05:38:27 DEBUG 05:38:27 DEBUG 05:38:27 DEBUG [...] 05:38:27 DEBUG 05:38:27 DEBUG 05:38:27 DEBUG 05:38:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:38:27 DEBUG }, 05:38:27 DEBUG "value": [ 05:38:27 DEBUG 1712723907, 05:38:27 DEBUG "1252.0333333333333" 05:38:27 DEBUG ] 05:38:27 DEBUG } 05:38:27 DEBUG ] 05:38:27 DEBUG } 05:38:27 DEBUG } 05:38:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712723907" 05:38:27 INFO [http_cmd]: http status code OK 05:38:27 DEBUG --- status code --- 05:38:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:27 DEBUG --- http response --- 05:38:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:38:27 DEBUG ----- output ----- 05:38:27 DEBUG { 05:38:27 DEBUG "status": "success", 05:38:27 DEBUG "data": { 05:38:27 DEBUG "resultType": "vector", 05:38:27 DEBUG "result": [ 05:38:27 DEBUG { 05:38:27 DEBUG "metric": { 05:38:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:38:27 DEBUG }, 05:38:27 DEBUG "value": [ 05:38:27 DEBUG 05:38:27 DEBUG 05:38:27 DEBUG 05:38:27 DEBUG [...] 05:38:27 DEBUG 05:38:27 DEBUG 05:38:27 DEBUG 05:38:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:38:27 DEBUG }, 05:38:27 DEBUG "value": [ 05:38:27 DEBUG 1712723907, 05:38:27 DEBUG "0.042499999999999996" 05:38:27 DEBUG ] 05:38:27 DEBUG } 05:38:27 DEBUG ] 05:38:27 DEBUG } 05:38:27 DEBUG } 05:38:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723907" 05:38:27 INFO [http_cmd]: http status code OK 05:38:27 DEBUG --- status code --- 05:38:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:27 DEBUG --- http response --- 05:38:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:38:27 DEBUG ----- output ----- 05:38:27 DEBUG { 05:38:27 DEBUG "status": "success", 05:38:27 DEBUG "data": { 05:38:27 DEBUG "resultType": "vector", 05:38:27 DEBUG "result": [ 05:38:27 DEBUG { 05:38:27 DEBUG "metric": { 05:38:27 DEBUG "pod": "ds-idrepo-0" 05:38:27 DEBUG }, 05:38:27 DEBUG "value": [ 05:38:27 DEBUG 05:38:27 DEBUG 05:38:27 DEBUG 05:38:27 DEBUG [...] 05:38:27 DEBUG 05:38:27 DEBUG 05:38:27 DEBUG 05:38:27 DEBUG "pod": "ds-idrepo-1" 05:38:27 DEBUG }, 05:38:27 DEBUG "value": [ 05:38:27 DEBUG 1712723907, 05:38:27 DEBUG "0" 05:38:27 DEBUG ] 05:38:27 DEBUG } 05:38:27 DEBUG ] 05:38:27 DEBUG } 05:38:27 DEBUG } 05:38:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723908" 05:38:28 INFO [http_cmd]: http status code OK 05:38:28 DEBUG --- status code --- 05:38:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:28 DEBUG --- http response --- 05:38:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:38:28 DEBUG ----- output ----- 05:38:28 DEBUG { 05:38:28 DEBUG "status": "success", 05:38:28 DEBUG "data": { 05:38:28 DEBUG "resultType": "vector", 05:38:28 DEBUG "result": [ 05:38:28 DEBUG { 05:38:28 DEBUG "metric": { 05:38:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:38:28 DEBUG }, 05:38:28 DEBUG "value": [ 05:38:28 DEBUG 05:38:28 DEBUG 05:38:28 DEBUG 05:38:28 DEBUG [...] 05:38:28 DEBUG 05:38:28 DEBUG 05:38:28 DEBUG 05:38:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:38:28 DEBUG }, 05:38:28 DEBUG "value": [ 05:38:28 DEBUG 1712723908, 05:38:28 DEBUG "1255.3333333333333" 05:38:28 DEBUG ] 05:38:28 DEBUG } 05:38:28 DEBUG ] 05:38:28 DEBUG } 05:38:28 DEBUG } 05:38:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723908" 05:38:28 INFO [http_cmd]: http status code OK 05:38:28 DEBUG --- status code --- 05:38:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:28 DEBUG --- http response --- 05:38:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:38:28 DEBUG ----- output ----- 05:38:28 DEBUG { 05:38:28 DEBUG "status": "success", 05:38:28 DEBUG "data": { 05:38:28 DEBUG "resultType": "vector", 05:38:28 DEBUG "result": [ 05:38:28 DEBUG { 05:38:28 DEBUG "metric": { 05:38:28 DEBUG "pod": "ds-idrepo-0" 05:38:28 DEBUG }, 05:38:28 DEBUG "value": [ 05:38:28 DEBUG 05:38:28 DEBUG 05:38:28 DEBUG 05:38:28 DEBUG [...] 05:38:28 DEBUG 05:38:28 DEBUG 05:38:28 DEBUG 05:38:28 DEBUG "pod": "ds-idrepo-1" 05:38:28 DEBUG }, 05:38:28 DEBUG "value": [ 05:38:28 DEBUG 1712723908, 05:38:28 DEBUG "0" 05:38:28 DEBUG ] 05:38:28 DEBUG } 05:38:28 DEBUG ] 05:38:28 DEBUG } 05:38:28 DEBUG } 05:38:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723908" 05:38:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712723908" 05:38:28 INFO [http_cmd]: http status code OK 05:38:28 DEBUG --- status code --- 05:38:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:28 DEBUG --- http response --- 05:38:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:38:28 DEBUG ----- output ----- 05:38:28 INFO [http_cmd]: http status code OK 05:38:28 DEBUG { 05:38:28 DEBUG "status": "success", 05:38:28 DEBUG "data": { 05:38:28 DEBUG "resultType": "vector", 05:38:28 DEBUG "result": [ 05:38:28 DEBUG { 05:38:28 DEBUG "metric": { 05:38:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:38:28 DEBUG }, 05:38:28 DEBUG "value": [ 05:38:28 DEBUG 05:38:28 DEBUG 05:38:28 DEBUG 05:38:28 DEBUG [...] 05:38:28 DEBUG 05:38:28 DEBUG 05:38:28 DEBUG 05:38:28 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 05:38:28 DEBUG }, 05:38:28 DEBUG "value": [ 05:38:28 DEBUG --- status code --- 05:38:28 DEBUG 1712723908, 05:38:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:28 DEBUG "0" 05:38:28 DEBUG --- http response --- 05:38:28 DEBUG ] 05:38:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:38:28 DEBUG ----- output ----- 05:38:28 DEBUG { 05:38:28 DEBUG } 05:38:28 DEBUG ] 05:38:28 DEBUG } 05:38:28 DEBUG } 05:38:28 DEBUG "status": "success", 05:38:28 DEBUG "data": { 05:38:28 DEBUG "resultType": "vector", 05:38:28 DEBUG "result": [ 05:38:28 DEBUG { 05:38:28 DEBUG "metric": { 05:38:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:38:28 DEBUG }, 05:38:28 DEBUG "value": [ 05:38:28 DEBUG 05:38:28 DEBUG 05:38:28 DEBUG 05:38:28 DEBUG [...] 05:38:28 DEBUG 05:38:28 DEBUG 05:38:28 DEBUG 05:38:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:38:28 DEBUG }, 05:38:28 DEBUG "value": [ 05:38:28 DEBUG 1712723908, 05:38:28 DEBUG "2488795136" 05:38:28 DEBUG ] 05:38:28 DEBUG } 05:38:28 DEBUG ] 05:38:28 DEBUG } 05:38:28 DEBUG } 05:38:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723908" 05:38:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723908" 05:38:28 INFO [http_cmd]: http status code OK 05:38:28 DEBUG --- status code --- 05:38:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:28 DEBUG --- http response --- 05:38:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:38:28 DEBUG ----- output ----- 05:38:28 DEBUG { 05:38:28 DEBUG "status": "success", 05:38:28 DEBUG "data": { 05:38:28 DEBUG "resultType": "vector", 05:38:28 DEBUG "result": [ 05:38:28 DEBUG { 05:38:28 DEBUG "metric": { 05:38:28 DEBUG "instance": "10.142.0.25:9100" 05:38:28 DEBUG }, 05:38:28 DEBUG "value": [ 05:38:28 DEBUG 05:38:28 DEBUG 05:38:28 DEBUG 05:38:28 DEBUG [...] 05:38:28 DEBUG 05:38:28 DEBUG 05:38:28 DEBUG 05:38:28 DEBUG "instance": "10.142.0.77:9100" 05:38:28 DEBUG }, 05:38:28 DEBUG "value": [ 05:38:28 DEBUG 1712723908, 05:38:28 DEBUG "37463669.7" 05:38:28 DEBUG ] 05:38:28 DEBUG } 05:38:28 DEBUG ] 05:38:28 DEBUG } 05:38:28 DEBUG } 05:38:28 INFO [http_cmd]: http status code OK 05:38:28 DEBUG --- status code --- 05:38:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:28 DEBUG --- http response --- 05:38:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:38:28 DEBUG ----- output ----- 05:38:28 DEBUG { 05:38:28 DEBUG "status": "success", 05:38:28 DEBUG "data": { 05:38:28 DEBUG "resultType": "vector", 05:38:28 DEBUG "result": [ 05:38:28 DEBUG { 05:38:28 DEBUG "metric": { 05:38:28 DEBUG "instance": "10.142.0.25:9100" 05:38:28 DEBUG }, 05:38:28 DEBUG "value": [ 05:38:28 DEBUG 05:38:28 DEBUG 05:38:28 DEBUG 05:38:28 DEBUG [...] 05:38:28 DEBUG 05:38:28 DEBUG 05:38:28 DEBUG 05:38:28 DEBUG "instance": "10.142.0.77:9100" 05:38:28 DEBUG }, 05:38:28 DEBUG "value": [ 05:38:28 DEBUG 1712723908, 05:38:28 DEBUG "0.050666666666666776" 05:38:28 DEBUG ] 05:38:28 DEBUG } 05:38:28 DEBUG ] 05:38:28 DEBUG } 05:38:28 DEBUG } 05:38:43 INFO 05:38:43 INFO [loop_until]: kubectl --namespace=xlou top pods 05:38:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:38:43 INFO [loop_until]: OK (rc = 0) 05:38:43 DEBUG --- stdout --- 05:38:43 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 9096m 8807Mi am-654c4fcbcf-9scrj 8721m 8421Mi am-654c4fcbcf-v5qb5 7236m 8399Mi ds-cts-0 6874m 13274Mi ds-cts-1 7413m 12394Mi ds-cts-2 6823m 13334Mi ds-idrepo-0 9660m 5622Mi ds-idrepo-1 322m 5669Mi ds-idrepo-2 329m 5698Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 5m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3523m 1671Mi 05:38:43 DEBUG --- stderr --- 05:38:43 DEBUG 05:38:50 INFO 05:38:50 INFO [loop_until]: kubectl --namespace=xlou top node 05:38:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:38:50 INFO [loop_until]: OK (rc = 0) 05:38:50 DEBUG --- stdout --- 05:38:50 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 95m 0% 2302Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8098m 50% 9279Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 81m 0% 3261Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10797m 67% 6519Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9599m 60% 9311Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9993m 62% 9647Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 78m 0% 852Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9759m 61% 6285Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 7280m 45% 13006Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 7172m 45% 13065Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 403m 2% 6371Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7468m 46% 13003Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 359m 2% 6330Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3700m 23% 5340Mi 9% 05:38:50 DEBUG --- stderr --- 05:38:50 DEBUG 05:39:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712723964" 05:39:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723964" 05:39:24 ERROR [http_cmd]: ERROR 05:39:24 INFO [http_cmd]: http status code OK 05:39:24 DEBUG --- status code --- 05:39:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:24 DEBUG --- http response --- 05:39:24 ERROR --- status code --- 05:39:24 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:39:24 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:39:24 ERROR --- http response --- 05:39:24 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:39:24 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:39:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723964" 05:39:24 INFO [http_cmd]: http status code OK 05:39:24 DEBUG --- status code --- 05:39:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:24 DEBUG --- http response --- 05:39:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:39:24 DEBUG ----- output ----- 05:39:24 DEBUG { 05:39:24 DEBUG "status": "success", 05:39:24 DEBUG "data": { 05:39:24 DEBUG "resultType": "vector", 05:39:24 DEBUG "result": [ 05:39:24 DEBUG { 05:39:24 DEBUG "metric": { 05:39:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:39:24 DEBUG }, 05:39:24 DEBUG "value": [ 05:39:24 DEBUG 05:39:24 DEBUG 05:39:24 DEBUG 05:39:24 DEBUG [...] 05:39:24 DEBUG 05:39:24 DEBUG 05:39:24 DEBUG 05:39:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:39:24 DEBUG }, 05:39:24 DEBUG "value": [ 05:39:24 DEBUG 1712723964, 05:39:24 DEBUG "0" 05:39:24 DEBUG ] 05:39:24 DEBUG } 05:39:24 DEBUG ] 05:39:24 DEBUG } 05:39:24 DEBUG } 05:39:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723964" 05:39:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723964" 05:39:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723964" 05:39:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723964" 05:39:24 INFO [http_cmd]: http status code OK 05:39:24 INFO [http_cmd]: http status code OK 05:39:24 DEBUG --- status code --- 05:39:24 DEBUG --- status code --- 05:39:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:24 DEBUG --- http response --- 05:39:24 DEBUG --- http response --- 05:39:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:39:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:39:24 INFO [http_cmd]: http status code OK 05:39:24 DEBUG ----- output ----- 05:39:24 DEBUG ----- output ----- 05:39:24 DEBUG --- status code --- 05:39:24 DEBUG { 05:39:24 DEBUG "status": "success", 05:39:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:24 DEBUG "data": { 05:39:24 DEBUG { 05:39:24 DEBUG "status": "success", 05:39:24 DEBUG "resultType": "vector", 05:39:24 DEBUG "data": { 05:39:24 DEBUG --- http response --- 05:39:24 DEBUG "result": [ 05:39:24 DEBUG "resultType": "vector", 05:39:24 DEBUG { 05:39:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:39:24 DEBUG "result": [ 05:39:24 DEBUG "metric": { 05:39:24 DEBUG ----- output ----- 05:39:24 DEBUG { 05:39:24 DEBUG "pod": "ds-idrepo-0" 05:39:24 DEBUG }, 05:39:24 INFO [http_cmd]: http status code OK 05:39:24 DEBUG "metric": { 05:39:24 DEBUG { 05:39:24 DEBUG "value": [ 05:39:24 DEBUG --- status code --- 05:39:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:39:24 DEBUG "status": "success", 05:39:24 DEBUG 05:39:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:24 DEBUG }, 05:39:24 DEBUG "data": { 05:39:24 DEBUG 05:39:24 DEBUG --- http response --- 05:39:24 DEBUG "value": [ 05:39:24 DEBUG "resultType": "vector", 05:39:24 DEBUG 05:39:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:39:24 DEBUG 05:39:24 DEBUG "result": [ 05:39:24 DEBUG [...] 05:39:24 DEBUG ----- output ----- 05:39:24 DEBUG 05:39:24 DEBUG { 05:39:24 DEBUG 05:39:24 DEBUG { 05:39:24 DEBUG 05:39:24 DEBUG "metric": { 05:39:24 DEBUG 05:39:24 DEBUG "status": "success", 05:39:24 DEBUG [...] 05:39:24 DEBUG "instance": "10.142.0.25:9100" 05:39:24 DEBUG 05:39:24 DEBUG "data": { 05:39:24 DEBUG 05:39:24 DEBUG }, 05:39:24 DEBUG "pod": "ds-cts-2" 05:39:24 DEBUG "resultType": "vector", 05:39:24 DEBUG 05:39:24 DEBUG "value": [ 05:39:24 DEBUG }, 05:39:24 DEBUG "result": [ 05:39:24 DEBUG 05:39:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:39:24 DEBUG "value": [ 05:39:24 DEBUG { 05:39:24 DEBUG }, 05:39:24 DEBUG 05:39:24 DEBUG 1712723964, 05:39:24 DEBUG "metric": { 05:39:24 DEBUG "value": [ 05:39:24 DEBUG 05:39:24 DEBUG "29.966666666666665" 05:39:24 DEBUG "pod": "ds-idrepo-0" 05:39:24 DEBUG 1712723964, 05:39:24 DEBUG 05:39:24 DEBUG ] 05:39:24 DEBUG }, 05:39:24 DEBUG "value": [ 05:39:24 DEBUG [...] 05:39:24 DEBUG } 05:39:24 DEBUG "1252.9333333333334" 05:39:24 DEBUG 05:39:24 DEBUG 05:39:24 DEBUG ] 05:39:24 DEBUG ] 05:39:24 DEBUG 05:39:24 DEBUG 05:39:24 DEBUG } 05:39:24 DEBUG } 05:39:24 DEBUG 05:39:24 DEBUG 05:39:24 DEBUG "instance": "10.142.0.77:9100" 05:39:24 DEBUG }, 05:39:24 DEBUG "value": [ 05:39:24 DEBUG } 05:39:24 DEBUG [...] 05:39:24 DEBUG 05:39:24 DEBUG ] 05:39:24 DEBUG } 05:39:24 DEBUG 05:39:24 DEBUG 1712723964, 05:39:24 DEBUG "305286695936" 05:39:24 DEBUG 05:39:24 DEBUG ] 05:39:24 DEBUG } 05:39:24 DEBUG "pod": "ds-cts-2" 05:39:24 DEBUG } 05:39:24 DEBUG }, 05:39:24 DEBUG ] 05:39:24 DEBUG "value": [ 05:39:24 DEBUG } 05:39:24 DEBUG 1712723964, 05:39:24 DEBUG } 05:39:24 DEBUG "0" 05:39:24 DEBUG ] 05:39:24 DEBUG } 05:39:24 DEBUG ] 05:39:24 DEBUG } 05:39:24 DEBUG } 05:39:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723965" 05:39:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723965" 05:39:25 INFO [http_cmd]: http status code OK 05:39:25 INFO [http_cmd]: http status code OK 05:39:25 DEBUG --- status code --- 05:39:25 DEBUG --- status code --- 05:39:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:25 DEBUG --- http response --- 05:39:25 DEBUG --- http response --- 05:39:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:39:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 05:39:25 DEBUG ----- output ----- 05:39:25 DEBUG ----- output ----- 05:39:25 DEBUG { 05:39:25 DEBUG { 05:39:25 DEBUG "status": "success", 05:39:25 DEBUG "status": "success", 05:39:25 DEBUG "data": { 05:39:25 DEBUG "data": { 05:39:25 DEBUG "resultType": "vector", 05:39:25 DEBUG "resultType": "vector", 05:39:25 DEBUG "result": [ 05:39:25 DEBUG "result": [ 05:39:25 DEBUG { 05:39:25 DEBUG { 05:39:25 DEBUG "metric": { 05:39:25 DEBUG "metric": { 05:39:25 DEBUG "instance": "10.142.0.25:9100" 05:39:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:39:25 DEBUG }, 05:39:25 DEBUG "value": [ 05:39:25 DEBUG }, 05:39:25 DEBUG 05:39:25 DEBUG "value": [ 05:39:25 DEBUG 05:39:25 DEBUG 05:39:25 DEBUG 05:39:25 DEBUG 05:39:25 DEBUG [...] 05:39:25 DEBUG 05:39:25 DEBUG 05:39:25 DEBUG [...] 05:39:25 DEBUG 05:39:25 DEBUG 05:39:25 DEBUG 05:39:25 DEBUG 05:39:25 DEBUG "instance": "10.142.0.77:9100" 05:39:25 DEBUG 05:39:25 DEBUG }, 05:39:25 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:39:25 DEBUG "value": [ 05:39:25 DEBUG }, 05:39:25 DEBUG 1712723965, 05:39:25 DEBUG "value": [ 05:39:25 DEBUG "37442368.43333334" 05:39:25 DEBUG 1712723965, 05:39:25 DEBUG ] 05:39:25 DEBUG "0" 05:39:25 DEBUG } 05:39:25 DEBUG ] 05:39:25 DEBUG ] 05:39:25 DEBUG } 05:39:25 DEBUG } 05:39:25 DEBUG ] 05:39:25 DEBUG } 05:39:25 DEBUG } 05:39:25 DEBUG } 05:39:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723965" 05:39:25 INFO [http_cmd]: http status code OK 05:39:25 DEBUG --- status code --- 05:39:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:25 DEBUG --- http response --- 05:39:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:39:25 DEBUG ----- output ----- 05:39:25 DEBUG { 05:39:25 DEBUG "status": "success", 05:39:25 DEBUG "data": { 05:39:25 DEBUG "resultType": "vector", 05:39:25 DEBUG "result": [ 05:39:25 DEBUG { 05:39:25 DEBUG "metric": { 05:39:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:39:25 DEBUG }, 05:39:25 DEBUG "value": [ 05:39:25 DEBUG 05:39:25 DEBUG 05:39:25 DEBUG 05:39:25 DEBUG [...] 05:39:25 DEBUG 05:39:25 DEBUG 05:39:25 DEBUG 05:39:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:39:25 DEBUG }, 05:39:25 DEBUG "value": [ 05:39:25 DEBUG 1712723965, 05:39:25 DEBUG "227082" 05:39:25 DEBUG ] 05:39:25 DEBUG } 05:39:25 DEBUG ] 05:39:25 DEBUG } 05:39:25 DEBUG } 05:39:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723965" 05:39:25 INFO [http_cmd]: http status code OK 05:39:25 DEBUG --- status code --- 05:39:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:25 DEBUG --- http response --- 05:39:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:39:25 DEBUG ----- output ----- 05:39:25 DEBUG { 05:39:25 DEBUG "status": "success", 05:39:25 DEBUG "data": { 05:39:25 DEBUG "resultType": "vector", 05:39:25 DEBUG "result": [ 05:39:25 DEBUG { 05:39:25 DEBUG "metric": { 05:39:25 DEBUG "pod": "ds-idrepo-0" 05:39:25 DEBUG }, 05:39:25 DEBUG "value": [ 05:39:25 DEBUG 05:39:25 DEBUG 05:39:25 DEBUG 05:39:25 DEBUG [...] 05:39:25 DEBUG 05:39:25 DEBUG 05:39:25 DEBUG 05:39:25 DEBUG "pod": "ds-idrepo-1" 05:39:25 DEBUG }, 05:39:25 DEBUG "value": [ 05:39:25 DEBUG 1712723965, 05:39:25 DEBUG "0" 05:39:25 DEBUG ] 05:39:25 DEBUG } 05:39:25 DEBUG ] 05:39:25 DEBUG } 05:39:25 DEBUG } 05:39:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723965" 05:39:25 INFO [http_cmd]: http status code OK 05:39:25 DEBUG --- status code --- 05:39:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:25 DEBUG --- http response --- 05:39:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:39:25 DEBUG ----- output ----- 05:39:25 DEBUG { 05:39:25 DEBUG "status": "success", 05:39:25 DEBUG "data": { 05:39:25 DEBUG "resultType": "vector", 05:39:25 DEBUG "result": [ 05:39:25 DEBUG { 05:39:25 DEBUG "metric": { 05:39:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:39:25 DEBUG }, 05:39:25 DEBUG "value": [ 05:39:25 DEBUG 05:39:25 DEBUG 05:39:25 DEBUG 05:39:25 DEBUG [...] 05:39:25 DEBUG 05:39:25 DEBUG 05:39:25 DEBUG 05:39:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:39:25 DEBUG }, 05:39:25 DEBUG "value": [ 05:39:25 DEBUG 1712723965, 05:39:25 DEBUG "1253.2666666666667" 05:39:25 DEBUG ] 05:39:25 DEBUG } 05:39:25 DEBUG ] 05:39:25 DEBUG } 05:39:25 DEBUG } 05:39:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723966" 05:39:26 INFO [http_cmd]: http status code OK 05:39:26 DEBUG --- status code --- 05:39:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:26 DEBUG --- http response --- 05:39:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:39:26 DEBUG ----- output ----- 05:39:26 DEBUG { 05:39:26 DEBUG "status": "success", 05:39:26 DEBUG "data": { 05:39:26 DEBUG "resultType": "vector", 05:39:26 DEBUG "result": [ 05:39:26 DEBUG { 05:39:26 DEBUG "metric": { 05:39:26 DEBUG "instance": "10.142.0.25:9100" 05:39:26 DEBUG }, 05:39:26 DEBUG "value": [ 05:39:26 DEBUG 05:39:26 DEBUG 05:39:26 DEBUG 05:39:26 DEBUG [...] 05:39:26 DEBUG 05:39:26 DEBUG 05:39:26 DEBUG 05:39:26 DEBUG "instance": "10.142.0.77:9100" 05:39:26 DEBUG }, 05:39:26 DEBUG "value": [ 05:39:26 DEBUG 1712723966, 05:39:26 DEBUG "0.03383333333333513" 05:39:26 DEBUG ] 05:39:26 DEBUG } 05:39:26 DEBUG ] 05:39:26 DEBUG } 05:39:26 DEBUG } 05:39:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723966" 05:39:26 INFO [http_cmd]: http status code OK 05:39:26 DEBUG --- status code --- 05:39:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:26 DEBUG --- http response --- 05:39:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:39:26 DEBUG ----- output ----- 05:39:26 DEBUG { 05:39:26 DEBUG "status": "success", 05:39:26 DEBUG "data": { 05:39:26 DEBUG "resultType": "vector", 05:39:26 DEBUG "result": [ 05:39:26 DEBUG { 05:39:26 DEBUG "metric": { 05:39:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:39:26 DEBUG }, 05:39:26 DEBUG "value": [ 05:39:26 DEBUG 05:39:26 DEBUG 05:39:26 DEBUG 05:39:26 DEBUG [...] 05:39:26 DEBUG 05:39:26 DEBUG 05:39:26 DEBUG 05:39:26 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:39:26 DEBUG }, 05:39:26 DEBUG "value": [ 05:39:26 DEBUG 1712723966, 05:39:26 DEBUG "8002.616491843871" 05:39:26 DEBUG ] 05:39:26 DEBUG } 05:39:26 DEBUG ] 05:39:26 DEBUG } 05:39:26 DEBUG } 05:39:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712723966" 05:39:26 INFO [http_cmd]: http status code OK 05:39:26 DEBUG --- status code --- 05:39:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:26 DEBUG --- http response --- 05:39:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:39:26 DEBUG ----- output ----- 05:39:26 DEBUG { 05:39:26 DEBUG "status": "success", 05:39:26 DEBUG "data": { 05:39:26 DEBUG "resultType": "vector", 05:39:26 DEBUG "result": [ 05:39:26 DEBUG { 05:39:26 DEBUG "metric": { 05:39:26 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:39:26 DEBUG }, 05:39:26 DEBUG "value": [ 05:39:26 DEBUG 05:39:26 DEBUG 05:39:26 DEBUG 05:39:26 DEBUG [...] 05:39:26 DEBUG 05:39:26 DEBUG 05:39:26 DEBUG 05:39:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:39:26 DEBUG }, 05:39:26 DEBUG "value": [ 05:39:26 DEBUG 1712723966, 05:39:26 DEBUG "3.609341312489656" 05:39:26 DEBUG ] 05:39:26 DEBUG } 05:39:26 DEBUG ] 05:39:26 DEBUG } 05:39:26 DEBUG } 05:39:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712723967" 05:39:27 INFO [http_cmd]: http status code OK 05:39:27 DEBUG --- status code --- 05:39:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:27 DEBUG --- http response --- 05:39:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:39:27 DEBUG ----- output ----- 05:39:27 DEBUG { 05:39:27 DEBUG "status": "success", 05:39:27 DEBUG "data": { 05:39:27 DEBUG "resultType": "vector", 05:39:27 DEBUG "result": [ 05:39:27 DEBUG { 05:39:27 DEBUG "metric": { 05:39:27 DEBUG "pod": "ds-idrepo-0" 05:39:27 DEBUG }, 05:39:27 DEBUG "value": [ 05:39:27 DEBUG 05:39:27 DEBUG 05:39:27 DEBUG 05:39:27 DEBUG [...] 05:39:27 DEBUG 05:39:27 DEBUG 05:39:27 DEBUG 05:39:27 DEBUG "pod": "ds-cts-2" 05:39:27 DEBUG }, 05:39:27 DEBUG "value": [ 05:39:27 DEBUG 1712723967, 05:39:27 DEBUG "0" 05:39:27 DEBUG ] 05:39:27 DEBUG } 05:39:27 DEBUG ] 05:39:27 DEBUG } 05:39:27 DEBUG } 05:39:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723967" 05:39:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723967" 05:39:27 INFO [http_cmd]: http status code OK 05:39:27 DEBUG --- status code --- 05:39:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:27 DEBUG --- http response --- 05:39:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:39:27 DEBUG ----- output ----- 05:39:27 DEBUG { 05:39:27 DEBUG "status": "success", 05:39:27 DEBUG "data": { 05:39:27 DEBUG "resultType": "vector", 05:39:27 DEBUG "result": [ 05:39:27 DEBUG { 05:39:27 DEBUG "metric": { 05:39:27 DEBUG "pod": "ds-idrepo-2" 05:39:27 DEBUG }, 05:39:27 DEBUG "value": [ 05:39:27 DEBUG 05:39:27 DEBUG 05:39:27 DEBUG 05:39:27 DEBUG [...] 05:39:27 DEBUG 05:39:27 DEBUG 05:39:27 DEBUG 05:39:27 DEBUG "pod": "ds-cts-2" 05:39:27 DEBUG }, 05:39:27 DEBUG "value": [ 05:39:27 DEBUG 1712723967, 05:39:27 DEBUG "0" 05:39:27 DEBUG ] 05:39:27 DEBUG } 05:39:27 DEBUG ] 05:39:27 DEBUG } 05:39:27 DEBUG } 05:39:27 INFO [http_cmd]: http status code OK 05:39:27 DEBUG --- status code --- 05:39:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:27 DEBUG --- http response --- 05:39:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:39:27 DEBUG ----- output ----- 05:39:27 DEBUG { 05:39:27 DEBUG "status": "success", 05:39:27 DEBUG "data": { 05:39:27 DEBUG "resultType": "vector", 05:39:27 DEBUG "result": [ 05:39:27 DEBUG { 05:39:27 DEBUG "metric": { 05:39:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:39:27 DEBUG }, 05:39:27 DEBUG "value": [ 05:39:27 DEBUG 05:39:27 DEBUG 05:39:27 DEBUG 05:39:27 DEBUG [...] 05:39:27 DEBUG 05:39:27 DEBUG 05:39:27 DEBUG 05:39:27 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:39:27 DEBUG }, 05:39:27 DEBUG "value": [ 05:39:27 DEBUG 1712723967, 05:39:27 DEBUG "2.4548408461370266" 05:39:27 DEBUG ] 05:39:27 DEBUG } 05:39:27 DEBUG ] 05:39:27 DEBUG } 05:39:27 DEBUG } 05:39:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723967" 05:39:27 INFO [http_cmd]: http status code OK 05:39:27 DEBUG --- status code --- 05:39:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:27 DEBUG --- http response --- 05:39:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:39:27 DEBUG ----- output ----- 05:39:27 DEBUG { 05:39:27 DEBUG "status": "success", 05:39:27 DEBUG "data": { 05:39:27 DEBUG "resultType": "vector", 05:39:27 DEBUG "result": [ 05:39:27 DEBUG { 05:39:27 DEBUG "metric": { 05:39:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:39:27 DEBUG }, 05:39:27 DEBUG "value": [ 05:39:27 DEBUG 05:39:27 DEBUG 05:39:27 DEBUG 05:39:27 DEBUG [...] 05:39:27 DEBUG 05:39:27 DEBUG 05:39:27 DEBUG 05:39:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:39:27 DEBUG }, 05:39:27 DEBUG "value": [ 05:39:27 DEBUG 1712723967, 05:39:27 DEBUG "1253.1333333333332" 05:39:27 DEBUG ] 05:39:27 DEBUG } 05:39:27 DEBUG ] 05:39:27 DEBUG } 05:39:27 DEBUG } 05:39:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712723967" 05:39:27 INFO [http_cmd]: http status code OK 05:39:27 DEBUG --- status code --- 05:39:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:27 DEBUG --- http response --- 05:39:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:39:27 DEBUG ----- output ----- 05:39:27 DEBUG { 05:39:27 DEBUG "status": "success", 05:39:27 DEBUG "data": { 05:39:27 DEBUG "resultType": "vector", 05:39:27 DEBUG "result": [ 05:39:27 DEBUG { 05:39:27 DEBUG "metric": { 05:39:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:39:27 DEBUG }, 05:39:27 DEBUG "value": [ 05:39:27 DEBUG 05:39:27 DEBUG 05:39:27 DEBUG 05:39:27 DEBUG [...] 05:39:27 DEBUG 05:39:27 DEBUG 05:39:27 DEBUG 05:39:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:39:27 DEBUG }, 05:39:27 DEBUG "value": [ 05:39:27 DEBUG 1712723967, 05:39:27 DEBUG "0.043" 05:39:27 DEBUG ] 05:39:27 DEBUG } 05:39:27 DEBUG ] 05:39:27 DEBUG } 05:39:27 DEBUG } 05:39:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723967" 05:39:27 INFO [http_cmd]: http status code OK 05:39:27 DEBUG --- status code --- 05:39:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:27 DEBUG --- http response --- 05:39:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:39:27 DEBUG ----- output ----- 05:39:27 DEBUG { 05:39:27 DEBUG "status": "success", 05:39:27 DEBUG "data": { 05:39:27 DEBUG "resultType": "vector", 05:39:27 DEBUG "result": [ 05:39:27 DEBUG { 05:39:27 DEBUG "metric": { 05:39:27 DEBUG "pod": "ds-idrepo-0" 05:39:27 DEBUG }, 05:39:27 DEBUG "value": [ 05:39:27 DEBUG 05:39:27 DEBUG 05:39:27 DEBUG 05:39:27 DEBUG [...] 05:39:27 DEBUG 05:39:27 DEBUG 05:39:27 DEBUG 05:39:27 DEBUG "pod": "ds-idrepo-1" 05:39:27 DEBUG }, 05:39:27 DEBUG "value": [ 05:39:27 DEBUG 1712723967, 05:39:27 DEBUG "0" 05:39:27 DEBUG ] 05:39:27 DEBUG } 05:39:27 DEBUG ] 05:39:27 DEBUG } 05:39:27 DEBUG } 05:39:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723968" 05:39:28 INFO [http_cmd]: http status code OK 05:39:28 DEBUG --- status code --- 05:39:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:28 DEBUG --- http response --- 05:39:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:39:28 DEBUG ----- output ----- 05:39:28 DEBUG { 05:39:28 DEBUG "status": "success", 05:39:28 DEBUG "data": { 05:39:28 DEBUG "resultType": "vector", 05:39:28 DEBUG "result": [ 05:39:28 DEBUG { 05:39:28 DEBUG "metric": { 05:39:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:39:28 DEBUG }, 05:39:28 DEBUG "value": [ 05:39:28 DEBUG 05:39:28 DEBUG 05:39:28 DEBUG 05:39:28 DEBUG [...] 05:39:28 DEBUG 05:39:28 DEBUG 05:39:28 DEBUG 05:39:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:39:28 DEBUG }, 05:39:28 DEBUG "value": [ 05:39:28 DEBUG 1712723968, 05:39:28 DEBUG "1261.8333333333333" 05:39:28 DEBUG ] 05:39:28 DEBUG } 05:39:28 DEBUG ] 05:39:28 DEBUG } 05:39:28 DEBUG } 05:39:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723968" 05:39:28 INFO [http_cmd]: http status code OK 05:39:28 DEBUG --- status code --- 05:39:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:28 DEBUG --- http response --- 05:39:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:39:28 DEBUG ----- output ----- 05:39:28 DEBUG { 05:39:28 DEBUG "status": "success", 05:39:28 DEBUG "data": { 05:39:28 DEBUG "resultType": "vector", 05:39:28 DEBUG "result": [ 05:39:28 DEBUG { 05:39:28 DEBUG "metric": { 05:39:28 DEBUG "pod": "ds-idrepo-0" 05:39:28 DEBUG }, 05:39:28 DEBUG "value": [ 05:39:28 DEBUG 05:39:28 DEBUG 05:39:28 DEBUG 05:39:28 DEBUG [...] 05:39:28 DEBUG 05:39:28 DEBUG 05:39:28 DEBUG 05:39:28 DEBUG "pod": "ds-idrepo-1" 05:39:28 DEBUG }, 05:39:28 DEBUG "value": [ 05:39:28 DEBUG 1712723968, 05:39:28 DEBUG "0" 05:39:28 DEBUG ] 05:39:28 DEBUG } 05:39:28 DEBUG ] 05:39:28 DEBUG } 05:39:28 DEBUG } 05:39:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712723968" 05:39:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712723968" 05:39:28 INFO [http_cmd]: http status code OK 05:39:28 DEBUG --- status code --- 05:39:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:28 DEBUG --- http response --- 05:39:28 INFO [http_cmd]: http status code OK 05:39:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:39:28 DEBUG --- status code --- 05:39:28 DEBUG ----- output ----- 05:39:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:28 DEBUG { 05:39:28 DEBUG --- http response --- 05:39:28 DEBUG "status": "success", 05:39:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:39:28 DEBUG "data": { 05:39:28 DEBUG ----- output ----- 05:39:28 DEBUG "resultType": "vector", 05:39:28 DEBUG { 05:39:28 DEBUG "result": [ 05:39:28 DEBUG "status": "success", 05:39:28 DEBUG { 05:39:28 DEBUG "data": { 05:39:28 DEBUG "metric": { 05:39:28 DEBUG "resultType": "vector", 05:39:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:39:28 DEBUG "result": [ 05:39:28 DEBUG }, 05:39:28 DEBUG { 05:39:28 DEBUG "value": [ 05:39:28 DEBUG "metric": { 05:39:28 DEBUG 05:39:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:39:28 DEBUG 05:39:28 DEBUG }, 05:39:28 DEBUG 05:39:28 DEBUG "value": [ 05:39:28 DEBUG [...] 05:39:28 DEBUG 05:39:28 DEBUG 05:39:28 DEBUG 05:39:28 DEBUG 05:39:28 DEBUG 05:39:28 DEBUG 05:39:28 DEBUG [...] 05:39:28 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:39:28 DEBUG 05:39:28 DEBUG }, 05:39:28 DEBUG 05:39:28 DEBUG "value": [ 05:39:28 DEBUG 05:39:28 DEBUG 1712723968, 05:39:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:39:28 DEBUG "0" 05:39:28 DEBUG }, 05:39:28 DEBUG ] 05:39:28 DEBUG "value": [ 05:39:28 DEBUG } 05:39:28 DEBUG 1712723968, 05:39:28 DEBUG ] 05:39:28 DEBUG "2485026816" 05:39:28 DEBUG } 05:39:28 DEBUG ] 05:39:28 DEBUG } 05:39:28 DEBUG } 05:39:28 DEBUG ] 05:39:28 DEBUG } 05:39:28 DEBUG } 05:39:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723968" 05:39:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712723968" 05:39:28 INFO [http_cmd]: http status code OK 05:39:28 DEBUG --- status code --- 05:39:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:28 DEBUG --- http response --- 05:39:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:39:28 DEBUG ----- output ----- 05:39:28 DEBUG { 05:39:28 DEBUG "status": "success", 05:39:28 DEBUG "data": { 05:39:28 DEBUG "resultType": "vector", 05:39:28 DEBUG "result": [ 05:39:28 DEBUG { 05:39:28 DEBUG "metric": { 05:39:28 DEBUG "instance": "10.142.0.25:9100" 05:39:28 DEBUG }, 05:39:28 DEBUG "value": [ 05:39:28 DEBUG 05:39:28 DEBUG 05:39:28 DEBUG 05:39:28 DEBUG [...] 05:39:28 DEBUG 05:39:28 DEBUG 05:39:28 DEBUG 05:39:28 DEBUG "instance": "10.142.0.77:9100" 05:39:28 DEBUG }, 05:39:28 DEBUG "value": [ 05:39:28 DEBUG 1712723968, 05:39:28 DEBUG "37444189.53333333" 05:39:28 DEBUG ] 05:39:28 DEBUG } 05:39:28 DEBUG ] 05:39:28 DEBUG } 05:39:28 DEBUG } 05:39:28 INFO [http_cmd]: http status code OK 05:39:28 DEBUG --- status code --- 05:39:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:28 DEBUG --- http response --- 05:39:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:39:28 DEBUG ----- output ----- 05:39:28 DEBUG { 05:39:28 DEBUG "status": "success", 05:39:28 DEBUG "data": { 05:39:28 DEBUG "resultType": "vector", 05:39:28 DEBUG "result": [ 05:39:28 DEBUG { 05:39:28 DEBUG "metric": { 05:39:28 DEBUG "instance": "10.142.0.25:9100" 05:39:28 DEBUG }, 05:39:28 DEBUG "value": [ 05:39:28 DEBUG 05:39:28 DEBUG 05:39:28 DEBUG 05:39:28 DEBUG [...] 05:39:28 DEBUG 05:39:28 DEBUG 05:39:28 DEBUG 05:39:28 DEBUG "instance": "10.142.0.77:9100" 05:39:28 DEBUG }, 05:39:28 DEBUG "value": [ 05:39:28 DEBUG 1712723968, 05:39:28 DEBUG "0.01566666666666621" 05:39:28 DEBUG ] 05:39:28 DEBUG } 05:39:28 DEBUG ] 05:39:28 DEBUG } 05:39:28 DEBUG } 05:39:43 INFO 05:39:43 INFO [loop_until]: kubectl --namespace=xlou top pods 05:39:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:39:43 INFO [loop_until]: OK (rc = 0) 05:39:43 DEBUG --- stdout --- 05:39:43 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 8824m 8810Mi am-654c4fcbcf-9scrj 8639m 8423Mi am-654c4fcbcf-v5qb5 7346m 8400Mi ds-cts-0 7224m 12769Mi ds-cts-1 7635m 13139Mi ds-cts-2 7003m 13014Mi ds-idrepo-0 9720m 5622Mi ds-idrepo-1 315m 5669Mi ds-idrepo-2 332m 5698Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 2m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3613m 1672Mi 05:39:43 DEBUG --- stderr --- 05:39:43 DEBUG 05:39:50 INFO 05:39:50 INFO [loop_until]: kubectl --namespace=xlou top node 05:39:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:39:50 INFO [loop_until]: OK (rc = 0) 05:39:50 DEBUG --- stdout --- 05:39:50 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 98m 0% 2306Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 7894m 49% 9281Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 86m 0% 3257Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10813m 68% 6516Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9530m 59% 9313Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9888m 62% 9648Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 77m 0% 853Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9781m 61% 6286Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 7489m 47% 13790Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 7098m 44% 13688Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 437m 2% 6368Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7425m 46% 13826Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 372m 2% 6331Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3740m 23% 5341Mi 9% 05:39:50 DEBUG --- stderr --- 05:39:50 DEBUG 05:40:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724024" 05:40:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712724024" 05:40:24 ERROR [http_cmd]: ERROR 05:40:24 ERROR --- status code --- 05:40:24 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:40:24 ERROR --- http response --- 05:40:24 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:40:24 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:40:24 INFO [http_cmd]: http status code OK 05:40:24 DEBUG --- status code --- 05:40:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:24 DEBUG --- http response --- 05:40:24 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:40:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724024" 05:40:24 INFO [http_cmd]: http status code OK 05:40:24 DEBUG --- status code --- 05:40:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:24 DEBUG --- http response --- 05:40:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:40:24 DEBUG ----- output ----- 05:40:24 DEBUG { 05:40:24 DEBUG "status": "success", 05:40:24 DEBUG "data": { 05:40:24 DEBUG "resultType": "vector", 05:40:24 DEBUG "result": [ 05:40:24 DEBUG { 05:40:24 DEBUG "metric": { 05:40:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:40:24 DEBUG }, 05:40:24 DEBUG "value": [ 05:40:24 DEBUG 05:40:24 DEBUG 05:40:24 DEBUG 05:40:24 DEBUG [...] 05:40:24 DEBUG 05:40:24 DEBUG 05:40:24 DEBUG 05:40:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:40:24 DEBUG }, 05:40:24 DEBUG "value": [ 05:40:24 DEBUG 1712724024, 05:40:24 DEBUG "0" 05:40:24 DEBUG ] 05:40:24 DEBUG } 05:40:24 DEBUG ] 05:40:24 DEBUG } 05:40:24 DEBUG } 05:40:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724025" 05:40:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724025" 05:40:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724025" 05:40:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724025" 05:40:25 INFO [http_cmd]: http status code OK 05:40:25 INFO [http_cmd]: http status code OK 05:40:25 DEBUG --- status code --- 05:40:25 DEBUG --- status code --- 05:40:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:25 DEBUG --- http response --- 05:40:25 DEBUG --- http response --- 05:40:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:40:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:40:25 DEBUG ----- output ----- 05:40:25 DEBUG ----- output ----- 05:40:25 DEBUG { 05:40:25 DEBUG { 05:40:25 DEBUG "status": "success", 05:40:25 DEBUG "status": "success", 05:40:25 DEBUG "data": { 05:40:25 DEBUG "data": { 05:40:25 DEBUG "resultType": "vector", 05:40:25 DEBUG "resultType": "vector", 05:40:25 DEBUG "result": [ 05:40:25 DEBUG "result": [ 05:40:25 DEBUG { 05:40:25 DEBUG { 05:40:25 DEBUG "metric": { 05:40:25 DEBUG "metric": { 05:40:25 DEBUG "pod": "ds-idrepo-0" 05:40:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:40:25 INFO [http_cmd]: http status code OK 05:40:25 INFO [http_cmd]: http status code OK 05:40:25 DEBUG }, 05:40:25 DEBUG }, 05:40:25 DEBUG --- status code --- 05:40:25 DEBUG --- status code --- 05:40:25 DEBUG "value": [ 05:40:25 DEBUG "value": [ 05:40:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:25 DEBUG 05:40:25 DEBUG 05:40:25 DEBUG --- http response --- 05:40:25 DEBUG --- http response --- 05:40:25 DEBUG 05:40:25 DEBUG 05:40:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:40:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:40:25 DEBUG 05:40:25 DEBUG 05:40:25 DEBUG ----- output ----- 05:40:25 DEBUG ----- output ----- 05:40:25 DEBUG [...] 05:40:25 DEBUG [...] 05:40:25 DEBUG { 05:40:25 DEBUG { 05:40:25 DEBUG 05:40:25 DEBUG 05:40:25 DEBUG "status": "success", 05:40:25 DEBUG "data": { 05:40:25 DEBUG 05:40:25 DEBUG 05:40:25 DEBUG "status": "success", 05:40:25 DEBUG "resultType": "vector", 05:40:25 DEBUG 05:40:25 DEBUG 05:40:25 DEBUG "data": { 05:40:25 DEBUG "result": [ 05:40:25 DEBUG "pod": "ds-cts-2" 05:40:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:40:25 DEBUG "resultType": "vector", 05:40:25 DEBUG { 05:40:25 DEBUG }, 05:40:25 DEBUG }, 05:40:25 DEBUG "result": [ 05:40:25 DEBUG "metric": { 05:40:25 DEBUG "value": [ 05:40:25 DEBUG "value": [ 05:40:25 DEBUG { 05:40:25 DEBUG "pod": "ds-idrepo-0" 05:40:25 DEBUG 1712724025, 05:40:25 DEBUG "29.933333333333334" 05:40:25 DEBUG "metric": { 05:40:25 DEBUG "instance": "10.142.0.25:9100" 05:40:25 DEBUG ] 05:40:25 DEBUG 1712724025, 05:40:25 DEBUG }, 05:40:25 DEBUG }, 05:40:25 DEBUG } 05:40:25 DEBUG ] 05:40:25 DEBUG "value": [ 05:40:25 DEBUG 05:40:25 DEBUG 05:40:25 DEBUG "1265.2333333333333" 05:40:25 DEBUG 05:40:25 DEBUG "value": [ 05:40:25 DEBUG } 05:40:25 DEBUG } 05:40:25 DEBUG [...] 05:40:25 DEBUG ] 05:40:25 DEBUG 05:40:25 DEBUG 05:40:25 DEBUG } 05:40:25 DEBUG 05:40:25 DEBUG 05:40:25 DEBUG ] 05:40:25 DEBUG 05:40:25 DEBUG 05:40:25 DEBUG } 05:40:25 DEBUG [...] 05:40:25 DEBUG "pod": "ds-cts-2" 05:40:25 DEBUG } 05:40:25 DEBUG 05:40:25 DEBUG }, 05:40:25 DEBUG 05:40:25 DEBUG "value": [ 05:40:25 DEBUG 05:40:25 DEBUG 1712724025, 05:40:25 DEBUG "instance": "10.142.0.77:9100" 05:40:25 DEBUG "0" 05:40:25 DEBUG ] 05:40:25 DEBUG } 05:40:25 DEBUG ] 05:40:25 DEBUG } 05:40:25 DEBUG } 05:40:25 DEBUG }, 05:40:25 DEBUG "value": [ 05:40:25 DEBUG 1712724025, 05:40:25 DEBUG "307206638592" 05:40:25 DEBUG ] 05:40:25 DEBUG } 05:40:25 DEBUG ] 05:40:25 DEBUG } 05:40:25 DEBUG } 05:40:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724025" 05:40:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724025" 05:40:25 INFO [http_cmd]: http status code OK 05:40:25 DEBUG --- status code --- 05:40:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:25 DEBUG --- http response --- 05:40:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:40:25 DEBUG ----- output ----- 05:40:25 DEBUG { 05:40:25 DEBUG "status": "success", 05:40:25 DEBUG "data": { 05:40:25 DEBUG "resultType": "vector", 05:40:25 DEBUG "result": [ 05:40:25 DEBUG { 05:40:25 DEBUG "metric": { 05:40:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:40:25 DEBUG }, 05:40:25 DEBUG "value": [ 05:40:25 DEBUG 05:40:25 DEBUG 05:40:25 DEBUG 05:40:25 DEBUG [...] 05:40:25 DEBUG 05:40:25 DEBUG 05:40:25 DEBUG 05:40:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:40:25 DEBUG }, 05:40:25 DEBUG "value": [ 05:40:25 DEBUG 1712724025, 05:40:25 DEBUG "0" 05:40:25 DEBUG ] 05:40:25 DEBUG } 05:40:25 DEBUG ] 05:40:25 DEBUG } 05:40:25 DEBUG } 05:40:25 INFO [http_cmd]: http status code OK 05:40:25 DEBUG --- status code --- 05:40:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:25 DEBUG --- http response --- 05:40:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:40:25 DEBUG ----- output ----- 05:40:25 DEBUG { 05:40:25 DEBUG "status": "success", 05:40:25 DEBUG "data": { 05:40:25 DEBUG "resultType": "vector", 05:40:25 DEBUG "result": [ 05:40:25 DEBUG { 05:40:25 DEBUG "metric": { 05:40:25 DEBUG "instance": "10.142.0.25:9100" 05:40:25 DEBUG }, 05:40:25 DEBUG "value": [ 05:40:25 DEBUG 05:40:25 DEBUG 05:40:25 DEBUG 05:40:25 DEBUG [...] 05:40:25 DEBUG 05:40:25 DEBUG 05:40:25 DEBUG 05:40:25 DEBUG "instance": "10.142.0.77:9100" 05:40:25 DEBUG }, 05:40:25 DEBUG "value": [ 05:40:25 DEBUG 1712724025, 05:40:25 DEBUG "37499102.92304103" 05:40:25 DEBUG ] 05:40:25 DEBUG } 05:40:25 DEBUG ] 05:40:25 DEBUG } 05:40:25 DEBUG } 05:40:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724025" 05:40:25 INFO [http_cmd]: http status code OK 05:40:25 DEBUG --- status code --- 05:40:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:25 DEBUG --- http response --- 05:40:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:40:25 DEBUG ----- output ----- 05:40:25 DEBUG { 05:40:25 DEBUG "status": "success", 05:40:25 DEBUG "data": { 05:40:25 DEBUG "resultType": "vector", 05:40:25 DEBUG "result": [ 05:40:25 DEBUG { 05:40:25 DEBUG "metric": { 05:40:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:40:25 DEBUG }, 05:40:25 DEBUG "value": [ 05:40:25 DEBUG 05:40:25 DEBUG 05:40:25 DEBUG 05:40:25 DEBUG [...] 05:40:25 DEBUG 05:40:25 DEBUG 05:40:25 DEBUG 05:40:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:40:25 DEBUG }, 05:40:25 DEBUG "value": [ 05:40:25 DEBUG 1712724025, 05:40:25 DEBUG "226837" 05:40:25 DEBUG ] 05:40:25 DEBUG } 05:40:25 DEBUG ] 05:40:25 DEBUG } 05:40:25 DEBUG } 05:40:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724025" 05:40:25 INFO [http_cmd]: http status code OK 05:40:25 DEBUG --- status code --- 05:40:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:25 DEBUG --- http response --- 05:40:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:40:25 DEBUG ----- output ----- 05:40:25 DEBUG { 05:40:25 DEBUG "status": "success", 05:40:25 DEBUG "data": { 05:40:25 DEBUG "resultType": "vector", 05:40:25 DEBUG "result": [ 05:40:25 DEBUG { 05:40:25 DEBUG "metric": { 05:40:25 DEBUG "pod": "ds-idrepo-0" 05:40:25 DEBUG }, 05:40:25 DEBUG "value": [ 05:40:25 DEBUG 05:40:25 DEBUG 05:40:25 DEBUG 05:40:25 DEBUG [...] 05:40:25 DEBUG 05:40:25 DEBUG 05:40:25 DEBUG 05:40:25 DEBUG "pod": "ds-idrepo-1" 05:40:25 DEBUG }, 05:40:25 DEBUG "value": [ 05:40:25 DEBUG 1712724025, 05:40:25 DEBUG "0" 05:40:25 DEBUG ] 05:40:25 DEBUG } 05:40:25 DEBUG ] 05:40:25 DEBUG } 05:40:25 DEBUG } 05:40:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724025" 05:40:25 INFO [http_cmd]: http status code OK 05:40:25 DEBUG --- status code --- 05:40:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:25 DEBUG --- http response --- 05:40:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:40:25 DEBUG ----- output ----- 05:40:25 DEBUG { 05:40:25 DEBUG "status": "success", 05:40:25 DEBUG "data": { 05:40:25 DEBUG "resultType": "vector", 05:40:25 DEBUG "result": [ 05:40:25 DEBUG { 05:40:25 DEBUG "metric": { 05:40:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:40:25 DEBUG }, 05:40:25 DEBUG "value": [ 05:40:25 DEBUG 05:40:25 DEBUG 05:40:25 DEBUG 05:40:25 DEBUG [...] 05:40:25 DEBUG 05:40:25 DEBUG 05:40:25 DEBUG 05:40:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:40:25 DEBUG }, 05:40:25 DEBUG "value": [ 05:40:25 DEBUG 1712724025, 05:40:25 DEBUG "1264.9" 05:40:25 DEBUG ] 05:40:25 DEBUG } 05:40:25 DEBUG ] 05:40:25 DEBUG } 05:40:25 DEBUG } 05:40:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724026" 05:40:26 INFO [http_cmd]: http status code OK 05:40:26 DEBUG --- status code --- 05:40:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:26 DEBUG --- http response --- 05:40:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:40:26 DEBUG ----- output ----- 05:40:26 DEBUG { 05:40:26 DEBUG "status": "success", 05:40:26 DEBUG "data": { 05:40:26 DEBUG "resultType": "vector", 05:40:26 DEBUG "result": [ 05:40:26 DEBUG { 05:40:26 DEBUG "metric": { 05:40:26 DEBUG "instance": "10.142.0.25:9100" 05:40:26 DEBUG }, 05:40:26 DEBUG "value": [ 05:40:26 DEBUG 05:40:26 DEBUG 05:40:26 DEBUG 05:40:26 DEBUG [...] 05:40:26 DEBUG 05:40:26 DEBUG 05:40:26 DEBUG 05:40:26 DEBUG "instance": "10.142.0.77:9100" 05:40:26 DEBUG }, 05:40:26 DEBUG "value": [ 05:40:26 DEBUG 1712724026, 05:40:26 DEBUG "0.14645202146451983" 05:40:26 DEBUG ] 05:40:26 DEBUG } 05:40:26 DEBUG ] 05:40:26 DEBUG } 05:40:26 DEBUG } 05:40:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724026" 05:40:26 INFO [http_cmd]: http status code OK 05:40:26 DEBUG --- status code --- 05:40:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:26 DEBUG --- http response --- 05:40:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:40:26 DEBUG ----- output ----- 05:40:26 DEBUG { 05:40:26 DEBUG "status": "success", 05:40:26 DEBUG "data": { 05:40:26 DEBUG "resultType": "vector", 05:40:26 DEBUG "result": [ 05:40:26 DEBUG { 05:40:26 DEBUG "metric": { 05:40:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:40:26 DEBUG }, 05:40:26 DEBUG "value": [ 05:40:26 DEBUG 05:40:26 DEBUG 05:40:26 DEBUG 05:40:26 DEBUG [...] 05:40:26 DEBUG 05:40:26 DEBUG 05:40:26 DEBUG 05:40:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:40:26 DEBUG }, 05:40:26 DEBUG "value": [ 05:40:26 DEBUG 1712724026, 05:40:26 DEBUG "290169078.4591558" 05:40:26 DEBUG ] 05:40:26 DEBUG } 05:40:26 DEBUG ] 05:40:26 DEBUG } 05:40:26 DEBUG } 05:40:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712724027" 05:40:27 INFO [http_cmd]: http status code OK 05:40:27 DEBUG --- status code --- 05:40:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:27 DEBUG --- http response --- 05:40:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:40:27 DEBUG ----- output ----- 05:40:27 DEBUG { 05:40:27 DEBUG "status": "success", 05:40:27 DEBUG "data": { 05:40:27 DEBUG "resultType": "vector", 05:40:27 DEBUG "result": [ 05:40:27 DEBUG { 05:40:27 DEBUG "metric": { 05:40:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:40:27 DEBUG }, 05:40:27 DEBUG "value": [ 05:40:27 DEBUG 05:40:27 DEBUG 05:40:27 DEBUG 05:40:27 DEBUG [...] 05:40:27 DEBUG 05:40:27 DEBUG 05:40:27 DEBUG 05:40:27 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:40:27 DEBUG }, 05:40:27 DEBUG "value": [ 05:40:27 DEBUG 1712724027, 05:40:27 DEBUG "3.613549116199567" 05:40:27 DEBUG ] 05:40:27 DEBUG } 05:40:27 DEBUG ] 05:40:27 DEBUG } 05:40:27 DEBUG } 05:40:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712724027" 05:40:27 INFO [http_cmd]: http status code OK 05:40:27 DEBUG --- status code --- 05:40:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:27 DEBUG --- http response --- 05:40:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:40:27 DEBUG ----- output ----- 05:40:27 DEBUG { 05:40:27 DEBUG "status": "success", 05:40:27 DEBUG "data": { 05:40:27 DEBUG "resultType": "vector", 05:40:27 DEBUG "result": [ 05:40:27 DEBUG { 05:40:27 DEBUG "metric": { 05:40:27 DEBUG "pod": "ds-idrepo-0" 05:40:27 DEBUG }, 05:40:27 DEBUG "value": [ 05:40:27 DEBUG 05:40:27 DEBUG 05:40:27 DEBUG 05:40:27 DEBUG [...] 05:40:27 DEBUG 05:40:27 DEBUG 05:40:27 DEBUG 05:40:27 DEBUG "pod": "ds-cts-2" 05:40:27 DEBUG }, 05:40:27 DEBUG "value": [ 05:40:27 DEBUG 1712724027, 05:40:27 DEBUG "0" 05:40:27 DEBUG ] 05:40:27 DEBUG } 05:40:27 DEBUG ] 05:40:27 DEBUG } 05:40:27 DEBUG } 05:40:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724027" 05:40:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724027" 05:40:27 INFO [http_cmd]: http status code OK 05:40:27 DEBUG --- status code --- 05:40:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:27 DEBUG --- http response --- 05:40:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:40:27 DEBUG ----- output ----- 05:40:27 DEBUG { 05:40:27 DEBUG "status": "success", 05:40:27 INFO [http_cmd]: http status code OK 05:40:27 DEBUG "data": { 05:40:27 DEBUG --- status code --- 05:40:27 DEBUG "resultType": "vector", 05:40:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:27 DEBUG "result": [ 05:40:27 DEBUG --- http response --- 05:40:27 DEBUG { 05:40:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:40:27 DEBUG "metric": { 05:40:27 DEBUG ----- output ----- 05:40:27 DEBUG "pod": "ds-idrepo-2" 05:40:27 DEBUG { 05:40:27 DEBUG }, 05:40:27 DEBUG "status": "success", 05:40:27 DEBUG "value": [ 05:40:27 DEBUG 05:40:27 DEBUG "data": { 05:40:27 DEBUG 05:40:27 DEBUG "resultType": "vector", 05:40:27 DEBUG 05:40:27 DEBUG "result": [ 05:40:27 DEBUG { 05:40:27 DEBUG [...] 05:40:27 DEBUG "metric": { 05:40:27 DEBUG 05:40:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:40:27 DEBUG }, 05:40:27 DEBUG "value": [ 05:40:27 DEBUG 05:40:27 DEBUG 05:40:27 DEBUG 05:40:27 DEBUG [...] 05:40:27 DEBUG 05:40:27 DEBUG 05:40:27 DEBUG 05:40:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:40:27 DEBUG }, 05:40:27 DEBUG "value": [ 05:40:27 DEBUG 1712724027, 05:40:27 DEBUG "7.3304837709493595" 05:40:27 DEBUG ] 05:40:27 DEBUG } 05:40:27 DEBUG ] 05:40:27 DEBUG } 05:40:27 DEBUG } 05:40:27 DEBUG 05:40:27 DEBUG 05:40:27 DEBUG "pod": "ds-cts-2" 05:40:27 DEBUG }, 05:40:27 DEBUG "value": [ 05:40:27 DEBUG 1712724027, 05:40:27 DEBUG "0.000033333333333333335" 05:40:27 DEBUG ] 05:40:27 DEBUG } 05:40:27 DEBUG ] 05:40:27 DEBUG } 05:40:27 DEBUG } 05:40:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724027" 05:40:27 INFO [http_cmd]: http status code OK 05:40:27 DEBUG --- status code --- 05:40:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:27 DEBUG --- http response --- 05:40:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:40:27 DEBUG ----- output ----- 05:40:27 DEBUG { 05:40:27 DEBUG "status": "success", 05:40:27 DEBUG "data": { 05:40:27 DEBUG "resultType": "vector", 05:40:27 DEBUG "result": [ 05:40:27 DEBUG { 05:40:27 DEBUG "metric": { 05:40:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:40:27 DEBUG }, 05:40:27 DEBUG "value": [ 05:40:27 DEBUG 05:40:27 DEBUG 05:40:27 DEBUG 05:40:27 DEBUG [...] 05:40:27 DEBUG 05:40:27 DEBUG 05:40:27 DEBUG 05:40:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:40:27 DEBUG }, 05:40:27 DEBUG "value": [ 05:40:27 DEBUG 1712724027, 05:40:27 DEBUG "1264.9" 05:40:27 DEBUG ] 05:40:27 DEBUG } 05:40:27 DEBUG ] 05:40:27 DEBUG } 05:40:27 DEBUG } 05:40:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712724027" 05:40:27 INFO [http_cmd]: http status code OK 05:40:27 DEBUG --- status code --- 05:40:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:27 DEBUG --- http response --- 05:40:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:40:27 DEBUG ----- output ----- 05:40:27 DEBUG { 05:40:27 DEBUG "status": "success", 05:40:27 DEBUG "data": { 05:40:27 DEBUG "resultType": "vector", 05:40:27 DEBUG "result": [ 05:40:27 DEBUG { 05:40:27 DEBUG "metric": { 05:40:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:40:27 DEBUG }, 05:40:27 DEBUG "value": [ 05:40:27 DEBUG 05:40:27 DEBUG 05:40:27 DEBUG 05:40:27 DEBUG [...] 05:40:27 DEBUG 05:40:27 DEBUG 05:40:27 DEBUG 05:40:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:40:27 DEBUG }, 05:40:27 DEBUG "value": [ 05:40:27 DEBUG 1712724027, 05:40:27 DEBUG "0.043" 05:40:27 DEBUG ] 05:40:27 DEBUG } 05:40:27 DEBUG ] 05:40:27 DEBUG } 05:40:27 DEBUG } 05:40:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724027" 05:40:27 INFO [http_cmd]: http status code OK 05:40:27 DEBUG --- status code --- 05:40:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:27 DEBUG --- http response --- 05:40:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:40:27 DEBUG ----- output ----- 05:40:27 DEBUG { 05:40:27 DEBUG "status": "success", 05:40:27 DEBUG "data": { 05:40:27 DEBUG "resultType": "vector", 05:40:27 DEBUG "result": [ 05:40:27 DEBUG { 05:40:27 DEBUG "metric": { 05:40:27 DEBUG "pod": "ds-idrepo-0" 05:40:27 DEBUG }, 05:40:27 DEBUG "value": [ 05:40:27 DEBUG 05:40:27 DEBUG 05:40:27 DEBUG 05:40:27 DEBUG [...] 05:40:27 DEBUG 05:40:27 DEBUG 05:40:27 DEBUG 05:40:27 DEBUG "pod": "ds-idrepo-1" 05:40:27 DEBUG }, 05:40:27 DEBUG "value": [ 05:40:27 DEBUG 1712724027, 05:40:27 DEBUG "0" 05:40:27 DEBUG ] 05:40:27 DEBUG } 05:40:27 DEBUG ] 05:40:27 DEBUG } 05:40:27 DEBUG } 05:40:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724028" 05:40:28 INFO [http_cmd]: http status code OK 05:40:28 DEBUG --- status code --- 05:40:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:28 DEBUG --- http response --- 05:40:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:40:28 DEBUG ----- output ----- 05:40:28 DEBUG { 05:40:28 DEBUG "status": "success", 05:40:28 DEBUG "data": { 05:40:28 DEBUG "resultType": "vector", 05:40:28 DEBUG "result": [ 05:40:28 DEBUG { 05:40:28 DEBUG "metric": { 05:40:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:40:28 DEBUG }, 05:40:28 DEBUG "value": [ 05:40:28 DEBUG 05:40:28 DEBUG 05:40:28 DEBUG 05:40:28 DEBUG [...] 05:40:28 DEBUG 05:40:28 DEBUG 05:40:28 DEBUG 05:40:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:40:28 DEBUG }, 05:40:28 DEBUG "value": [ 05:40:28 DEBUG 1712724028, 05:40:28 DEBUG "1254.1333333333332" 05:40:28 DEBUG ] 05:40:28 DEBUG } 05:40:28 DEBUG ] 05:40:28 DEBUG } 05:40:28 DEBUG } 05:40:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724028" 05:40:28 INFO [http_cmd]: http status code OK 05:40:28 DEBUG --- status code --- 05:40:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:28 DEBUG --- http response --- 05:40:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:40:28 DEBUG ----- output ----- 05:40:28 DEBUG { 05:40:28 DEBUG "status": "success", 05:40:28 DEBUG "data": { 05:40:28 DEBUG "resultType": "vector", 05:40:28 DEBUG "result": [ 05:40:28 DEBUG { 05:40:28 DEBUG "metric": { 05:40:28 DEBUG "pod": "ds-idrepo-0" 05:40:28 DEBUG }, 05:40:28 DEBUG "value": [ 05:40:28 DEBUG 05:40:28 DEBUG 05:40:28 DEBUG 05:40:28 DEBUG [...] 05:40:28 DEBUG 05:40:28 DEBUG 05:40:28 DEBUG 05:40:28 DEBUG "pod": "ds-idrepo-1" 05:40:28 DEBUG }, 05:40:28 DEBUG "value": [ 05:40:28 DEBUG 1712724028, 05:40:28 DEBUG "0" 05:40:28 DEBUG ] 05:40:28 DEBUG } 05:40:28 DEBUG ] 05:40:28 DEBUG } 05:40:28 DEBUG } 05:40:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712724028" 05:40:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724028" 05:40:28 INFO [http_cmd]: http status code OK 05:40:28 INFO [http_cmd]: http status code OK 05:40:28 DEBUG --- status code --- 05:40:28 DEBUG --- status code --- 05:40:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:28 DEBUG --- http response --- 05:40:28 DEBUG --- http response --- 05:40:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:40:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:40:28 DEBUG ----- output ----- 05:40:28 DEBUG ----- output ----- 05:40:28 DEBUG { 05:40:28 DEBUG { 05:40:28 DEBUG "status": "success", 05:40:28 DEBUG "status": "success", 05:40:28 DEBUG "data": { 05:40:28 DEBUG "data": { 05:40:28 DEBUG "resultType": "vector", 05:40:28 DEBUG "resultType": "vector", 05:40:28 DEBUG "result": [ 05:40:28 DEBUG "result": [ 05:40:28 DEBUG { 05:40:28 DEBUG { 05:40:28 DEBUG "metric": { 05:40:28 DEBUG "metric": { 05:40:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:40:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:40:28 DEBUG }, 05:40:28 DEBUG }, 05:40:28 DEBUG "value": [ 05:40:28 DEBUG "value": [ 05:40:28 DEBUG 05:40:28 DEBUG 05:40:28 DEBUG 05:40:28 DEBUG 05:40:28 DEBUG 05:40:28 DEBUG 05:40:28 DEBUG [...] 05:40:28 DEBUG [...] 05:40:28 DEBUG 05:40:28 DEBUG 05:40:28 DEBUG 05:40:28 DEBUG 05:40:28 DEBUG 05:40:28 DEBUG 05:40:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:40:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:40:28 DEBUG }, 05:40:28 DEBUG }, 05:40:28 DEBUG "value": [ 05:40:28 DEBUG "value": [ 05:40:28 DEBUG 1712724028, 05:40:28 DEBUG 1712724028, 05:40:28 DEBUG "2488029184" 05:40:28 DEBUG "0" 05:40:28 DEBUG ] 05:40:28 DEBUG ] 05:40:28 DEBUG } 05:40:28 DEBUG } 05:40:28 DEBUG ] 05:40:28 DEBUG ] 05:40:28 DEBUG } 05:40:28 DEBUG } 05:40:28 DEBUG } 05:40:28 DEBUG } 05:40:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724028" 05:40:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724028" 05:40:28 INFO [http_cmd]: http status code OK 05:40:28 DEBUG --- status code --- 05:40:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:28 DEBUG --- http response --- 05:40:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:40:28 DEBUG ----- output ----- 05:40:28 DEBUG { 05:40:28 DEBUG "status": "success", 05:40:28 DEBUG "data": { 05:40:28 DEBUG "resultType": "vector", 05:40:28 DEBUG "result": [ 05:40:28 DEBUG { 05:40:28 DEBUG "metric": { 05:40:28 DEBUG "instance": "10.142.0.25:9100" 05:40:28 DEBUG }, 05:40:28 DEBUG "value": [ 05:40:28 DEBUG 05:40:28 DEBUG 05:40:28 DEBUG 05:40:28 DEBUG [...] 05:40:28 DEBUG 05:40:28 DEBUG 05:40:28 DEBUG 05:40:28 DEBUG "instance": "10.142.0.77:9100" 05:40:28 DEBUG }, 05:40:28 DEBUG "value": [ 05:40:28 DEBUG 1712724028, 05:40:28 DEBUG "37501170.0163317" 05:40:28 DEBUG ] 05:40:28 DEBUG } 05:40:28 DEBUG ] 05:40:28 DEBUG } 05:40:28 DEBUG } 05:40:28 INFO [http_cmd]: http status code OK 05:40:28 DEBUG --- status code --- 05:40:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:28 DEBUG --- http response --- 05:40:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:40:28 DEBUG ----- output ----- 05:40:28 DEBUG { 05:40:28 DEBUG "status": "success", 05:40:28 DEBUG "data": { 05:40:28 DEBUG "resultType": "vector", 05:40:28 DEBUG "result": [ 05:40:28 DEBUG { 05:40:28 DEBUG "metric": { 05:40:28 DEBUG "instance": "10.142.0.25:9100" 05:40:28 DEBUG }, 05:40:28 DEBUG "value": [ 05:40:28 DEBUG 05:40:28 DEBUG 05:40:28 DEBUG 05:40:28 DEBUG [...] 05:40:28 DEBUG 05:40:28 DEBUG 05:40:28 DEBUG 05:40:28 DEBUG "instance": "10.142.0.77:9100" 05:40:28 DEBUG }, 05:40:28 DEBUG "value": [ 05:40:28 DEBUG 1712724028, 05:40:28 DEBUG "0.04666200046662044" 05:40:28 DEBUG ] 05:40:28 DEBUG } 05:40:28 DEBUG ] 05:40:28 DEBUG } 05:40:28 DEBUG } 05:40:43 INFO 05:40:43 INFO [loop_until]: kubectl --namespace=xlou top pods 05:40:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:40:43 INFO [loop_until]: OK (rc = 0) 05:40:43 DEBUG --- stdout --- 05:40:43 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 9006m 8804Mi am-654c4fcbcf-9scrj 8813m 8422Mi am-654c4fcbcf-v5qb5 7358m 8400Mi ds-cts-0 7509m 12615Mi ds-cts-1 7282m 12644Mi ds-cts-2 7082m 13273Mi ds-idrepo-0 9631m 5622Mi ds-idrepo-1 336m 5669Mi ds-idrepo-2 358m 5698Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 5m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3521m 1673Mi 05:40:43 DEBUG --- stderr --- 05:40:43 DEBUG 05:40:50 INFO 05:40:50 INFO [loop_until]: kubectl --namespace=xlou top node 05:40:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:40:50 INFO [loop_until]: OK (rc = 0) 05:40:50 DEBUG --- stdout --- 05:40:50 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 95m 0% 2306Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8171m 51% 9281Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 82m 0% 3260Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10801m 67% 6518Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9249m 58% 9313Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9938m 62% 9647Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 77m 0% 854Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9720m 61% 6285Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 7479m 47% 13258Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 7143m 44% 13942Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 415m 2% 6368Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7608m 47% 13258Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 376m 2% 6333Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3703m 23% 5345Mi 9% 05:40:50 DEBUG --- stderr --- 05:40:50 DEBUG 05:41:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712724084" 05:41:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724084" 05:41:24 INFO [http_cmd]: http status code OK 05:41:24 DEBUG --- status code --- 05:41:24 ERROR [http_cmd]: ERROR 05:41:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:24 ERROR --- status code --- 05:41:24 DEBUG --- http response --- 05:41:24 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:41:24 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:41:24 ERROR --- http response --- 05:41:24 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:41:24 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:41:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724084" 05:41:24 INFO [http_cmd]: http status code OK 05:41:24 DEBUG --- status code --- 05:41:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:24 DEBUG --- http response --- 05:41:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:41:24 DEBUG ----- output ----- 05:41:24 DEBUG { 05:41:24 DEBUG "status": "success", 05:41:24 DEBUG "data": { 05:41:24 DEBUG "resultType": "vector", 05:41:24 DEBUG "result": [ 05:41:24 DEBUG { 05:41:24 DEBUG "metric": { 05:41:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:41:24 DEBUG }, 05:41:24 DEBUG "value": [ 05:41:24 DEBUG 05:41:24 DEBUG 05:41:24 DEBUG 05:41:24 DEBUG [...] 05:41:24 DEBUG 05:41:24 DEBUG 05:41:24 DEBUG 05:41:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:41:24 DEBUG }, 05:41:24 DEBUG "value": [ 05:41:24 DEBUG 1712724084, 05:41:24 DEBUG "0" 05:41:24 DEBUG ] 05:41:24 DEBUG } 05:41:24 DEBUG ] 05:41:24 DEBUG } 05:41:24 DEBUG } 05:41:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724085" 05:41:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724085" 05:41:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724085" 05:41:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724085" 05:41:25 INFO [http_cmd]: http status code OK 05:41:25 INFO [http_cmd]: http status code OK 05:41:25 DEBUG --- status code --- 05:41:25 INFO [http_cmd]: http status code OK 05:41:25 DEBUG --- status code --- 05:41:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:25 DEBUG --- status code --- 05:41:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:25 DEBUG --- http response --- 05:41:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:25 DEBUG --- http response --- 05:41:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:41:25 DEBUG --- http response --- 05:41:25 INFO [http_cmd]: http status code OK 05:41:25 DEBUG ----- output ----- 05:41:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:41:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:41:25 DEBUG --- status code --- 05:41:25 DEBUG { 05:41:25 DEBUG ----- output ----- 05:41:25 DEBUG ----- output ----- 05:41:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:25 DEBUG "status": "success", 05:41:25 DEBUG { 05:41:25 DEBUG { 05:41:25 DEBUG --- http response --- 05:41:25 DEBUG "data": { 05:41:25 DEBUG "status": "success", 05:41:25 DEBUG "status": "success", 05:41:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:41:25 DEBUG ----- output ----- 05:41:25 DEBUG "data": { 05:41:25 DEBUG "data": { 05:41:25 DEBUG "resultType": "vector", 05:41:25 DEBUG { 05:41:25 DEBUG "resultType": "vector", 05:41:25 DEBUG "resultType": "vector", 05:41:25 DEBUG "result": [ 05:41:25 DEBUG "status": "success", 05:41:25 DEBUG "result": [ 05:41:25 DEBUG { 05:41:25 DEBUG "metric": { 05:41:25 DEBUG "data": { 05:41:25 DEBUG "result": [ 05:41:25 DEBUG { 05:41:25 DEBUG "metric": { 05:41:25 DEBUG "resultType": "vector", 05:41:25 DEBUG { 05:41:25 DEBUG "instance": "10.142.0.25:9100" 05:41:25 DEBUG "pod": "ds-idrepo-0" 05:41:25 DEBUG "result": [ 05:41:25 DEBUG "metric": { 05:41:25 DEBUG }, 05:41:25 DEBUG }, 05:41:25 DEBUG { 05:41:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:41:25 DEBUG "value": [ 05:41:25 DEBUG "value": [ 05:41:25 DEBUG "metric": { 05:41:25 DEBUG }, 05:41:25 DEBUG 05:41:25 DEBUG 05:41:25 DEBUG "pod": "ds-idrepo-0" 05:41:25 DEBUG "value": [ 05:41:25 DEBUG 05:41:25 DEBUG 05:41:25 DEBUG }, 05:41:25 DEBUG 05:41:25 DEBUG 05:41:25 DEBUG 05:41:25 DEBUG "value": [ 05:41:25 DEBUG 05:41:25 DEBUG [...] 05:41:25 DEBUG [...] 05:41:25 DEBUG 05:41:25 DEBUG 05:41:25 DEBUG 05:41:25 DEBUG 05:41:25 DEBUG 05:41:25 DEBUG [...] 05:41:25 DEBUG 05:41:25 DEBUG 05:41:25 DEBUG 05:41:25 DEBUG 05:41:25 DEBUG 05:41:25 DEBUG 05:41:25 DEBUG [...] 05:41:25 DEBUG 05:41:25 DEBUG "instance": "10.142.0.77:9100" 05:41:25 DEBUG "pod": "ds-cts-2" 05:41:25 DEBUG 05:41:25 DEBUG 05:41:25 DEBUG }, 05:41:25 DEBUG }, 05:41:25 DEBUG 05:41:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:41:25 DEBUG "value": [ 05:41:25 DEBUG "value": [ 05:41:25 DEBUG 05:41:25 DEBUG }, 05:41:25 DEBUG 1712724085, 05:41:25 DEBUG 1712724085, 05:41:25 DEBUG "pod": "ds-cts-2" 05:41:25 DEBUG "value": [ 05:41:25 DEBUG "308919309312" 05:41:25 DEBUG "0" 05:41:25 DEBUG }, 05:41:25 DEBUG 1712724085, 05:41:25 DEBUG ] 05:41:25 DEBUG ] 05:41:25 DEBUG "value": [ 05:41:25 DEBUG "1256.6" 05:41:25 DEBUG } 05:41:25 DEBUG } 05:41:25 DEBUG 1712724085, 05:41:25 DEBUG ] 05:41:25 DEBUG ] 05:41:25 DEBUG ] 05:41:25 DEBUG "29.333333333333332" 05:41:25 DEBUG } 05:41:25 DEBUG } 05:41:25 DEBUG } 05:41:25 DEBUG ] 05:41:25 DEBUG ] 05:41:25 DEBUG } 05:41:25 DEBUG } 05:41:25 DEBUG } 05:41:25 DEBUG } 05:41:25 DEBUG ] 05:41:25 DEBUG } 05:41:25 DEBUG } 05:41:25 DEBUG } 05:41:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724085" 05:41:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724085" 05:41:25 INFO [http_cmd]: http status code OK 05:41:25 DEBUG --- status code --- 05:41:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:25 DEBUG --- http response --- 05:41:25 INFO [http_cmd]: http status code OK 05:41:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:41:25 DEBUG --- status code --- 05:41:25 DEBUG ----- output ----- 05:41:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:25 DEBUG { 05:41:25 DEBUG --- http response --- 05:41:25 DEBUG "status": "success", 05:41:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:41:25 DEBUG "data": { 05:41:25 DEBUG ----- output ----- 05:41:25 DEBUG "resultType": "vector", 05:41:25 DEBUG { 05:41:25 DEBUG "result": [ 05:41:25 DEBUG "status": "success", 05:41:25 DEBUG { 05:41:25 DEBUG "data": { 05:41:25 DEBUG "metric": { 05:41:25 DEBUG "resultType": "vector", 05:41:25 DEBUG "instance": "10.142.0.25:9100" 05:41:25 DEBUG "result": [ 05:41:25 DEBUG }, 05:41:25 DEBUG { 05:41:25 DEBUG "value": [ 05:41:25 DEBUG "metric": { 05:41:25 DEBUG 05:41:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:41:25 DEBUG 05:41:25 DEBUG }, 05:41:25 DEBUG 05:41:25 DEBUG "value": [ 05:41:25 DEBUG [...] 05:41:25 DEBUG 05:41:25 DEBUG 05:41:25 DEBUG 05:41:25 DEBUG 05:41:25 DEBUG 05:41:25 DEBUG 05:41:25 DEBUG [...] 05:41:25 DEBUG "instance": "10.142.0.77:9100" 05:41:25 DEBUG 05:41:25 DEBUG }, 05:41:25 DEBUG 05:41:25 DEBUG "value": [ 05:41:25 DEBUG 05:41:25 DEBUG 1712724085, 05:41:25 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:41:25 DEBUG "37591899" 05:41:25 DEBUG }, 05:41:25 DEBUG ] 05:41:25 DEBUG "value": [ 05:41:25 DEBUG } 05:41:25 DEBUG 1712724085, 05:41:25 DEBUG ] 05:41:25 DEBUG "0" 05:41:25 DEBUG } 05:41:25 DEBUG ] 05:41:25 DEBUG } 05:41:25 DEBUG } 05:41:25 DEBUG ] 05:41:25 DEBUG } 05:41:25 DEBUG } 05:41:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724085" 05:41:25 INFO [http_cmd]: http status code OK 05:41:25 DEBUG --- status code --- 05:41:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:25 DEBUG --- http response --- 05:41:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:41:25 DEBUG ----- output ----- 05:41:25 DEBUG { 05:41:25 DEBUG "status": "success", 05:41:25 DEBUG "data": { 05:41:25 DEBUG "resultType": "vector", 05:41:25 DEBUG "result": [ 05:41:25 DEBUG { 05:41:25 DEBUG "metric": { 05:41:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:41:25 DEBUG }, 05:41:25 DEBUG "value": [ 05:41:25 DEBUG 05:41:25 DEBUG 05:41:25 DEBUG 05:41:25 DEBUG [...] 05:41:25 DEBUG 05:41:25 DEBUG 05:41:25 DEBUG 05:41:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:41:25 DEBUG }, 05:41:25 DEBUG "value": [ 05:41:25 DEBUG 1712724085, 05:41:25 DEBUG "227022" 05:41:25 DEBUG ] 05:41:25 DEBUG } 05:41:25 DEBUG ] 05:41:25 DEBUG } 05:41:25 DEBUG } 05:41:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724085" 05:41:25 INFO [http_cmd]: http status code OK 05:41:25 DEBUG --- status code --- 05:41:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:25 DEBUG --- http response --- 05:41:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:41:25 DEBUG ----- output ----- 05:41:25 DEBUG { 05:41:25 DEBUG "status": "success", 05:41:25 DEBUG "data": { 05:41:25 DEBUG "resultType": "vector", 05:41:25 DEBUG "result": [ 05:41:25 DEBUG { 05:41:25 DEBUG "metric": { 05:41:25 DEBUG "pod": "ds-idrepo-0" 05:41:25 DEBUG }, 05:41:25 DEBUG "value": [ 05:41:25 DEBUG 05:41:25 DEBUG 05:41:25 DEBUG 05:41:25 DEBUG [...] 05:41:25 DEBUG 05:41:25 DEBUG 05:41:25 DEBUG 05:41:25 DEBUG "pod": "ds-idrepo-1" 05:41:25 DEBUG }, 05:41:25 DEBUG "value": [ 05:41:25 DEBUG 1712724085, 05:41:25 DEBUG "0" 05:41:25 DEBUG ] 05:41:25 DEBUG } 05:41:25 DEBUG ] 05:41:25 DEBUG } 05:41:25 DEBUG } 05:41:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724085" 05:41:25 INFO [http_cmd]: http status code OK 05:41:25 DEBUG --- status code --- 05:41:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:25 DEBUG --- http response --- 05:41:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:41:25 DEBUG ----- output ----- 05:41:25 DEBUG { 05:41:25 DEBUG "status": "success", 05:41:25 DEBUG "data": { 05:41:25 DEBUG "resultType": "vector", 05:41:25 DEBUG "result": [ 05:41:25 DEBUG { 05:41:25 DEBUG "metric": { 05:41:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:41:25 DEBUG }, 05:41:25 DEBUG "value": [ 05:41:25 DEBUG 05:41:25 DEBUG 05:41:25 DEBUG 05:41:25 DEBUG [...] 05:41:25 DEBUG 05:41:25 DEBUG 05:41:25 DEBUG 05:41:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:41:25 DEBUG }, 05:41:25 DEBUG "value": [ 05:41:25 DEBUG 1712724085, 05:41:25 DEBUG "1254.5" 05:41:25 DEBUG ] 05:41:25 DEBUG } 05:41:25 DEBUG ] 05:41:25 DEBUG } 05:41:25 DEBUG } 05:41:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724086" 05:41:26 INFO [http_cmd]: http status code OK 05:41:26 DEBUG --- status code --- 05:41:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:26 DEBUG --- http response --- 05:41:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:41:26 DEBUG ----- output ----- 05:41:26 DEBUG { 05:41:26 DEBUG "status": "success", 05:41:26 DEBUG "data": { 05:41:26 DEBUG "resultType": "vector", 05:41:26 DEBUG "result": [ 05:41:26 DEBUG { 05:41:26 DEBUG "metric": { 05:41:26 DEBUG "instance": "10.142.0.25:9100" 05:41:26 DEBUG }, 05:41:26 DEBUG "value": [ 05:41:26 DEBUG 05:41:26 DEBUG 05:41:26 DEBUG 05:41:26 DEBUG [...] 05:41:26 DEBUG 05:41:26 DEBUG 05:41:26 DEBUG 05:41:26 DEBUG "instance": "10.142.0.77:9100" 05:41:26 DEBUG }, 05:41:26 DEBUG "value": [ 05:41:26 DEBUG 1712724086, 05:41:26 DEBUG "0.14723333333333813" 05:41:26 DEBUG ] 05:41:26 DEBUG } 05:41:26 DEBUG ] 05:41:26 DEBUG } 05:41:26 DEBUG } 05:41:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724086" 05:41:26 INFO [http_cmd]: http status code OK 05:41:26 DEBUG --- status code --- 05:41:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:26 DEBUG --- http response --- 05:41:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:41:26 DEBUG ----- output ----- 05:41:26 DEBUG { 05:41:26 DEBUG "status": "success", 05:41:26 DEBUG "data": { 05:41:26 DEBUG "resultType": "vector", 05:41:26 DEBUG "result": [ 05:41:26 DEBUG { 05:41:26 DEBUG "metric": { 05:41:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:41:26 DEBUG }, 05:41:26 DEBUG "value": [ 05:41:26 DEBUG 05:41:26 DEBUG 05:41:26 DEBUG 05:41:26 DEBUG [...] 05:41:26 DEBUG 05:41:26 DEBUG 05:41:26 DEBUG 05:41:26 DEBUG "pod": "ds-idrepo-1" 05:41:26 DEBUG }, 05:41:26 DEBUG "value": [ 05:41:26 DEBUG 1712724086, 05:41:26 DEBUG "0" 05:41:26 DEBUG ] 05:41:26 DEBUG } 05:41:26 DEBUG ] 05:41:26 DEBUG } 05:41:26 DEBUG } 05:41:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712724087" 05:41:27 INFO [http_cmd]: http status code OK 05:41:27 DEBUG --- status code --- 05:41:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:27 DEBUG --- http response --- 05:41:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:41:27 DEBUG ----- output ----- 05:41:27 DEBUG { 05:41:27 DEBUG "status": "success", 05:41:27 DEBUG "data": { 05:41:27 DEBUG "resultType": "vector", 05:41:27 DEBUG "result": [ 05:41:27 DEBUG { 05:41:27 DEBUG "metric": { 05:41:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:41:27 DEBUG }, 05:41:27 DEBUG "value": [ 05:41:27 DEBUG 05:41:27 DEBUG 05:41:27 DEBUG 05:41:27 DEBUG [...] 05:41:27 DEBUG 05:41:27 DEBUG 05:41:27 DEBUG 05:41:27 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:41:27 DEBUG }, 05:41:27 DEBUG "value": [ 05:41:27 DEBUG 1712724087, 05:41:27 DEBUG "3.588209084776081" 05:41:27 DEBUG ] 05:41:27 DEBUG } 05:41:27 DEBUG ] 05:41:27 DEBUG } 05:41:27 DEBUG } 05:41:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712724087" 05:41:27 INFO [http_cmd]: http status code OK 05:41:27 DEBUG --- status code --- 05:41:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:27 DEBUG --- http response --- 05:41:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:41:27 DEBUG ----- output ----- 05:41:27 DEBUG { 05:41:27 DEBUG "status": "success", 05:41:27 DEBUG "data": { 05:41:27 DEBUG "resultType": "vector", 05:41:27 DEBUG "result": [ 05:41:27 DEBUG { 05:41:27 DEBUG "metric": { 05:41:27 DEBUG "pod": "ds-idrepo-0" 05:41:27 DEBUG }, 05:41:27 DEBUG "value": [ 05:41:27 DEBUG 05:41:27 DEBUG 05:41:27 DEBUG 05:41:27 DEBUG [...] 05:41:27 DEBUG 05:41:27 DEBUG 05:41:27 DEBUG 05:41:27 DEBUG "pod": "ds-cts-2" 05:41:27 DEBUG }, 05:41:27 DEBUG "value": [ 05:41:27 DEBUG 1712724087, 05:41:27 DEBUG "0" 05:41:27 DEBUG ] 05:41:27 DEBUG } 05:41:27 DEBUG ] 05:41:27 DEBUG } 05:41:27 DEBUG } 05:41:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724087" 05:41:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724087" 05:41:27 INFO [http_cmd]: http status code OK 05:41:27 INFO [http_cmd]: http status code OK 05:41:27 DEBUG --- status code --- 05:41:27 DEBUG --- status code --- 05:41:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:27 DEBUG --- http response --- 05:41:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:41:27 DEBUG ----- output ----- 05:41:27 DEBUG { 05:41:27 DEBUG "status": "success", 05:41:27 DEBUG "data": { 05:41:27 DEBUG "resultType": "vector", 05:41:27 DEBUG "result": [ 05:41:27 DEBUG { 05:41:27 DEBUG "metric": { 05:41:27 DEBUG "pod": "ds-idrepo-2" 05:41:27 DEBUG }, 05:41:27 DEBUG "value": [ 05:41:27 DEBUG 05:41:27 DEBUG 05:41:27 DEBUG 05:41:27 DEBUG [...] 05:41:27 DEBUG 05:41:27 DEBUG 05:41:27 DEBUG 05:41:27 DEBUG "pod": "ds-cts-2" 05:41:27 DEBUG }, 05:41:27 DEBUG "value": [ 05:41:27 DEBUG 1712724087, 05:41:27 DEBUG "0.000033333333333333335" 05:41:27 DEBUG ] 05:41:27 DEBUG } 05:41:27 DEBUG ] 05:41:27 DEBUG } 05:41:27 DEBUG } 05:41:27 DEBUG --- http response --- 05:41:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:41:27 DEBUG ----- output ----- 05:41:27 DEBUG { 05:41:27 DEBUG "status": "success", 05:41:27 DEBUG "data": { 05:41:27 DEBUG "resultType": "vector", 05:41:27 DEBUG "result": [ 05:41:27 DEBUG { 05:41:27 DEBUG "metric": { 05:41:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:41:27 DEBUG }, 05:41:27 DEBUG "value": [ 05:41:27 DEBUG 05:41:27 DEBUG 05:41:27 DEBUG 05:41:27 DEBUG [...] 05:41:27 DEBUG 05:41:27 DEBUG 05:41:27 DEBUG 05:41:27 DEBUG "pod": "ds-idrepo-1" 05:41:27 DEBUG }, 05:41:27 DEBUG "value": [ 05:41:27 DEBUG 1712724087, 05:41:27 DEBUG "0" 05:41:27 DEBUG ] 05:41:27 DEBUG } 05:41:27 DEBUG ] 05:41:27 DEBUG } 05:41:27 DEBUG } 05:41:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724087" 05:41:27 INFO [http_cmd]: http status code OK 05:41:27 DEBUG --- status code --- 05:41:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:27 DEBUG --- http response --- 05:41:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:41:27 DEBUG ----- output ----- 05:41:27 DEBUG { 05:41:27 DEBUG "status": "success", 05:41:27 DEBUG "data": { 05:41:27 DEBUG "resultType": "vector", 05:41:27 DEBUG "result": [ 05:41:27 DEBUG { 05:41:27 DEBUG "metric": { 05:41:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:41:27 DEBUG }, 05:41:27 DEBUG "value": [ 05:41:27 DEBUG 05:41:27 DEBUG 05:41:27 DEBUG 05:41:27 DEBUG [...] 05:41:27 DEBUG 05:41:27 DEBUG 05:41:27 DEBUG 05:41:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:41:27 DEBUG }, 05:41:27 DEBUG "value": [ 05:41:27 DEBUG 1712724087, 05:41:27 DEBUG "1254.2666666666667" 05:41:27 DEBUG ] 05:41:27 DEBUG } 05:41:27 DEBUG ] 05:41:27 DEBUG } 05:41:27 DEBUG } 05:41:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712724087" 05:41:27 INFO [http_cmd]: http status code OK 05:41:27 DEBUG --- status code --- 05:41:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:27 DEBUG --- http response --- 05:41:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:41:27 DEBUG ----- output ----- 05:41:27 DEBUG { 05:41:27 DEBUG "status": "success", 05:41:27 DEBUG "data": { 05:41:27 DEBUG "resultType": "vector", 05:41:27 DEBUG "result": [ 05:41:27 DEBUG { 05:41:27 DEBUG "metric": { 05:41:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:41:27 DEBUG }, 05:41:27 DEBUG "value": [ 05:41:27 DEBUG 05:41:27 DEBUG 05:41:27 DEBUG 05:41:27 DEBUG [...] 05:41:27 DEBUG 05:41:27 DEBUG 05:41:27 DEBUG 05:41:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:41:27 DEBUG }, 05:41:27 DEBUG "value": [ 05:41:27 DEBUG 1712724087, 05:41:27 DEBUG "0.043" 05:41:27 DEBUG ] 05:41:27 DEBUG } 05:41:27 DEBUG ] 05:41:27 DEBUG } 05:41:27 DEBUG } 05:41:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724088" 05:41:28 INFO [http_cmd]: http status code OK 05:41:28 DEBUG --- status code --- 05:41:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:28 DEBUG --- http response --- 05:41:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:41:28 DEBUG ----- output ----- 05:41:28 DEBUG { 05:41:28 DEBUG "status": "success", 05:41:28 DEBUG "data": { 05:41:28 DEBUG "resultType": "vector", 05:41:28 DEBUG "result": [ 05:41:28 DEBUG { 05:41:28 DEBUG "metric": { 05:41:28 DEBUG "pod": "ds-idrepo-0" 05:41:28 DEBUG }, 05:41:28 DEBUG "value": [ 05:41:28 DEBUG 05:41:28 DEBUG 05:41:28 DEBUG 05:41:28 DEBUG [...] 05:41:28 DEBUG 05:41:28 DEBUG 05:41:28 DEBUG 05:41:28 DEBUG "pod": "ds-idrepo-1" 05:41:28 DEBUG }, 05:41:28 DEBUG "value": [ 05:41:28 DEBUG 1712724088, 05:41:28 DEBUG "0" 05:41:28 DEBUG ] 05:41:28 DEBUG } 05:41:28 DEBUG ] 05:41:28 DEBUG } 05:41:28 DEBUG } 05:41:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724088" 05:41:28 INFO [http_cmd]: http status code OK 05:41:28 DEBUG --- status code --- 05:41:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:28 DEBUG --- http response --- 05:41:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:41:28 DEBUG ----- output ----- 05:41:28 DEBUG { 05:41:28 DEBUG "status": "success", 05:41:28 DEBUG "data": { 05:41:28 DEBUG "resultType": "vector", 05:41:28 DEBUG "result": [ 05:41:28 DEBUG { 05:41:28 DEBUG "metric": { 05:41:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:41:28 DEBUG }, 05:41:28 DEBUG "value": [ 05:41:28 DEBUG 05:41:28 DEBUG 05:41:28 DEBUG 05:41:28 DEBUG [...] 05:41:28 DEBUG 05:41:28 DEBUG 05:41:28 DEBUG 05:41:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:41:28 DEBUG }, 05:41:28 DEBUG "value": [ 05:41:28 DEBUG 1712724088, 05:41:28 DEBUG "1253.7" 05:41:28 DEBUG ] 05:41:28 DEBUG } 05:41:28 DEBUG ] 05:41:28 DEBUG } 05:41:28 DEBUG } 05:41:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724088" 05:41:28 INFO [http_cmd]: http status code OK 05:41:28 DEBUG --- status code --- 05:41:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:28 DEBUG --- http response --- 05:41:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:41:28 DEBUG ----- output ----- 05:41:28 DEBUG { 05:41:28 DEBUG "status": "success", 05:41:28 DEBUG "data": { 05:41:28 DEBUG "resultType": "vector", 05:41:28 DEBUG "result": [ 05:41:28 DEBUG { 05:41:28 DEBUG "metric": { 05:41:28 DEBUG "pod": "ds-idrepo-0" 05:41:28 DEBUG }, 05:41:28 DEBUG "value": [ 05:41:28 DEBUG 05:41:28 DEBUG 05:41:28 DEBUG 05:41:28 DEBUG [...] 05:41:28 DEBUG 05:41:28 DEBUG 05:41:28 DEBUG 05:41:28 DEBUG "pod": "ds-idrepo-1" 05:41:28 DEBUG }, 05:41:28 DEBUG "value": [ 05:41:28 DEBUG 1712724088, 05:41:28 DEBUG "0" 05:41:28 DEBUG ] 05:41:28 DEBUG } 05:41:28 DEBUG ] 05:41:28 DEBUG } 05:41:28 DEBUG } 05:41:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724088" 05:41:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712724088" 05:41:28 INFO [http_cmd]: http status code OK 05:41:28 DEBUG --- status code --- 05:41:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:28 DEBUG --- http response --- 05:41:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:41:28 DEBUG ----- output ----- 05:41:28 DEBUG { 05:41:28 DEBUG "status": "success", 05:41:28 DEBUG "data": { 05:41:28 DEBUG "resultType": "vector", 05:41:28 DEBUG "result": [ 05:41:28 DEBUG { 05:41:28 DEBUG "metric": { 05:41:28 INFO [http_cmd]: http status code OK 05:41:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:41:28 DEBUG --- status code --- 05:41:28 DEBUG }, 05:41:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:28 DEBUG "value": [ 05:41:28 DEBUG --- http response --- 05:41:28 DEBUG 05:41:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:41:28 DEBUG 05:41:28 DEBUG ----- output ----- 05:41:28 DEBUG 05:41:28 DEBUG { 05:41:28 DEBUG [...] 05:41:28 DEBUG "status": "success", 05:41:28 DEBUG 05:41:28 DEBUG "data": { 05:41:28 DEBUG 05:41:28 DEBUG "resultType": "vector", 05:41:28 DEBUG 05:41:28 DEBUG "result": [ 05:41:28 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:41:28 DEBUG { 05:41:28 DEBUG }, 05:41:28 DEBUG "metric": { 05:41:28 DEBUG "value": [ 05:41:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:41:28 DEBUG 1712724088, 05:41:28 DEBUG }, 05:41:28 DEBUG "0" 05:41:28 DEBUG "value": [ 05:41:28 DEBUG ] 05:41:28 DEBUG 05:41:28 DEBUG } 05:41:28 DEBUG 05:41:28 DEBUG 05:41:28 DEBUG ] 05:41:28 DEBUG [...] 05:41:28 DEBUG } 05:41:28 DEBUG 05:41:28 DEBUG } 05:41:28 DEBUG 05:41:28 DEBUG 05:41:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:41:28 DEBUG }, 05:41:28 DEBUG "value": [ 05:41:28 DEBUG 1712724088, 05:41:28 DEBUG "2489335808" 05:41:28 DEBUG ] 05:41:28 DEBUG } 05:41:28 DEBUG ] 05:41:28 DEBUG } 05:41:28 DEBUG } 05:41:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724088" 05:41:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724088" 05:41:28 INFO [http_cmd]: http status code OK 05:41:28 DEBUG --- status code --- 05:41:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:28 DEBUG --- http response --- 05:41:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:41:28 DEBUG ----- output ----- 05:41:28 DEBUG { 05:41:28 INFO [http_cmd]: http status code OK 05:41:28 DEBUG "status": "success", 05:41:28 DEBUG --- status code --- 05:41:28 DEBUG "data": { 05:41:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:28 DEBUG "resultType": "vector", 05:41:28 DEBUG --- http response --- 05:41:28 DEBUG "result": [ 05:41:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:41:28 DEBUG { 05:41:28 DEBUG ----- output ----- 05:41:28 DEBUG "metric": { 05:41:28 DEBUG { 05:41:28 DEBUG "instance": "10.142.0.25:9100" 05:41:28 DEBUG "status": "success", 05:41:28 DEBUG "data": { 05:41:28 DEBUG }, 05:41:28 DEBUG "resultType": "vector", 05:41:28 DEBUG "value": [ 05:41:28 DEBUG "result": [ 05:41:28 DEBUG 05:41:28 DEBUG { 05:41:28 DEBUG 05:41:28 DEBUG "metric": { 05:41:28 DEBUG 05:41:28 DEBUG "instance": "10.142.0.25:9100" 05:41:28 DEBUG [...] 05:41:28 DEBUG }, 05:41:28 DEBUG 05:41:28 DEBUG "value": [ 05:41:28 DEBUG 05:41:28 DEBUG 05:41:28 DEBUG 05:41:28 DEBUG 05:41:28 DEBUG "instance": "10.142.0.77:9100" 05:41:28 DEBUG 05:41:28 DEBUG }, 05:41:28 DEBUG "value": [ 05:41:28 DEBUG 1712724088, 05:41:28 DEBUG "37593261.166666664" 05:41:28 DEBUG [...] 05:41:28 DEBUG ] 05:41:28 DEBUG 05:41:28 DEBUG } 05:41:28 DEBUG 05:41:28 DEBUG ] 05:41:28 DEBUG 05:41:28 DEBUG } 05:41:28 DEBUG "instance": "10.142.0.77:9100" 05:41:28 DEBUG } 05:41:28 DEBUG }, 05:41:28 DEBUG "value": [ 05:41:28 DEBUG 1712724088, 05:41:28 DEBUG "0.049000000000000606" 05:41:28 DEBUG ] 05:41:28 DEBUG } 05:41:28 DEBUG ] 05:41:28 DEBUG } 05:41:28 DEBUG } 05:41:43 INFO 05:41:43 INFO [loop_until]: kubectl --namespace=xlou top pods 05:41:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:41:44 INFO [loop_until]: OK (rc = 0) 05:41:44 DEBUG --- stdout --- 05:41:44 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 8999m 8804Mi am-654c4fcbcf-9scrj 8663m 8424Mi am-654c4fcbcf-v5qb5 7309m 8402Mi ds-cts-0 7352m 13270Mi ds-cts-1 7694m 13183Mi ds-cts-2 6909m 13335Mi ds-idrepo-0 9793m 5622Mi ds-idrepo-1 319m 5671Mi ds-idrepo-2 390m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 5m 1190Mi idm-6bfb7c96cc-vsb58 4m 2212Mi lodemon-b776b7d88-thrqf 8m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3606m 1673Mi 05:41:44 DEBUG --- stderr --- 05:41:44 DEBUG 05:41:50 INFO 05:41:50 INFO [loop_until]: kubectl --namespace=xlou top node 05:41:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:41:50 INFO [loop_until]: OK (rc = 0) 05:41:50 DEBUG --- stdout --- 05:41:50 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 97m 0% 2303Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8204m 51% 9277Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 80m 0% 3258Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10535m 66% 6506Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9612m 60% 9309Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9646m 60% 9646Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 78m 0% 856Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9786m 61% 6285Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 7459m 46% 13890Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 6647m 41% 13945Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 447m 2% 6372Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7581m 47% 13893Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 413m 2% 6334Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3720m 23% 5343Mi 9% 05:41:50 DEBUG --- stderr --- 05:41:50 DEBUG 05:42:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712724144" 05:42:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724144" 05:42:24 INFO [http_cmd]: http status code OK 05:42:24 ERROR [http_cmd]: ERROR 05:42:24 DEBUG --- status code --- 05:42:24 ERROR --- status code --- 05:42:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:24 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:42:24 DEBUG --- http response --- 05:42:24 ERROR --- http response --- 05:42:24 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:42:24 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:42:24 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:42:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724144" 05:42:24 INFO [http_cmd]: http status code OK 05:42:24 DEBUG --- status code --- 05:42:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:24 DEBUG --- http response --- 05:42:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:42:24 DEBUG ----- output ----- 05:42:24 DEBUG { 05:42:24 DEBUG "status": "success", 05:42:24 DEBUG "data": { 05:42:24 DEBUG "resultType": "vector", 05:42:24 DEBUG "result": [ 05:42:24 DEBUG { 05:42:24 DEBUG "metric": { 05:42:24 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:42:24 DEBUG }, 05:42:24 DEBUG "value": [ 05:42:24 DEBUG 05:42:24 DEBUG 05:42:24 DEBUG 05:42:24 DEBUG [...] 05:42:24 DEBUG 05:42:24 DEBUG 05:42:24 DEBUG 05:42:24 DEBUG "pod": "am-654c4fcbcf-487cv" 05:42:24 DEBUG }, 05:42:24 DEBUG "value": [ 05:42:24 DEBUG 1712724144, 05:42:24 DEBUG "0" 05:42:24 DEBUG ] 05:42:24 DEBUG } 05:42:24 DEBUG ] 05:42:24 DEBUG } 05:42:24 DEBUG } 05:42:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724145" 05:42:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724145" 05:42:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724145" 05:42:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724145" 05:42:25 INFO [http_cmd]: http status code OK 05:42:25 INFO [http_cmd]: http status code OK 05:42:25 DEBUG --- status code --- 05:42:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:25 DEBUG --- status code --- 05:42:25 DEBUG --- http response --- 05:42:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:25 DEBUG --- http response --- 05:42:25 INFO [http_cmd]: http status code OK 05:42:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:42:25 INFO [http_cmd]: http status code OK 05:42:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:42:25 DEBUG --- status code --- 05:42:25 DEBUG ----- output ----- 05:42:25 DEBUG --- status code --- 05:42:25 DEBUG ----- output ----- 05:42:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:25 DEBUG { 05:42:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:25 DEBUG { 05:42:25 DEBUG --- http response --- 05:42:25 DEBUG "status": "success", 05:42:25 DEBUG --- http response --- 05:42:25 DEBUG "status": "success", 05:42:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:42:25 DEBUG "data": { 05:42:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:42:25 DEBUG "data": { 05:42:25 DEBUG ----- output ----- 05:42:25 DEBUG "resultType": "vector", 05:42:25 DEBUG ----- output ----- 05:42:25 DEBUG "resultType": "vector", 05:42:25 DEBUG { 05:42:25 DEBUG "result": [ 05:42:25 DEBUG { 05:42:25 DEBUG "result": [ 05:42:25 DEBUG "status": "success", 05:42:25 DEBUG { 05:42:25 DEBUG "status": "success", 05:42:25 DEBUG { 05:42:25 DEBUG "data": { 05:42:25 DEBUG "metric": { 05:42:25 DEBUG "data": { 05:42:25 DEBUG "metric": { 05:42:25 DEBUG "resultType": "vector", 05:42:25 DEBUG "instance": "10.142.0.25:9100" 05:42:25 DEBUG "resultType": "vector", 05:42:25 DEBUG "pod": "ds-idrepo-0" 05:42:25 DEBUG "result": [ 05:42:25 DEBUG }, 05:42:25 DEBUG "result": [ 05:42:25 DEBUG }, 05:42:25 DEBUG { 05:42:25 DEBUG "value": [ 05:42:25 DEBUG { 05:42:25 DEBUG "value": [ 05:42:25 DEBUG "metric": { 05:42:25 DEBUG 05:42:25 DEBUG "metric": { 05:42:25 DEBUG 05:42:25 DEBUG "pod": "ds-idrepo-0" 05:42:25 DEBUG 05:42:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:42:25 DEBUG 05:42:25 DEBUG }, 05:42:25 DEBUG 05:42:25 DEBUG }, 05:42:25 DEBUG 05:42:25 DEBUG "value": [ 05:42:25 DEBUG [...] 05:42:25 DEBUG "value": [ 05:42:25 DEBUG [...] 05:42:25 DEBUG 05:42:25 DEBUG 05:42:25 DEBUG 05:42:25 DEBUG 05:42:25 DEBUG 05:42:25 DEBUG 05:42:25 DEBUG 05:42:25 DEBUG 05:42:25 DEBUG 05:42:25 DEBUG 05:42:25 DEBUG 05:42:25 DEBUG 05:42:25 DEBUG [...] 05:42:25 DEBUG "instance": "10.142.0.77:9100" 05:42:25 DEBUG [...] 05:42:25 DEBUG "pod": "ds-cts-2" 05:42:25 DEBUG 05:42:25 DEBUG }, 05:42:25 DEBUG 05:42:25 DEBUG }, 05:42:25 DEBUG 05:42:25 DEBUG "value": [ 05:42:25 DEBUG 05:42:25 DEBUG "value": [ 05:42:25 DEBUG 05:42:25 DEBUG 1712724145, 05:42:25 DEBUG 05:42:25 DEBUG 1712724145, 05:42:25 DEBUG "pod": "ds-cts-2" 05:42:25 DEBUG "310672206848" 05:42:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:42:25 DEBUG "0" 05:42:25 DEBUG }, 05:42:25 DEBUG ] 05:42:25 DEBUG }, 05:42:25 DEBUG ] 05:42:25 DEBUG "value": [ 05:42:25 DEBUG } 05:42:25 DEBUG "value": [ 05:42:25 DEBUG } 05:42:25 DEBUG 1712724145, 05:42:25 DEBUG ] 05:42:25 DEBUG 1712724145, 05:42:25 DEBUG ] 05:42:25 DEBUG "29.933333333333334" 05:42:25 DEBUG } 05:42:25 DEBUG "1253.7666666666667" 05:42:25 DEBUG } 05:42:25 DEBUG ] 05:42:25 DEBUG } 05:42:25 DEBUG ] 05:42:25 DEBUG } 05:42:25 DEBUG } 05:42:25 DEBUG } 05:42:25 DEBUG ] 05:42:25 DEBUG ] 05:42:25 DEBUG } 05:42:25 DEBUG } 05:42:25 DEBUG } 05:42:25 DEBUG } 05:42:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724145" 05:42:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724145" 05:42:25 INFO [http_cmd]: http status code OK 05:42:25 DEBUG --- status code --- 05:42:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:25 DEBUG --- http response --- 05:42:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:42:25 INFO [http_cmd]: http status code OK 05:42:25 DEBUG ----- output ----- 05:42:25 DEBUG --- status code --- 05:42:25 DEBUG { 05:42:25 DEBUG "status": "success", 05:42:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:25 DEBUG "data": { 05:42:25 DEBUG --- http response --- 05:42:25 DEBUG "resultType": "vector", 05:42:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:42:25 DEBUG "result": [ 05:42:25 DEBUG ----- output ----- 05:42:25 DEBUG { 05:42:25 DEBUG { 05:42:25 DEBUG "metric": { 05:42:25 DEBUG "status": "success", 05:42:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:42:25 DEBUG "data": { 05:42:25 DEBUG }, 05:42:25 DEBUG "resultType": "vector", 05:42:25 DEBUG "value": [ 05:42:25 DEBUG "result": [ 05:42:25 DEBUG 05:42:25 DEBUG { 05:42:25 DEBUG 05:42:25 DEBUG "metric": { 05:42:25 DEBUG 05:42:25 DEBUG "instance": "10.142.0.25:9100" 05:42:25 DEBUG [...] 05:42:25 DEBUG }, 05:42:25 DEBUG 05:42:25 DEBUG "value": [ 05:42:25 DEBUG 05:42:25 DEBUG 05:42:25 DEBUG 05:42:25 DEBUG 05:42:25 DEBUG "pod": "ds-cts-1" 05:42:25 DEBUG 05:42:25 DEBUG }, 05:42:25 DEBUG [...] 05:42:25 DEBUG "value": [ 05:42:25 DEBUG 05:42:25 DEBUG 1712724145, 05:42:25 DEBUG 05:42:25 DEBUG "26530921.758062318" 05:42:25 DEBUG 05:42:25 DEBUG ] 05:42:25 DEBUG "instance": "10.142.0.77:9100" 05:42:25 DEBUG } 05:42:25 DEBUG }, 05:42:25 DEBUG ] 05:42:25 DEBUG "value": [ 05:42:25 DEBUG } 05:42:25 DEBUG 1712724145, 05:42:25 DEBUG } 05:42:25 DEBUG "37496710.333333336" 05:42:25 DEBUG ] 05:42:25 DEBUG } 05:42:25 DEBUG ] 05:42:25 DEBUG } 05:42:25 DEBUG } 05:42:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724145" 05:42:25 INFO [http_cmd]: http status code OK 05:42:25 DEBUG --- status code --- 05:42:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:25 DEBUG --- http response --- 05:42:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:42:25 DEBUG ----- output ----- 05:42:25 DEBUG { 05:42:25 DEBUG "status": "success", 05:42:25 DEBUG "data": { 05:42:25 DEBUG "resultType": "vector", 05:42:25 DEBUG "result": [ 05:42:25 DEBUG { 05:42:25 DEBUG "metric": { 05:42:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:42:25 DEBUG }, 05:42:25 DEBUG "value": [ 05:42:25 DEBUG 05:42:25 DEBUG 05:42:25 DEBUG 05:42:25 DEBUG [...] 05:42:25 DEBUG 05:42:25 DEBUG 05:42:25 DEBUG 05:42:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:42:25 DEBUG }, 05:42:25 DEBUG "value": [ 05:42:25 DEBUG 1712724145, 05:42:25 DEBUG "226916.5" 05:42:25 DEBUG ] 05:42:25 DEBUG } 05:42:25 DEBUG ] 05:42:25 DEBUG } 05:42:25 DEBUG } 05:42:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724145" 05:42:25 INFO [http_cmd]: http status code OK 05:42:25 DEBUG --- status code --- 05:42:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:25 DEBUG --- http response --- 05:42:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:42:25 DEBUG ----- output ----- 05:42:25 DEBUG { 05:42:25 DEBUG "status": "success", 05:42:25 DEBUG "data": { 05:42:25 DEBUG "resultType": "vector", 05:42:25 DEBUG "result": [ 05:42:25 DEBUG { 05:42:25 DEBUG "metric": { 05:42:25 DEBUG "pod": "ds-idrepo-0" 05:42:25 DEBUG }, 05:42:25 DEBUG "value": [ 05:42:25 DEBUG 05:42:25 DEBUG 05:42:25 DEBUG 05:42:25 DEBUG [...] 05:42:25 DEBUG 05:42:25 DEBUG 05:42:25 DEBUG 05:42:25 DEBUG "pod": "ds-idrepo-1" 05:42:25 DEBUG }, 05:42:25 DEBUG "value": [ 05:42:25 DEBUG 1712724145, 05:42:25 DEBUG "0" 05:42:25 DEBUG ] 05:42:25 DEBUG } 05:42:25 DEBUG ] 05:42:25 DEBUG } 05:42:25 DEBUG } 05:42:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724145" 05:42:26 INFO [http_cmd]: http status code OK 05:42:26 DEBUG --- status code --- 05:42:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:26 DEBUG --- http response --- 05:42:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:42:26 DEBUG ----- output ----- 05:42:26 DEBUG { 05:42:26 DEBUG "status": "success", 05:42:26 DEBUG "data": { 05:42:26 DEBUG "resultType": "vector", 05:42:26 DEBUG "result": [ 05:42:26 DEBUG { 05:42:26 DEBUG "metric": { 05:42:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:42:26 DEBUG }, 05:42:26 DEBUG "value": [ 05:42:26 DEBUG 05:42:26 DEBUG 05:42:26 DEBUG 05:42:26 DEBUG [...] 05:42:26 DEBUG 05:42:26 DEBUG 05:42:26 DEBUG 05:42:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:42:26 DEBUG }, 05:42:26 DEBUG "value": [ 05:42:26 DEBUG 1712724145, 05:42:26 DEBUG "1253.8333333333333" 05:42:26 DEBUG ] 05:42:26 DEBUG } 05:42:26 DEBUG ] 05:42:26 DEBUG } 05:42:26 DEBUG } 05:42:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724146" 05:42:26 INFO [http_cmd]: http status code OK 05:42:26 DEBUG --- status code --- 05:42:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:26 DEBUG --- http response --- 05:42:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:42:26 DEBUG ----- output ----- 05:42:26 DEBUG { 05:42:26 DEBUG "status": "success", 05:42:26 DEBUG "data": { 05:42:26 DEBUG "resultType": "vector", 05:42:26 DEBUG "result": [ 05:42:26 DEBUG { 05:42:26 DEBUG "metric": { 05:42:26 DEBUG "instance": "10.142.0.25:9100" 05:42:26 DEBUG }, 05:42:26 DEBUG "value": [ 05:42:26 DEBUG 05:42:26 DEBUG 05:42:26 DEBUG 05:42:26 DEBUG [...] 05:42:26 DEBUG 05:42:26 DEBUG 05:42:26 DEBUG 05:42:26 DEBUG "instance": "10.142.0.77:9100" 05:42:26 DEBUG }, 05:42:26 DEBUG "value": [ 05:42:26 DEBUG 1712724146, 05:42:26 DEBUG "0.14289999999999534" 05:42:26 DEBUG ] 05:42:26 DEBUG } 05:42:26 DEBUG ] 05:42:26 DEBUG } 05:42:26 DEBUG } 05:42:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724147" 05:42:27 INFO [http_cmd]: http status code OK 05:42:27 DEBUG --- status code --- 05:42:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:27 DEBUG --- http response --- 05:42:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:42:27 DEBUG ----- output ----- 05:42:27 DEBUG { 05:42:27 DEBUG "status": "success", 05:42:27 DEBUG "data": { 05:42:27 DEBUG "resultType": "vector", 05:42:27 DEBUG "result": [ 05:42:27 DEBUG { 05:42:27 DEBUG "metric": { 05:42:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:42:27 DEBUG }, 05:42:27 DEBUG "value": [ 05:42:27 DEBUG 05:42:27 DEBUG 05:42:27 DEBUG 05:42:27 DEBUG [...] 05:42:27 DEBUG 05:42:27 DEBUG 05:42:27 DEBUG 05:42:27 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:42:27 DEBUG }, 05:42:27 DEBUG "value": [ 05:42:27 DEBUG 1712724147, 05:42:27 DEBUG "13113.738813681784" 05:42:27 DEBUG ] 05:42:27 DEBUG } 05:42:27 DEBUG ] 05:42:27 DEBUG } 05:42:27 DEBUG } 05:42:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712724147" 05:42:27 INFO [http_cmd]: http status code OK 05:42:27 DEBUG --- status code --- 05:42:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:27 DEBUG --- http response --- 05:42:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:42:27 DEBUG ----- output ----- 05:42:27 DEBUG { 05:42:27 DEBUG "status": "success", 05:42:27 DEBUG "data": { 05:42:27 DEBUG "resultType": "vector", 05:42:27 DEBUG "result": [ 05:42:27 DEBUG { 05:42:27 DEBUG "metric": { 05:42:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:42:27 DEBUG }, 05:42:27 DEBUG "value": [ 05:42:27 DEBUG 05:42:27 DEBUG 05:42:27 DEBUG 05:42:27 DEBUG [...] 05:42:27 DEBUG 05:42:27 DEBUG 05:42:27 DEBUG 05:42:27 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:42:27 DEBUG }, 05:42:27 DEBUG "value": [ 05:42:27 DEBUG 1712724147, 05:42:27 DEBUG "3.6387566047554443" 05:42:27 DEBUG ] 05:42:27 DEBUG } 05:42:27 DEBUG ] 05:42:27 DEBUG } 05:42:27 DEBUG } 05:42:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712724147" 05:42:27 INFO [http_cmd]: http status code OK 05:42:27 DEBUG --- status code --- 05:42:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:27 DEBUG --- http response --- 05:42:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:42:27 DEBUG ----- output ----- 05:42:27 DEBUG { 05:42:27 DEBUG "status": "success", 05:42:27 DEBUG "data": { 05:42:27 DEBUG "resultType": "vector", 05:42:27 DEBUG "result": [ 05:42:27 DEBUG { 05:42:27 DEBUG "metric": { 05:42:27 DEBUG "pod": "ds-idrepo-0" 05:42:27 DEBUG }, 05:42:27 DEBUG "value": [ 05:42:27 DEBUG 05:42:27 DEBUG 05:42:27 DEBUG 05:42:27 DEBUG [...] 05:42:27 DEBUG 05:42:27 DEBUG 05:42:27 DEBUG 05:42:27 DEBUG "pod": "ds-cts-2" 05:42:27 DEBUG }, 05:42:27 DEBUG "value": [ 05:42:27 DEBUG 1712724147, 05:42:27 DEBUG "0" 05:42:27 DEBUG ] 05:42:27 DEBUG } 05:42:27 DEBUG ] 05:42:27 DEBUG } 05:42:27 DEBUG } 05:42:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724147" 05:42:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724147" 05:42:27 INFO [http_cmd]: http status code OK 05:42:27 DEBUG --- status code --- 05:42:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:27 DEBUG --- http response --- 05:42:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:42:27 DEBUG ----- output ----- 05:42:27 DEBUG { 05:42:27 DEBUG "status": "success", 05:42:27 DEBUG "data": { 05:42:27 DEBUG "resultType": "vector", 05:42:27 DEBUG "result": [ 05:42:27 DEBUG { 05:42:27 DEBUG "metric": { 05:42:27 DEBUG "pod": "ds-idrepo-2" 05:42:27 DEBUG }, 05:42:27 DEBUG "value": [ 05:42:27 DEBUG 05:42:27 DEBUG 05:42:27 DEBUG 05:42:27 DEBUG [...] 05:42:27 DEBUG 05:42:27 DEBUG 05:42:27 DEBUG 05:42:27 DEBUG "pod": "ds-cts-2" 05:42:27 DEBUG }, 05:42:27 DEBUG "value": [ 05:42:27 DEBUG 1712724147, 05:42:27 DEBUG "0.000021218888888888892" 05:42:27 DEBUG ] 05:42:27 DEBUG } 05:42:27 DEBUG ] 05:42:27 DEBUG } 05:42:27 DEBUG } 05:42:27 INFO [http_cmd]: http status code OK 05:42:27 DEBUG --- status code --- 05:42:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:27 DEBUG --- http response --- 05:42:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:42:27 DEBUG ----- output ----- 05:42:27 DEBUG { 05:42:27 DEBUG "status": "success", 05:42:27 DEBUG "data": { 05:42:27 DEBUG "resultType": "vector", 05:42:27 DEBUG "result": [ 05:42:27 DEBUG { 05:42:27 DEBUG "metric": { 05:42:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:42:27 DEBUG }, 05:42:27 DEBUG "value": [ 05:42:27 DEBUG 05:42:27 DEBUG 05:42:27 DEBUG 05:42:27 DEBUG [...] 05:42:27 DEBUG 05:42:27 DEBUG 05:42:27 DEBUG 05:42:27 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:42:27 DEBUG }, 05:42:27 DEBUG "value": [ 05:42:27 DEBUG 1712724147, 05:42:27 DEBUG "2.273673568870588" 05:42:27 DEBUG ] 05:42:27 DEBUG } 05:42:27 DEBUG ] 05:42:27 DEBUG } 05:42:27 DEBUG } 05:42:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724147" 05:42:27 INFO [http_cmd]: http status code OK 05:42:27 DEBUG --- status code --- 05:42:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:27 DEBUG --- http response --- 05:42:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:42:27 DEBUG ----- output ----- 05:42:27 DEBUG { 05:42:27 DEBUG "status": "success", 05:42:27 DEBUG "data": { 05:42:27 DEBUG "resultType": "vector", 05:42:27 DEBUG "result": [ 05:42:27 DEBUG { 05:42:27 DEBUG "metric": { 05:42:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:42:27 DEBUG }, 05:42:27 DEBUG "value": [ 05:42:27 DEBUG 05:42:27 DEBUG 05:42:27 DEBUG 05:42:27 DEBUG [...] 05:42:27 DEBUG 05:42:27 DEBUG 05:42:27 DEBUG 05:42:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:42:27 DEBUG }, 05:42:27 DEBUG "value": [ 05:42:27 DEBUG 1712724147, 05:42:27 DEBUG "1253.8333333333333" 05:42:27 DEBUG ] 05:42:27 DEBUG } 05:42:27 DEBUG ] 05:42:27 DEBUG } 05:42:27 DEBUG } 05:42:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712724147" 05:42:27 INFO [http_cmd]: http status code OK 05:42:27 DEBUG --- status code --- 05:42:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:27 DEBUG --- http response --- 05:42:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:42:27 DEBUG ----- output ----- 05:42:27 DEBUG { 05:42:27 DEBUG "status": "success", 05:42:27 DEBUG "data": { 05:42:27 DEBUG "resultType": "vector", 05:42:27 DEBUG "result": [ 05:42:27 DEBUG { 05:42:27 DEBUG "metric": { 05:42:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:42:27 DEBUG }, 05:42:27 DEBUG "value": [ 05:42:27 DEBUG 05:42:27 DEBUG 05:42:27 DEBUG 05:42:27 DEBUG [...] 05:42:27 DEBUG 05:42:27 DEBUG 05:42:27 DEBUG 05:42:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:42:27 DEBUG }, 05:42:27 DEBUG "value": [ 05:42:27 DEBUG 1712724147, 05:42:27 DEBUG "0.043" 05:42:27 DEBUG ] 05:42:27 DEBUG } 05:42:27 DEBUG ] 05:42:27 DEBUG } 05:42:27 DEBUG } 05:42:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724148" 05:42:28 INFO [http_cmd]: http status code OK 05:42:28 DEBUG --- status code --- 05:42:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:28 DEBUG --- http response --- 05:42:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:42:28 DEBUG ----- output ----- 05:42:28 DEBUG { 05:42:28 DEBUG "status": "success", 05:42:28 DEBUG "data": { 05:42:28 DEBUG "resultType": "vector", 05:42:28 DEBUG "result": [ 05:42:28 DEBUG { 05:42:28 DEBUG "metric": { 05:42:28 DEBUG "pod": "ds-idrepo-0" 05:42:28 DEBUG }, 05:42:28 DEBUG "value": [ 05:42:28 DEBUG 05:42:28 DEBUG 05:42:28 DEBUG 05:42:28 DEBUG [...] 05:42:28 DEBUG 05:42:28 DEBUG 05:42:28 DEBUG 05:42:28 DEBUG "pod": "ds-idrepo-1" 05:42:28 DEBUG }, 05:42:28 DEBUG "value": [ 05:42:28 DEBUG 1712724148, 05:42:28 DEBUG "0" 05:42:28 DEBUG ] 05:42:28 DEBUG } 05:42:28 DEBUG ] 05:42:28 DEBUG } 05:42:28 DEBUG } 05:42:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724148" 05:42:28 INFO [http_cmd]: http status code OK 05:42:28 DEBUG --- status code --- 05:42:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:28 DEBUG --- http response --- 05:42:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:42:28 DEBUG ----- output ----- 05:42:28 DEBUG { 05:42:28 DEBUG "status": "success", 05:42:28 DEBUG "data": { 05:42:28 DEBUG "resultType": "vector", 05:42:28 DEBUG "result": [ 05:42:28 DEBUG { 05:42:28 DEBUG "metric": { 05:42:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:42:28 DEBUG }, 05:42:28 DEBUG "value": [ 05:42:28 DEBUG 05:42:28 DEBUG 05:42:28 DEBUG 05:42:28 DEBUG [...] 05:42:28 DEBUG 05:42:28 DEBUG 05:42:28 DEBUG 05:42:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:42:28 DEBUG }, 05:42:28 DEBUG "value": [ 05:42:28 DEBUG 1712724148, 05:42:28 DEBUG "1253.1333333333332" 05:42:28 DEBUG ] 05:42:28 DEBUG } 05:42:28 DEBUG ] 05:42:28 DEBUG } 05:42:28 DEBUG } 05:42:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724148" 05:42:28 INFO [http_cmd]: http status code OK 05:42:28 DEBUG --- status code --- 05:42:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:28 DEBUG --- http response --- 05:42:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:42:28 DEBUG ----- output ----- 05:42:28 DEBUG { 05:42:28 DEBUG "status": "success", 05:42:28 DEBUG "data": { 05:42:28 DEBUG "resultType": "vector", 05:42:28 DEBUG "result": [ 05:42:28 DEBUG { 05:42:28 DEBUG "metric": { 05:42:28 DEBUG "pod": "ds-idrepo-0" 05:42:28 DEBUG }, 05:42:28 DEBUG "value": [ 05:42:28 DEBUG 05:42:28 DEBUG 05:42:28 DEBUG 05:42:28 DEBUG [...] 05:42:28 DEBUG 05:42:28 DEBUG 05:42:28 DEBUG 05:42:28 DEBUG "pod": "ds-idrepo-1" 05:42:28 DEBUG }, 05:42:28 DEBUG "value": [ 05:42:28 DEBUG 1712724148, 05:42:28 DEBUG "0" 05:42:28 DEBUG ] 05:42:28 DEBUG } 05:42:28 DEBUG ] 05:42:28 DEBUG } 05:42:28 DEBUG } 05:42:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724148" 05:42:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712724148" 05:42:28 INFO [http_cmd]: http status code OK 05:42:28 DEBUG --- status code --- 05:42:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:28 DEBUG --- http response --- 05:42:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:42:28 DEBUG ----- output ----- 05:42:28 DEBUG { 05:42:28 DEBUG "status": "success", 05:42:28 DEBUG "data": { 05:42:28 DEBUG "resultType": "vector", 05:42:28 DEBUG "result": [ 05:42:28 DEBUG { 05:42:28 DEBUG "metric": { 05:42:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:42:28 DEBUG }, 05:42:28 DEBUG "value": [ 05:42:28 DEBUG 05:42:28 DEBUG 05:42:28 DEBUG 05:42:28 DEBUG [...] 05:42:28 DEBUG 05:42:28 DEBUG 05:42:28 DEBUG 05:42:28 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:42:28 DEBUG }, 05:42:28 DEBUG "value": [ 05:42:28 DEBUG 1712724148, 05:42:28 DEBUG "0" 05:42:28 DEBUG ] 05:42:28 DEBUG } 05:42:28 DEBUG ] 05:42:28 DEBUG } 05:42:28 DEBUG } 05:42:28 INFO [http_cmd]: http status code OK 05:42:28 DEBUG --- status code --- 05:42:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:28 DEBUG --- http response --- 05:42:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:42:28 DEBUG ----- output ----- 05:42:28 DEBUG { 05:42:28 DEBUG "status": "success", 05:42:28 DEBUG "data": { 05:42:28 DEBUG "resultType": "vector", 05:42:28 DEBUG "result": [ 05:42:28 DEBUG { 05:42:28 DEBUG "metric": { 05:42:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:42:28 DEBUG }, 05:42:28 DEBUG "value": [ 05:42:28 DEBUG 05:42:28 DEBUG 05:42:28 DEBUG 05:42:28 DEBUG [...] 05:42:28 DEBUG 05:42:28 DEBUG 05:42:28 DEBUG 05:42:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:42:28 DEBUG }, 05:42:28 DEBUG "value": [ 05:42:28 DEBUG 1712724148, 05:42:28 DEBUG "2489032704" 05:42:28 DEBUG ] 05:42:28 DEBUG } 05:42:28 DEBUG ] 05:42:28 DEBUG } 05:42:28 DEBUG } 05:42:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724148" 05:42:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724148" 05:42:28 INFO [http_cmd]: http status code OK 05:42:28 DEBUG --- status code --- 05:42:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:28 DEBUG --- http response --- 05:42:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:42:28 DEBUG ----- output ----- 05:42:28 DEBUG { 05:42:28 DEBUG "status": "success", 05:42:28 DEBUG "data": { 05:42:28 DEBUG "resultType": "vector", 05:42:28 DEBUG "result": [ 05:42:28 DEBUG { 05:42:28 DEBUG "metric": { 05:42:28 DEBUG "instance": "10.142.0.25:9100" 05:42:28 DEBUG }, 05:42:28 DEBUG "value": [ 05:42:28 DEBUG 05:42:28 DEBUG 05:42:28 DEBUG 05:42:28 INFO [http_cmd]: http status code OK 05:42:28 DEBUG [...] 05:42:28 DEBUG --- status code --- 05:42:28 DEBUG 05:42:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:28 DEBUG 05:42:28 DEBUG --- http response --- 05:42:28 DEBUG 05:42:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:42:28 DEBUG "instance": "10.142.0.77:9100" 05:42:28 DEBUG ----- output ----- 05:42:28 DEBUG }, 05:42:28 DEBUG { 05:42:28 DEBUG "value": [ 05:42:28 DEBUG "status": "success", 05:42:28 DEBUG 1712724148, 05:42:28 DEBUG "data": { 05:42:28 DEBUG "37498314.733333334" 05:42:28 DEBUG "resultType": "vector", 05:42:28 DEBUG ] 05:42:28 DEBUG "result": [ 05:42:28 DEBUG } 05:42:28 DEBUG { 05:42:28 DEBUG ] 05:42:28 DEBUG "metric": { 05:42:28 DEBUG } 05:42:28 DEBUG "instance": "10.142.0.25:9100" 05:42:28 DEBUG } 05:42:28 DEBUG }, 05:42:28 DEBUG "value": [ 05:42:28 DEBUG 05:42:28 DEBUG 05:42:28 DEBUG 05:42:28 DEBUG [...] 05:42:28 DEBUG 05:42:28 DEBUG 05:42:28 DEBUG 05:42:28 DEBUG "instance": "10.142.0.77:9100" 05:42:28 DEBUG }, 05:42:28 DEBUG "value": [ 05:42:28 DEBUG 1712724148, 05:42:28 DEBUG "0.04899999999999907" 05:42:28 DEBUG ] 05:42:28 DEBUG } 05:42:28 DEBUG ] 05:42:28 DEBUG } 05:42:28 DEBUG } 05:42:44 INFO 05:42:44 INFO [loop_until]: kubectl --namespace=xlou top pods 05:42:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:42:44 INFO [loop_until]: OK (rc = 0) 05:42:44 DEBUG --- stdout --- 05:42:44 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 9265m 8808Mi am-654c4fcbcf-9scrj 8572m 8419Mi am-654c4fcbcf-v5qb5 7417m 8398Mi ds-cts-0 7048m 13319Mi ds-cts-1 7620m 13320Mi ds-cts-2 6917m 12905Mi ds-idrepo-0 9701m 5622Mi ds-idrepo-1 334m 5671Mi ds-idrepo-2 327m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 7m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3573m 1674Mi 05:42:44 DEBUG --- stderr --- 05:42:44 DEBUG 05:42:50 INFO 05:42:50 INFO [loop_until]: kubectl --namespace=xlou top node 05:42:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:42:51 INFO [loop_until]: OK (rc = 0) 05:42:51 DEBUG --- stdout --- 05:42:51 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 95m 0% 2303Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8120m 51% 9279Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 79m 0% 3262Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 10748m 67% 6513Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9502m 59% 9312Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9825m 61% 9645Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 77m 0% 855Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9691m 60% 6289Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 7241m 45% 13033Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 6876m 43% 13498Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 404m 2% 6373Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 7410m 46% 13264Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 381m 2% 6335Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3697m 23% 5347Mi 9% 05:42:51 DEBUG --- stderr --- 05:42:51 DEBUG 05:43:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724204" 05:43:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712724204" 05:43:24 INFO [http_cmd]: http status code OK 05:43:24 ERROR [http_cmd]: ERROR 05:43:24 DEBUG --- status code --- 05:43:24 ERROR --- status code --- 05:43:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:24 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:43:24 DEBUG --- http response --- 05:43:24 ERROR --- http response --- 05:43:24 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:43:24 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:43:24 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:43:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724205" 05:43:25 INFO [http_cmd]: http status code OK 05:43:25 DEBUG --- status code --- 05:43:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:25 DEBUG --- http response --- 05:43:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:43:25 DEBUG ----- output ----- 05:43:25 DEBUG { 05:43:25 DEBUG "status": "success", 05:43:25 DEBUG "data": { 05:43:25 DEBUG "resultType": "vector", 05:43:25 DEBUG "result": [ 05:43:25 DEBUG { 05:43:25 DEBUG "metric": { 05:43:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:43:25 DEBUG }, 05:43:25 DEBUG "value": [ 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG [...] 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:43:25 DEBUG }, 05:43:25 DEBUG "value": [ 05:43:25 DEBUG 1712724205, 05:43:25 DEBUG "0" 05:43:25 DEBUG ] 05:43:25 DEBUG } 05:43:25 DEBUG ] 05:43:25 DEBUG } 05:43:25 DEBUG } 05:43:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724205" 05:43:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724205" 05:43:25 INFO [http_cmd]: http status code OK 05:43:25 DEBUG --- status code --- 05:43:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:25 DEBUG --- http response --- 05:43:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:43:25 DEBUG ----- output ----- 05:43:25 DEBUG { 05:43:25 DEBUG "status": "success", 05:43:25 DEBUG "data": { 05:43:25 DEBUG "resultType": "vector", 05:43:25 DEBUG "result": [ 05:43:25 DEBUG { 05:43:25 DEBUG "metric": { 05:43:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:43:25 DEBUG }, 05:43:25 DEBUG "value": [ 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724205" 05:43:25 DEBUG 05:43:25 DEBUG [...] 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:43:25 DEBUG }, 05:43:25 DEBUG "value": [ 05:43:25 DEBUG 1712724205, 05:43:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724205" 05:43:25 DEBUG "1249.4333333333334" 05:43:25 DEBUG ] 05:43:25 DEBUG } 05:43:25 DEBUG ] 05:43:25 DEBUG } 05:43:25 DEBUG } 05:43:25 INFO [http_cmd]: http status code OK 05:43:25 INFO [http_cmd]: http status code OK 05:43:25 DEBUG --- status code --- 05:43:25 DEBUG --- status code --- 05:43:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:25 DEBUG --- http response --- 05:43:25 DEBUG --- http response --- 05:43:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:43:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:43:25 DEBUG ----- output ----- 05:43:25 DEBUG ----- output ----- 05:43:25 DEBUG { 05:43:25 DEBUG { 05:43:25 DEBUG "status": "success", 05:43:25 DEBUG "status": "success", 05:43:25 DEBUG "data": { 05:43:25 DEBUG "data": { 05:43:25 DEBUG "resultType": "vector", 05:43:25 DEBUG "resultType": "vector", 05:43:25 DEBUG "result": [ 05:43:25 DEBUG "result": [ 05:43:25 DEBUG { 05:43:25 DEBUG { 05:43:25 DEBUG "metric": { 05:43:25 DEBUG "metric": { 05:43:25 DEBUG "pod": "ds-idrepo-0" 05:43:25 DEBUG "pod": "ds-idrepo-0" 05:43:25 DEBUG }, 05:43:25 DEBUG }, 05:43:25 DEBUG "value": [ 05:43:25 DEBUG "value": [ 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG [...] 05:43:25 DEBUG [...] 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG "pod": "ds-cts-2" 05:43:25 DEBUG "pod": "ds-cts-2" 05:43:25 DEBUG }, 05:43:25 DEBUG }, 05:43:25 DEBUG "value": [ 05:43:25 INFO [http_cmd]: http status code OK 05:43:25 DEBUG "value": [ 05:43:25 DEBUG 1712724205, 05:43:25 DEBUG --- status code --- 05:43:25 DEBUG 1712724205, 05:43:25 DEBUG "0" 05:43:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:25 DEBUG "29.933333333333334" 05:43:25 DEBUG ] 05:43:25 DEBUG --- http response --- 05:43:25 DEBUG ] 05:43:25 DEBUG } 05:43:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:43:25 DEBUG } 05:43:25 DEBUG ] 05:43:25 DEBUG ----- output ----- 05:43:25 DEBUG ] 05:43:25 DEBUG } 05:43:25 DEBUG { 05:43:25 DEBUG } 05:43:25 DEBUG } 05:43:25 DEBUG "status": "success", 05:43:25 DEBUG } 05:43:25 DEBUG "data": { 05:43:25 DEBUG "resultType": "vector", 05:43:25 DEBUG "result": [ 05:43:25 DEBUG { 05:43:25 DEBUG "metric": { 05:43:25 DEBUG "instance": "10.142.0.25:9100" 05:43:25 DEBUG }, 05:43:25 DEBUG "value": [ 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG [...] 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG "instance": "10.142.0.77:9100" 05:43:25 DEBUG }, 05:43:25 DEBUG "value": [ 05:43:25 DEBUG 1712724205, 05:43:25 DEBUG "312630434816" 05:43:25 DEBUG ] 05:43:25 DEBUG } 05:43:25 DEBUG ] 05:43:25 DEBUG } 05:43:25 DEBUG } 05:43:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724205" 05:43:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724205" 05:43:25 INFO [http_cmd]: http status code OK 05:43:25 DEBUG --- status code --- 05:43:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:25 DEBUG --- http response --- 05:43:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:43:25 DEBUG ----- output ----- 05:43:25 DEBUG { 05:43:25 DEBUG "status": "success", 05:43:25 DEBUG "data": { 05:43:25 DEBUG "resultType": "vector", 05:43:25 DEBUG "result": [ 05:43:25 DEBUG { 05:43:25 DEBUG "metric": { 05:43:25 DEBUG "instance": "10.142.0.25:9100" 05:43:25 DEBUG }, 05:43:25 DEBUG "value": [ 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG [...] 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG "instance": "10.142.0.77:9100" 05:43:25 DEBUG }, 05:43:25 DEBUG "value": [ 05:43:25 DEBUG 1712724205, 05:43:25 DEBUG "37392360.599999994" 05:43:25 DEBUG ] 05:43:25 DEBUG } 05:43:25 DEBUG ] 05:43:25 DEBUG } 05:43:25 DEBUG } 05:43:25 INFO [http_cmd]: http status code OK 05:43:25 DEBUG --- status code --- 05:43:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:25 DEBUG --- http response --- 05:43:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:43:25 DEBUG ----- output ----- 05:43:25 DEBUG { 05:43:25 DEBUG "status": "success", 05:43:25 DEBUG "data": { 05:43:25 DEBUG "resultType": "vector", 05:43:25 DEBUG "result": [ 05:43:25 DEBUG { 05:43:25 DEBUG "metric": { 05:43:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:43:25 DEBUG }, 05:43:25 DEBUG "value": [ 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG [...] 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG "pod": "ds-idrepo-2" 05:43:25 DEBUG }, 05:43:25 DEBUG "value": [ 05:43:25 DEBUG 1712724205, 05:43:25 DEBUG "0" 05:43:25 DEBUG ] 05:43:25 DEBUG } 05:43:25 DEBUG ] 05:43:25 DEBUG } 05:43:25 DEBUG } 05:43:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724205" 05:43:25 INFO [http_cmd]: http status code OK 05:43:25 DEBUG --- status code --- 05:43:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:25 DEBUG --- http response --- 05:43:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:43:25 DEBUG ----- output ----- 05:43:25 DEBUG { 05:43:25 DEBUG "status": "success", 05:43:25 DEBUG "data": { 05:43:25 DEBUG "resultType": "vector", 05:43:25 DEBUG "result": [ 05:43:25 DEBUG { 05:43:25 DEBUG "metric": { 05:43:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:43:25 DEBUG }, 05:43:25 DEBUG "value": [ 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG [...] 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:43:25 DEBUG }, 05:43:25 DEBUG "value": [ 05:43:25 DEBUG 1712724205, 05:43:25 DEBUG "226715.5" 05:43:25 DEBUG ] 05:43:25 DEBUG } 05:43:25 DEBUG ] 05:43:25 DEBUG } 05:43:25 DEBUG } 05:43:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724205" 05:43:25 INFO [http_cmd]: http status code OK 05:43:25 DEBUG --- status code --- 05:43:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:25 DEBUG --- http response --- 05:43:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:43:25 DEBUG ----- output ----- 05:43:25 DEBUG { 05:43:25 DEBUG "status": "success", 05:43:25 DEBUG "data": { 05:43:25 DEBUG "resultType": "vector", 05:43:25 DEBUG "result": [ 05:43:25 DEBUG { 05:43:25 DEBUG "metric": { 05:43:25 DEBUG "pod": "ds-idrepo-0" 05:43:25 DEBUG }, 05:43:25 DEBUG "value": [ 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG [...] 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG 05:43:25 DEBUG "pod": "ds-idrepo-1" 05:43:25 DEBUG }, 05:43:25 DEBUG "value": [ 05:43:25 DEBUG 1712724205, 05:43:25 DEBUG "0" 05:43:25 DEBUG ] 05:43:25 DEBUG } 05:43:25 DEBUG ] 05:43:25 DEBUG } 05:43:25 DEBUG } 05:43:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724206" 05:43:26 INFO [http_cmd]: http status code OK 05:43:26 DEBUG --- status code --- 05:43:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:26 DEBUG --- http response --- 05:43:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:43:26 DEBUG ----- output ----- 05:43:26 DEBUG { 05:43:26 DEBUG "status": "success", 05:43:26 DEBUG "data": { 05:43:26 DEBUG "resultType": "vector", 05:43:26 DEBUG "result": [ 05:43:26 DEBUG { 05:43:26 DEBUG "metric": { 05:43:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:43:26 DEBUG }, 05:43:26 DEBUG "value": [ 05:43:26 DEBUG 05:43:26 DEBUG 05:43:26 DEBUG 05:43:26 DEBUG [...] 05:43:26 DEBUG 05:43:26 DEBUG 05:43:26 DEBUG 05:43:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:43:26 DEBUG }, 05:43:26 DEBUG "value": [ 05:43:26 DEBUG 1712724206, 05:43:26 DEBUG "1249.4666666666667" 05:43:26 DEBUG ] 05:43:26 DEBUG } 05:43:26 DEBUG ] 05:43:26 DEBUG } 05:43:26 DEBUG } 05:43:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724206" 05:43:26 INFO [http_cmd]: http status code OK 05:43:26 DEBUG --- status code --- 05:43:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:26 DEBUG --- http response --- 05:43:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:43:26 DEBUG ----- output ----- 05:43:26 DEBUG { 05:43:26 DEBUG "status": "success", 05:43:26 DEBUG "data": { 05:43:26 DEBUG "resultType": "vector", 05:43:26 DEBUG "result": [ 05:43:26 DEBUG { 05:43:26 DEBUG "metric": { 05:43:26 DEBUG "instance": "10.142.0.25:9100" 05:43:26 DEBUG }, 05:43:26 DEBUG "value": [ 05:43:26 DEBUG 05:43:26 DEBUG 05:43:26 DEBUG 05:43:26 DEBUG [...] 05:43:26 DEBUG 05:43:26 DEBUG 05:43:26 DEBUG 05:43:26 DEBUG "instance": "10.142.0.77:9100" 05:43:26 DEBUG }, 05:43:26 DEBUG "value": [ 05:43:26 DEBUG 1712724206, 05:43:26 DEBUG "0.034066666666667446" 05:43:26 DEBUG ] 05:43:26 DEBUG } 05:43:26 DEBUG ] 05:43:26 DEBUG } 05:43:26 DEBUG } 05:43:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724207" 05:43:27 INFO [http_cmd]: http status code OK 05:43:27 DEBUG --- status code --- 05:43:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:27 DEBUG --- http response --- 05:43:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:43:27 DEBUG ----- output ----- 05:43:27 DEBUG { 05:43:27 DEBUG "status": "success", 05:43:27 DEBUG "data": { 05:43:27 DEBUG "resultType": "vector", 05:43:27 DEBUG "result": [ 05:43:27 DEBUG { 05:43:27 DEBUG "metric": { 05:43:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:43:27 DEBUG }, 05:43:27 DEBUG "value": [ 05:43:27 DEBUG 05:43:27 DEBUG 05:43:27 DEBUG 05:43:27 DEBUG [...] 05:43:27 DEBUG 05:43:27 DEBUG 05:43:27 DEBUG 05:43:27 DEBUG "pod": "ds-idrepo-1" 05:43:27 DEBUG }, 05:43:27 DEBUG "value": [ 05:43:27 DEBUG 1712724207, 05:43:27 DEBUG "730.8917944932053" 05:43:27 DEBUG ] 05:43:27 DEBUG } 05:43:27 DEBUG ] 05:43:27 DEBUG } 05:43:27 DEBUG } 05:43:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712724207" 05:43:27 INFO [http_cmd]: http status code OK 05:43:27 DEBUG --- status code --- 05:43:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:27 DEBUG --- http response --- 05:43:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:43:27 DEBUG ----- output ----- 05:43:27 DEBUG { 05:43:27 DEBUG "status": "success", 05:43:27 DEBUG "data": { 05:43:27 DEBUG "resultType": "vector", 05:43:27 DEBUG "result": [ 05:43:27 DEBUG { 05:43:27 DEBUG "metric": { 05:43:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:43:27 DEBUG }, 05:43:27 DEBUG "value": [ 05:43:27 DEBUG 05:43:27 DEBUG 05:43:27 DEBUG 05:43:27 DEBUG [...] 05:43:27 DEBUG 05:43:27 DEBUG 05:43:27 DEBUG 05:43:27 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:43:27 DEBUG }, 05:43:27 DEBUG "value": [ 05:43:27 DEBUG 1712724207, 05:43:27 DEBUG "3.6253527067600086" 05:43:27 DEBUG ] 05:43:27 DEBUG } 05:43:27 DEBUG ] 05:43:27 DEBUG } 05:43:27 DEBUG } 05:43:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712724207" 05:43:27 INFO [http_cmd]: http status code OK 05:43:27 DEBUG --- status code --- 05:43:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:27 DEBUG --- http response --- 05:43:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:43:27 DEBUG ----- output ----- 05:43:27 DEBUG { 05:43:27 DEBUG "status": "success", 05:43:27 DEBUG "data": { 05:43:27 DEBUG "resultType": "vector", 05:43:27 DEBUG "result": [ 05:43:27 DEBUG { 05:43:27 DEBUG "metric": { 05:43:27 DEBUG "pod": "ds-idrepo-0" 05:43:27 DEBUG }, 05:43:27 DEBUG "value": [ 05:43:27 DEBUG 05:43:27 DEBUG 05:43:27 DEBUG 05:43:27 DEBUG [...] 05:43:27 DEBUG 05:43:27 DEBUG 05:43:27 DEBUG 05:43:27 DEBUG "pod": "ds-cts-2" 05:43:27 DEBUG }, 05:43:27 DEBUG "value": [ 05:43:27 DEBUG 1712724207, 05:43:27 DEBUG "0" 05:43:27 DEBUG ] 05:43:27 DEBUG } 05:43:27 DEBUG ] 05:43:27 DEBUG } 05:43:27 DEBUG } 05:43:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724207" 05:43:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724207" 05:43:27 INFO [http_cmd]: http status code OK 05:43:27 DEBUG --- status code --- 05:43:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:27 DEBUG --- http response --- 05:43:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:43:27 DEBUG ----- output ----- 05:43:27 DEBUG { 05:43:27 DEBUG "status": "success", 05:43:27 DEBUG "data": { 05:43:27 DEBUG "resultType": "vector", 05:43:27 DEBUG "result": [ 05:43:27 DEBUG { 05:43:27 DEBUG "metric": { 05:43:27 DEBUG "pod": "ds-idrepo-2" 05:43:27 DEBUG }, 05:43:27 DEBUG "value": [ 05:43:27 DEBUG 05:43:27 DEBUG 05:43:27 DEBUG 05:43:27 DEBUG [...] 05:43:27 DEBUG 05:43:27 DEBUG 05:43:27 DEBUG 05:43:27 DEBUG "pod": "ds-cts-2" 05:43:27 DEBUG }, 05:43:27 DEBUG "value": [ 05:43:27 DEBUG 1712724207, 05:43:27 DEBUG "0" 05:43:27 DEBUG ] 05:43:27 DEBUG } 05:43:27 DEBUG ] 05:43:27 DEBUG } 05:43:27 DEBUG } 05:43:27 INFO [http_cmd]: http status code OK 05:43:27 DEBUG --- status code --- 05:43:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:27 DEBUG --- http response --- 05:43:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:43:27 DEBUG ----- output ----- 05:43:27 DEBUG { 05:43:27 DEBUG "status": "success", 05:43:27 DEBUG "data": { 05:43:27 DEBUG "resultType": "vector", 05:43:27 DEBUG "result": [ 05:43:27 DEBUG { 05:43:27 DEBUG "metric": { 05:43:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:43:27 DEBUG }, 05:43:27 DEBUG "value": [ 05:43:27 DEBUG 05:43:27 DEBUG 05:43:27 DEBUG 05:43:27 DEBUG [...] 05:43:27 DEBUG 05:43:27 DEBUG 05:43:27 DEBUG 05:43:27 DEBUG "pod": "ds-idrepo-1" 05:43:27 DEBUG }, 05:43:27 DEBUG "value": [ 05:43:27 DEBUG 1712724207, 05:43:27 DEBUG "0.1958635502305562" 05:43:27 DEBUG ] 05:43:27 DEBUG } 05:43:27 DEBUG ] 05:43:27 DEBUG } 05:43:27 DEBUG } 05:43:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724207" 05:43:27 INFO [http_cmd]: http status code OK 05:43:27 DEBUG --- status code --- 05:43:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:27 DEBUG --- http response --- 05:43:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:43:27 DEBUG ----- output ----- 05:43:27 DEBUG { 05:43:27 DEBUG "status": "success", 05:43:27 DEBUG "data": { 05:43:27 DEBUG "resultType": "vector", 05:43:27 DEBUG "result": [ 05:43:27 DEBUG { 05:43:27 DEBUG "metric": { 05:43:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:43:27 DEBUG }, 05:43:27 DEBUG "value": [ 05:43:27 DEBUG 05:43:27 DEBUG 05:43:27 DEBUG 05:43:27 DEBUG [...] 05:43:27 DEBUG 05:43:27 DEBUG 05:43:27 DEBUG 05:43:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:43:27 DEBUG }, 05:43:27 DEBUG "value": [ 05:43:27 DEBUG 1712724207, 05:43:27 DEBUG "1249.8333333333333" 05:43:27 DEBUG ] 05:43:27 DEBUG } 05:43:27 DEBUG ] 05:43:27 DEBUG } 05:43:27 DEBUG } 05:43:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712724207" 05:43:27 INFO [http_cmd]: http status code OK 05:43:27 DEBUG --- status code --- 05:43:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:27 DEBUG --- http response --- 05:43:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:43:27 DEBUG ----- output ----- 05:43:27 DEBUG { 05:43:27 DEBUG "status": "success", 05:43:27 DEBUG "data": { 05:43:27 DEBUG "resultType": "vector", 05:43:27 DEBUG "result": [ 05:43:27 DEBUG { 05:43:27 DEBUG "metric": { 05:43:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:43:27 DEBUG }, 05:43:27 DEBUG "value": [ 05:43:27 DEBUG 05:43:27 DEBUG 05:43:27 DEBUG 05:43:27 DEBUG [...] 05:43:27 DEBUG 05:43:27 DEBUG 05:43:27 DEBUG 05:43:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:43:27 DEBUG }, 05:43:27 DEBUG "value": [ 05:43:27 DEBUG 1712724207, 05:43:27 DEBUG "0.043" 05:43:27 DEBUG ] 05:43:27 DEBUG } 05:43:27 DEBUG ] 05:43:27 DEBUG } 05:43:27 DEBUG } 05:43:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724208" 05:43:28 INFO [http_cmd]: http status code OK 05:43:28 DEBUG --- status code --- 05:43:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:28 DEBUG --- http response --- 05:43:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:43:28 DEBUG ----- output ----- 05:43:28 DEBUG { 05:43:28 DEBUG "status": "success", 05:43:28 DEBUG "data": { 05:43:28 DEBUG "resultType": "vector", 05:43:28 DEBUG "result": [ 05:43:28 DEBUG { 05:43:28 DEBUG "metric": { 05:43:28 DEBUG "pod": "ds-idrepo-0" 05:43:28 DEBUG }, 05:43:28 DEBUG "value": [ 05:43:28 DEBUG 05:43:28 DEBUG 05:43:28 DEBUG 05:43:28 DEBUG [...] 05:43:28 DEBUG 05:43:28 DEBUG 05:43:28 DEBUG 05:43:28 DEBUG "pod": "ds-idrepo-1" 05:43:28 DEBUG }, 05:43:28 DEBUG "value": [ 05:43:28 DEBUG 1712724208, 05:43:28 DEBUG "0" 05:43:28 DEBUG ] 05:43:28 DEBUG } 05:43:28 DEBUG ] 05:43:28 DEBUG } 05:43:28 DEBUG } 05:43:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724208" 05:43:28 INFO [http_cmd]: http status code OK 05:43:28 DEBUG --- status code --- 05:43:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:28 DEBUG --- http response --- 05:43:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:43:28 DEBUG ----- output ----- 05:43:28 DEBUG { 05:43:28 DEBUG "status": "success", 05:43:28 DEBUG "data": { 05:43:28 DEBUG "resultType": "vector", 05:43:28 DEBUG "result": [ 05:43:28 DEBUG { 05:43:28 DEBUG "metric": { 05:43:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:43:28 DEBUG }, 05:43:28 DEBUG "value": [ 05:43:28 DEBUG 05:43:28 DEBUG 05:43:28 DEBUG 05:43:28 DEBUG [...] 05:43:28 DEBUG 05:43:28 DEBUG 05:43:28 DEBUG 05:43:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:43:28 DEBUG }, 05:43:28 DEBUG "value": [ 05:43:28 DEBUG 1712724208, 05:43:28 DEBUG "1264.4" 05:43:28 DEBUG ] 05:43:28 DEBUG } 05:43:28 DEBUG ] 05:43:28 DEBUG } 05:43:28 DEBUG } 05:43:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724208" 05:43:28 INFO [http_cmd]: http status code OK 05:43:28 DEBUG --- status code --- 05:43:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:28 DEBUG --- http response --- 05:43:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:43:28 DEBUG ----- output ----- 05:43:28 DEBUG { 05:43:28 DEBUG "status": "success", 05:43:28 DEBUG "data": { 05:43:28 DEBUG "resultType": "vector", 05:43:28 DEBUG "result": [ 05:43:28 DEBUG { 05:43:28 DEBUG "metric": { 05:43:28 DEBUG "pod": "ds-idrepo-0" 05:43:28 DEBUG }, 05:43:28 DEBUG "value": [ 05:43:28 DEBUG 05:43:28 DEBUG 05:43:28 DEBUG 05:43:28 DEBUG [...] 05:43:28 DEBUG 05:43:28 DEBUG 05:43:28 DEBUG 05:43:28 DEBUG "pod": "ds-idrepo-1" 05:43:28 DEBUG }, 05:43:28 DEBUG "value": [ 05:43:28 DEBUG 1712724208, 05:43:28 DEBUG "0" 05:43:28 DEBUG ] 05:43:28 DEBUG } 05:43:28 DEBUG ] 05:43:28 DEBUG } 05:43:28 DEBUG } 05:43:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712724208" 05:43:28 INFO [http_cmd]: http status code OK 05:43:28 DEBUG --- status code --- 05:43:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:28 DEBUG --- http response --- 05:43:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:43:28 DEBUG ----- output ----- 05:43:28 DEBUG { 05:43:28 DEBUG "status": "success", 05:43:28 DEBUG "data": { 05:43:28 DEBUG "resultType": "vector", 05:43:28 DEBUG "result": [ 05:43:28 DEBUG { 05:43:28 DEBUG "metric": { 05:43:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:43:28 DEBUG }, 05:43:28 DEBUG "value": [ 05:43:28 DEBUG 05:43:28 DEBUG 05:43:28 DEBUG 05:43:28 DEBUG [...] 05:43:28 DEBUG 05:43:28 DEBUG 05:43:28 DEBUG 05:43:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:43:28 DEBUG }, 05:43:28 DEBUG "value": [ 05:43:28 DEBUG 1712724208, 05:43:28 DEBUG "2490556416" 05:43:28 DEBUG ] 05:43:28 DEBUG } 05:43:28 DEBUG ] 05:43:28 DEBUG } 05:43:28 DEBUG } 05:43:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724208" 05:43:28 INFO [http_cmd]: http status code OK 05:43:28 DEBUG --- status code --- 05:43:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:28 DEBUG --- http response --- 05:43:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:43:28 DEBUG ----- output ----- 05:43:28 DEBUG { 05:43:28 DEBUG "status": "success", 05:43:28 DEBUG "data": { 05:43:28 DEBUG "resultType": "vector", 05:43:28 DEBUG "result": [ 05:43:28 DEBUG { 05:43:28 DEBUG "metric": { 05:43:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:43:28 DEBUG }, 05:43:28 DEBUG "value": [ 05:43:28 DEBUG 05:43:28 DEBUG 05:43:28 DEBUG 05:43:28 DEBUG [...] 05:43:28 DEBUG 05:43:28 DEBUG 05:43:28 DEBUG 05:43:28 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 05:43:28 DEBUG }, 05:43:28 DEBUG "value": [ 05:43:28 DEBUG 1712724208, 05:43:28 DEBUG "0" 05:43:28 DEBUG ] 05:43:28 DEBUG } 05:43:28 DEBUG ] 05:43:28 DEBUG } 05:43:28 DEBUG } 05:43:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724208" 05:43:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724208" 05:43:28 INFO [http_cmd]: http status code OK 05:43:28 DEBUG --- status code --- 05:43:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:28 DEBUG --- http response --- 05:43:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:43:28 DEBUG ----- output ----- 05:43:28 INFO [http_cmd]: http status code OK 05:43:28 DEBUG { 05:43:28 DEBUG --- status code --- 05:43:28 DEBUG "status": "success", 05:43:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:28 DEBUG "data": { 05:43:28 DEBUG --- http response --- 05:43:28 DEBUG "resultType": "vector", 05:43:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:43:28 DEBUG "result": [ 05:43:28 DEBUG ----- output ----- 05:43:28 DEBUG { 05:43:28 DEBUG { 05:43:28 DEBUG "metric": { 05:43:28 DEBUG "status": "success", 05:43:28 DEBUG "instance": "10.142.0.25:9100" 05:43:28 DEBUG "data": { 05:43:28 DEBUG }, 05:43:28 DEBUG "resultType": "vector", 05:43:28 DEBUG "value": [ 05:43:28 DEBUG "result": [ 05:43:28 DEBUG 05:43:28 DEBUG { 05:43:28 DEBUG 05:43:28 DEBUG "metric": { 05:43:28 DEBUG 05:43:28 DEBUG "instance": "10.142.0.25:9100" 05:43:28 DEBUG [...] 05:43:28 DEBUG }, 05:43:28 DEBUG 05:43:28 DEBUG "value": [ 05:43:28 DEBUG 05:43:28 DEBUG 05:43:28 DEBUG 05:43:28 DEBUG 05:43:28 DEBUG "instance": "10.142.0.77:9100" 05:43:28 DEBUG 05:43:28 DEBUG }, 05:43:28 DEBUG [...] 05:43:28 DEBUG "value": [ 05:43:28 DEBUG 05:43:28 DEBUG 1712724208, 05:43:28 DEBUG 05:43:28 DEBUG "37397088.03333333" 05:43:28 DEBUG 05:43:28 DEBUG ] 05:43:28 DEBUG "instance": "10.142.0.77:9100" 05:43:28 DEBUG } 05:43:28 DEBUG }, 05:43:28 DEBUG ] 05:43:28 DEBUG "value": [ 05:43:28 DEBUG } 05:43:28 DEBUG 1712724208, 05:43:28 DEBUG } 05:43:28 DEBUG "0.013666666666667084" 05:43:28 DEBUG ] 05:43:28 DEBUG } 05:43:28 DEBUG ] 05:43:28 DEBUG } 05:43:28 DEBUG } 05:43:44 INFO 05:43:44 INFO [loop_until]: kubectl --namespace=xlou top pods 05:43:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:43:44 INFO [loop_until]: OK (rc = 0) 05:43:44 DEBUG --- stdout --- 05:43:44 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 6328m 8804Mi am-654c4fcbcf-9scrj 5702m 8421Mi am-654c4fcbcf-v5qb5 5502m 8400Mi ds-cts-0 5494m 12402Mi ds-cts-1 6176m 12362Mi ds-cts-2 6538m 13136Mi ds-idrepo-0 5875m 5622Mi ds-idrepo-1 210m 5671Mi ds-idrepo-2 243m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 2m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 1894m 1154Mi 05:43:44 DEBUG --- stderr --- 05:43:44 DEBUG 05:43:51 INFO 05:43:51 INFO [loop_until]: kubectl --namespace=xlou top node 05:43:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:43:51 INFO [loop_until]: OK (rc = 0) 05:43:51 DEBUG --- stdout --- 05:43:51 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 90m 0% 2306Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 5108m 32% 9286Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 79m 0% 3260Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 5844m 36% 6505Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 4725m 29% 9312Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 6939m 43% 9647Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 78m 0% 857Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 6901m 43% 6288Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 5352m 33% 13041Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 5687m 35% 13762Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 275m 1% 6373Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 5164m 32% 13004Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 265m 1% 6336Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 1685m 10% 4834Mi 8% 05:43:51 DEBUG --- stderr --- 05:43:51 DEBUG 05:44:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712724265" 05:44:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724265" 05:44:25 ERROR [http_cmd]: ERROR 05:44:25 INFO [http_cmd]: http status code OK 05:44:25 ERROR --- status code --- 05:44:25 DEBUG --- status code --- 05:44:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:44:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:25 ERROR --- http response --- 05:44:25 DEBUG --- http response --- 05:44:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:44:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:44:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:44:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724265" 05:44:25 INFO [http_cmd]: http status code OK 05:44:25 DEBUG --- status code --- 05:44:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:25 DEBUG --- http response --- 05:44:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:44:25 DEBUG ----- output ----- 05:44:25 DEBUG { 05:44:25 DEBUG "status": "success", 05:44:25 DEBUG "data": { 05:44:25 DEBUG "resultType": "vector", 05:44:25 DEBUG "result": [ 05:44:25 DEBUG { 05:44:25 DEBUG "metric": { 05:44:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:44:25 DEBUG }, 05:44:25 DEBUG "value": [ 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG [...] 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:44:25 DEBUG }, 05:44:25 DEBUG "value": [ 05:44:25 DEBUG 1712724265, 05:44:25 DEBUG "0" 05:44:25 DEBUG ] 05:44:25 DEBUG } 05:44:25 DEBUG ] 05:44:25 DEBUG } 05:44:25 DEBUG } 05:44:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724265" 05:44:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724265" 05:44:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724265" 05:44:25 INFO [http_cmd]: http status code OK 05:44:25 DEBUG --- status code --- 05:44:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:25 DEBUG --- http response --- 05:44:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:44:25 DEBUG ----- output ----- 05:44:25 DEBUG { 05:44:25 DEBUG "status": "success", 05:44:25 DEBUG "data": { 05:44:25 DEBUG "resultType": "vector", 05:44:25 DEBUG "result": [ 05:44:25 DEBUG { 05:44:25 DEBUG "metric": { 05:44:25 DEBUG "pod": "ds-idrepo-0" 05:44:25 DEBUG }, 05:44:25 DEBUG "value": [ 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG [...] 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG "pod": "ds-cts-2" 05:44:25 DEBUG }, 05:44:25 DEBUG "value": [ 05:44:25 DEBUG 1712724265, 05:44:25 DEBUG "0" 05:44:25 DEBUG ] 05:44:25 DEBUG } 05:44:25 DEBUG ] 05:44:25 DEBUG } 05:44:25 DEBUG } 05:44:25 INFO [http_cmd]: http status code OK 05:44:25 DEBUG --- status code --- 05:44:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:25 DEBUG --- http response --- 05:44:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:44:25 DEBUG ----- output ----- 05:44:25 DEBUG { 05:44:25 DEBUG "status": "success", 05:44:25 DEBUG "data": { 05:44:25 DEBUG "resultType": "vector", 05:44:25 DEBUG "result": [ 05:44:25 DEBUG { 05:44:25 DEBUG "metric": { 05:44:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:44:25 DEBUG }, 05:44:25 DEBUG "value": [ 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG [...] 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:44:25 DEBUG }, 05:44:25 DEBUG "value": [ 05:44:25 DEBUG 1712724265, 05:44:25 DEBUG "0" 05:44:25 DEBUG ] 05:44:25 DEBUG } 05:44:25 DEBUG ] 05:44:25 DEBUG } 05:44:25 DEBUG } 05:44:25 INFO [http_cmd]: http status code OK 05:44:25 DEBUG --- status code --- 05:44:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:25 DEBUG --- http response --- 05:44:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:44:25 DEBUG ----- output ----- 05:44:25 DEBUG { 05:44:25 DEBUG "status": "success", 05:44:25 DEBUG "data": { 05:44:25 DEBUG "resultType": "vector", 05:44:25 DEBUG "result": [ 05:44:25 DEBUG { 05:44:25 DEBUG "metric": { 05:44:25 DEBUG "pod": "ds-idrepo-0" 05:44:25 DEBUG }, 05:44:25 DEBUG "value": [ 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG [...] 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG "pod": "ds-cts-2" 05:44:25 DEBUG }, 05:44:25 DEBUG "value": [ 05:44:25 DEBUG 1712724265, 05:44:25 DEBUG "29.866666666666667" 05:44:25 DEBUG ] 05:44:25 DEBUG } 05:44:25 DEBUG ] 05:44:25 DEBUG } 05:44:25 DEBUG } 05:44:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724265" 05:44:25 INFO [http_cmd]: http status code OK 05:44:25 DEBUG --- status code --- 05:44:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:25 DEBUG --- http response --- 05:44:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:44:25 DEBUG ----- output ----- 05:44:25 DEBUG { 05:44:25 DEBUG "status": "success", 05:44:25 DEBUG "data": { 05:44:25 DEBUG "resultType": "vector", 05:44:25 DEBUG "result": [ 05:44:25 DEBUG { 05:44:25 DEBUG "metric": { 05:44:25 DEBUG "instance": "10.142.0.25:9100" 05:44:25 DEBUG }, 05:44:25 DEBUG "value": [ 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG [...] 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG "instance": "10.142.0.77:9100" 05:44:25 DEBUG }, 05:44:25 DEBUG "value": [ 05:44:25 DEBUG 1712724265, 05:44:25 DEBUG "313328391168" 05:44:25 DEBUG ] 05:44:25 DEBUG } 05:44:25 DEBUG ] 05:44:25 DEBUG } 05:44:25 DEBUG } 05:44:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724265" 05:44:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724265" 05:44:25 INFO [http_cmd]: http status code OK 05:44:25 DEBUG --- status code --- 05:44:25 INFO [http_cmd]: http status code OK 05:44:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:25 DEBUG --- status code --- 05:44:25 DEBUG --- http response --- 05:44:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:44:25 DEBUG --- http response --- 05:44:25 DEBUG ----- output ----- 05:44:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:44:25 DEBUG { 05:44:25 DEBUG ----- output ----- 05:44:25 DEBUG "status": "success", 05:44:25 DEBUG { 05:44:25 DEBUG "data": { 05:44:25 DEBUG "status": "success", 05:44:25 DEBUG "resultType": "vector", 05:44:25 DEBUG "data": { 05:44:25 DEBUG "result": [ 05:44:25 DEBUG "resultType": "vector", 05:44:25 DEBUG { 05:44:25 DEBUG "result": [ 05:44:25 DEBUG "metric": { 05:44:25 DEBUG { 05:44:25 DEBUG "instance": "10.142.0.25:9100" 05:44:25 DEBUG "metric": { 05:44:25 DEBUG }, 05:44:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:44:25 DEBUG "value": [ 05:44:25 DEBUG }, 05:44:25 DEBUG 05:44:25 DEBUG "value": [ 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG [...] 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG [...] 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG "instance": "10.142.0.77:9100" 05:44:25 DEBUG 05:44:25 DEBUG }, 05:44:25 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 05:44:25 DEBUG "value": [ 05:44:25 DEBUG }, 05:44:25 DEBUG 1712724265, 05:44:25 DEBUG "value": [ 05:44:25 DEBUG "8447106.666666666" 05:44:25 DEBUG 1712724265, 05:44:25 DEBUG ] 05:44:25 DEBUG "0" 05:44:25 DEBUG } 05:44:25 DEBUG ] 05:44:25 DEBUG ] 05:44:25 DEBUG } 05:44:25 DEBUG } 05:44:25 DEBUG ] 05:44:25 DEBUG } 05:44:25 DEBUG } 05:44:25 DEBUG } 05:44:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724265" 05:44:25 INFO [http_cmd]: http status code OK 05:44:25 DEBUG --- status code --- 05:44:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:25 DEBUG --- http response --- 05:44:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:44:25 DEBUG ----- output ----- 05:44:25 DEBUG { 05:44:25 DEBUG "status": "success", 05:44:25 DEBUG "data": { 05:44:25 DEBUG "resultType": "vector", 05:44:25 DEBUG "result": [ 05:44:25 DEBUG { 05:44:25 DEBUG "metric": { 05:44:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:44:25 DEBUG }, 05:44:25 DEBUG "value": [ 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG [...] 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:44:25 DEBUG }, 05:44:25 DEBUG "value": [ 05:44:25 DEBUG 1712724265, 05:44:25 DEBUG "185094.5" 05:44:25 DEBUG ] 05:44:25 DEBUG } 05:44:25 DEBUG ] 05:44:25 DEBUG } 05:44:25 DEBUG } 05:44:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724265" 05:44:25 INFO [http_cmd]: http status code OK 05:44:25 DEBUG --- status code --- 05:44:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:25 DEBUG --- http response --- 05:44:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:44:25 DEBUG ----- output ----- 05:44:25 DEBUG { 05:44:25 DEBUG "status": "success", 05:44:25 DEBUG "data": { 05:44:25 DEBUG "resultType": "vector", 05:44:25 DEBUG "result": [ 05:44:25 DEBUG { 05:44:25 DEBUG "metric": { 05:44:25 DEBUG "pod": "ds-idrepo-0" 05:44:25 DEBUG }, 05:44:25 DEBUG "value": [ 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG [...] 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG 05:44:25 DEBUG "pod": "ds-idrepo-1" 05:44:25 DEBUG }, 05:44:25 DEBUG "value": [ 05:44:25 DEBUG 1712724265, 05:44:25 DEBUG "0" 05:44:25 DEBUG ] 05:44:25 DEBUG } 05:44:25 DEBUG ] 05:44:25 DEBUG } 05:44:25 DEBUG } 05:44:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724266" 05:44:26 INFO [http_cmd]: http status code OK 05:44:26 DEBUG --- status code --- 05:44:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:26 DEBUG --- http response --- 05:44:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:44:26 DEBUG ----- output ----- 05:44:26 DEBUG { 05:44:26 DEBUG "status": "success", 05:44:26 DEBUG "data": { 05:44:26 DEBUG "resultType": "vector", 05:44:26 DEBUG "result": [ 05:44:26 DEBUG { 05:44:26 DEBUG "metric": { 05:44:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:44:26 DEBUG }, 05:44:26 DEBUG "value": [ 05:44:26 DEBUG 05:44:26 DEBUG 05:44:26 DEBUG 05:44:26 DEBUG [...] 05:44:26 DEBUG 05:44:26 DEBUG 05:44:26 DEBUG 05:44:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:44:26 DEBUG }, 05:44:26 DEBUG "value": [ 05:44:26 DEBUG 1712724266, 05:44:26 DEBUG "0" 05:44:26 DEBUG ] 05:44:26 DEBUG } 05:44:26 DEBUG ] 05:44:26 DEBUG } 05:44:26 DEBUG } 05:44:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724266" 05:44:26 INFO [http_cmd]: http status code OK 05:44:26 DEBUG --- status code --- 05:44:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:26 DEBUG --- http response --- 05:44:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:44:26 DEBUG ----- output ----- 05:44:26 DEBUG { 05:44:26 DEBUG "status": "success", 05:44:26 DEBUG "data": { 05:44:26 DEBUG "resultType": "vector", 05:44:26 DEBUG "result": [ 05:44:26 DEBUG { 05:44:26 DEBUG "metric": { 05:44:26 DEBUG "instance": "10.142.0.25:9100" 05:44:26 DEBUG }, 05:44:26 DEBUG "value": [ 05:44:26 DEBUG 05:44:26 DEBUG 05:44:26 DEBUG 05:44:26 DEBUG [...] 05:44:26 DEBUG 05:44:26 DEBUG 05:44:26 DEBUG 05:44:26 DEBUG "instance": "10.142.0.77:9100" 05:44:26 DEBUG }, 05:44:26 DEBUG "value": [ 05:44:26 DEBUG 1712724266, 05:44:26 DEBUG "0.1138666666666617" 05:44:26 DEBUG ] 05:44:26 DEBUG } 05:44:26 DEBUG ] 05:44:26 DEBUG } 05:44:26 DEBUG } 05:44:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724267" 05:44:27 INFO [http_cmd]: http status code OK 05:44:27 DEBUG --- status code --- 05:44:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:27 DEBUG --- http response --- 05:44:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:44:27 DEBUG ----- output ----- 05:44:27 DEBUG { 05:44:27 DEBUG "status": "success", 05:44:27 DEBUG "data": { 05:44:27 DEBUG "resultType": "vector", 05:44:27 DEBUG "result": [ 05:44:27 DEBUG { 05:44:27 DEBUG "metric": { 05:44:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:44:27 DEBUG }, 05:44:27 DEBUG "value": [ 05:44:27 DEBUG 05:44:27 DEBUG 05:44:27 DEBUG 05:44:27 DEBUG [...] 05:44:27 DEBUG 05:44:27 DEBUG 05:44:27 DEBUG 05:44:27 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:44:27 DEBUG }, 05:44:27 DEBUG "value": [ 05:44:27 DEBUG 1712724267, 05:44:27 DEBUG "0" 05:44:27 DEBUG ] 05:44:27 DEBUG } 05:44:27 DEBUG ] 05:44:27 DEBUG } 05:44:27 DEBUG } 05:44:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712724267" 05:44:27 INFO [http_cmd]: http status code OK 05:44:27 DEBUG --- status code --- 05:44:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:27 DEBUG --- http response --- 05:44:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:44:27 DEBUG ----- output ----- 05:44:27 DEBUG { 05:44:27 DEBUG "status": "success", 05:44:27 DEBUG "data": { 05:44:27 DEBUG "resultType": "vector", 05:44:27 DEBUG "result": [ 05:44:27 DEBUG { 05:44:27 DEBUG "metric": { 05:44:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:44:27 DEBUG }, 05:44:27 DEBUG "value": [ 05:44:27 DEBUG 05:44:27 DEBUG 05:44:27 DEBUG 05:44:27 DEBUG [...] 05:44:27 DEBUG 05:44:27 DEBUG 05:44:27 DEBUG 05:44:27 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:44:27 DEBUG }, 05:44:27 DEBUG "value": [ 05:44:27 DEBUG 1712724267, 05:44:27 DEBUG "0.0267854516864272" 05:44:27 DEBUG ] 05:44:27 DEBUG } 05:44:27 DEBUG ] 05:44:27 DEBUG } 05:44:27 DEBUG } 05:44:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712724267" 05:44:27 INFO [http_cmd]: http status code OK 05:44:27 DEBUG --- status code --- 05:44:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:27 DEBUG --- http response --- 05:44:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:44:27 DEBUG ----- output ----- 05:44:27 DEBUG { 05:44:27 DEBUG "status": "success", 05:44:27 DEBUG "data": { 05:44:27 DEBUG "resultType": "vector", 05:44:27 DEBUG "result": [ 05:44:27 DEBUG { 05:44:27 DEBUG "metric": { 05:44:27 DEBUG "pod": "ds-idrepo-0" 05:44:27 DEBUG }, 05:44:27 DEBUG "value": [ 05:44:27 DEBUG 05:44:27 DEBUG 05:44:27 DEBUG 05:44:27 DEBUG [...] 05:44:27 DEBUG 05:44:27 DEBUG 05:44:27 DEBUG 05:44:27 DEBUG "pod": "ds-cts-2" 05:44:27 DEBUG }, 05:44:27 DEBUG "value": [ 05:44:27 DEBUG 1712724267, 05:44:27 DEBUG "0" 05:44:27 DEBUG ] 05:44:27 DEBUG } 05:44:27 DEBUG ] 05:44:27 DEBUG } 05:44:27 DEBUG } 05:44:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724267" 05:44:27 INFO [http_cmd]: http status code OK 05:44:27 DEBUG --- status code --- 05:44:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:27 DEBUG --- http response --- 05:44:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:44:27 DEBUG ----- output ----- 05:44:27 DEBUG { 05:44:27 DEBUG "status": "success", 05:44:27 DEBUG "data": { 05:44:27 DEBUG "resultType": "vector", 05:44:27 DEBUG "result": [ 05:44:27 DEBUG { 05:44:27 DEBUG "metric": { 05:44:27 DEBUG "pod": "ds-idrepo-2" 05:44:27 DEBUG }, 05:44:27 DEBUG "value": [ 05:44:27 DEBUG 05:44:27 DEBUG 05:44:27 DEBUG 05:44:27 DEBUG [...] 05:44:27 DEBUG 05:44:27 DEBUG 05:44:27 DEBUG 05:44:27 DEBUG "pod": "ds-cts-2" 05:44:27 DEBUG }, 05:44:27 DEBUG "value": [ 05:44:27 DEBUG 1712724267, 05:44:27 DEBUG "0.00013333333333333334" 05:44:27 DEBUG ] 05:44:27 DEBUG } 05:44:27 DEBUG ] 05:44:27 DEBUG } 05:44:27 DEBUG } 05:44:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724267" 05:44:27 INFO [http_cmd]: http status code OK 05:44:27 DEBUG --- status code --- 05:44:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:27 DEBUG --- http response --- 05:44:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:44:27 DEBUG ----- output ----- 05:44:27 DEBUG { 05:44:27 DEBUG "status": "success", 05:44:27 DEBUG "data": { 05:44:27 DEBUG "resultType": "vector", 05:44:27 DEBUG "result": [ 05:44:27 DEBUG { 05:44:27 DEBUG "metric": { 05:44:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:44:27 DEBUG }, 05:44:27 DEBUG "value": [ 05:44:27 DEBUG 05:44:27 DEBUG 05:44:27 DEBUG 05:44:27 DEBUG [...] 05:44:27 DEBUG 05:44:27 DEBUG 05:44:27 DEBUG 05:44:27 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:44:27 DEBUG }, 05:44:27 DEBUG "value": [ 05:44:27 DEBUG 1712724267, 05:44:27 DEBUG "0.00015751725943725423" 05:44:27 DEBUG ] 05:44:27 DEBUG } 05:44:27 DEBUG ] 05:44:27 DEBUG } 05:44:27 DEBUG } 05:44:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724267" 05:44:27 INFO [http_cmd]: http status code OK 05:44:27 DEBUG --- status code --- 05:44:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:27 DEBUG --- http response --- 05:44:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:44:27 DEBUG ----- output ----- 05:44:27 DEBUG { 05:44:27 DEBUG "status": "success", 05:44:27 DEBUG "data": { 05:44:27 DEBUG "resultType": "vector", 05:44:27 DEBUG "result": [ 05:44:27 DEBUG { 05:44:27 DEBUG "metric": { 05:44:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:44:27 DEBUG }, 05:44:27 DEBUG "value": [ 05:44:27 DEBUG 05:44:27 DEBUG 05:44:27 DEBUG 05:44:27 DEBUG [...] 05:44:27 DEBUG 05:44:27 DEBUG 05:44:27 DEBUG 05:44:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:44:27 DEBUG }, 05:44:27 DEBUG "value": [ 05:44:27 DEBUG 1712724267, 05:44:27 DEBUG "0" 05:44:27 DEBUG ] 05:44:27 DEBUG } 05:44:27 DEBUG ] 05:44:27 DEBUG } 05:44:27 DEBUG } 05:44:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712724267" 05:44:27 INFO [http_cmd]: http status code OK 05:44:27 DEBUG --- status code --- 05:44:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:27 DEBUG --- http response --- 05:44:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:44:27 DEBUG ----- output ----- 05:44:27 DEBUG { 05:44:27 DEBUG "status": "success", 05:44:27 DEBUG "data": { 05:44:27 DEBUG "resultType": "vector", 05:44:27 DEBUG "result": [ 05:44:27 DEBUG { 05:44:27 DEBUG "metric": { 05:44:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:44:27 DEBUG }, 05:44:27 DEBUG "value": [ 05:44:27 DEBUG 05:44:27 DEBUG 05:44:27 DEBUG 05:44:27 DEBUG [...] 05:44:27 DEBUG 05:44:27 DEBUG 05:44:27 DEBUG 05:44:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:44:27 DEBUG }, 05:44:27 DEBUG "value": [ 05:44:27 DEBUG 1712724267, 05:44:27 DEBUG "0.043" 05:44:27 DEBUG ] 05:44:27 DEBUG } 05:44:27 DEBUG ] 05:44:27 DEBUG } 05:44:27 DEBUG } 05:44:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724268" 05:44:28 INFO [http_cmd]: http status code OK 05:44:28 DEBUG --- status code --- 05:44:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:28 DEBUG --- http response --- 05:44:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:44:28 DEBUG ----- output ----- 05:44:28 DEBUG { 05:44:28 DEBUG "status": "success", 05:44:28 DEBUG "data": { 05:44:28 DEBUG "resultType": "vector", 05:44:28 DEBUG "result": [ 05:44:28 DEBUG { 05:44:28 DEBUG "metric": { 05:44:28 DEBUG "pod": "ds-idrepo-0" 05:44:28 DEBUG }, 05:44:28 DEBUG "value": [ 05:44:28 DEBUG 05:44:28 DEBUG 05:44:28 DEBUG 05:44:28 DEBUG [...] 05:44:28 DEBUG 05:44:28 DEBUG 05:44:28 DEBUG 05:44:28 DEBUG "pod": "ds-idrepo-1" 05:44:28 DEBUG }, 05:44:28 DEBUG "value": [ 05:44:28 DEBUG 1712724268, 05:44:28 DEBUG "0" 05:44:28 DEBUG ] 05:44:28 DEBUG } 05:44:28 DEBUG ] 05:44:28 DEBUG } 05:44:28 DEBUG } 05:44:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724268" 05:44:28 INFO [http_cmd]: http status code OK 05:44:28 DEBUG --- status code --- 05:44:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:28 DEBUG --- http response --- 05:44:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:44:28 DEBUG ----- output ----- 05:44:28 DEBUG { 05:44:28 DEBUG "status": "success", 05:44:28 DEBUG "data": { 05:44:28 DEBUG "resultType": "vector", 05:44:28 DEBUG "result": [ 05:44:28 DEBUG { 05:44:28 DEBUG "metric": { 05:44:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:44:28 DEBUG }, 05:44:28 DEBUG "value": [ 05:44:28 DEBUG 05:44:28 DEBUG 05:44:28 DEBUG 05:44:28 DEBUG [...] 05:44:28 DEBUG 05:44:28 DEBUG 05:44:28 DEBUG 05:44:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:44:28 DEBUG }, 05:44:28 DEBUG "value": [ 05:44:28 DEBUG 1712724268, 05:44:28 DEBUG "1226.6333333333332" 05:44:28 DEBUG ] 05:44:28 DEBUG } 05:44:28 DEBUG ] 05:44:28 DEBUG } 05:44:28 DEBUG } 05:44:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724268" 05:44:28 INFO [http_cmd]: http status code OK 05:44:28 DEBUG --- status code --- 05:44:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:28 DEBUG --- http response --- 05:44:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:44:28 DEBUG ----- output ----- 05:44:28 DEBUG { 05:44:28 DEBUG "status": "success", 05:44:28 DEBUG "data": { 05:44:28 DEBUG "resultType": "vector", 05:44:28 DEBUG "result": [ 05:44:28 DEBUG { 05:44:28 DEBUG "metric": { 05:44:28 DEBUG "pod": "ds-idrepo-0" 05:44:28 DEBUG }, 05:44:28 DEBUG "value": [ 05:44:28 DEBUG 05:44:28 DEBUG 05:44:28 DEBUG 05:44:28 DEBUG [...] 05:44:28 DEBUG 05:44:28 DEBUG 05:44:28 DEBUG 05:44:28 DEBUG "pod": "ds-idrepo-1" 05:44:28 DEBUG }, 05:44:28 DEBUG "value": [ 05:44:28 DEBUG 1712724268, 05:44:28 DEBUG "0" 05:44:28 DEBUG ] 05:44:28 DEBUG } 05:44:28 DEBUG ] 05:44:28 DEBUG } 05:44:28 DEBUG } 05:44:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712724268" 05:44:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724268" 05:44:28 INFO [http_cmd]: http status code OK 05:44:28 DEBUG --- status code --- 05:44:28 INFO [http_cmd]: http status code OK 05:44:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:28 DEBUG --- status code --- 05:44:28 DEBUG --- http response --- 05:44:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:44:28 DEBUG --- http response --- 05:44:28 DEBUG ----- output ----- 05:44:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:44:28 DEBUG { 05:44:28 DEBUG ----- output ----- 05:44:28 DEBUG "status": "success", 05:44:28 DEBUG { 05:44:28 DEBUG "data": { 05:44:28 DEBUG "status": "success", 05:44:28 DEBUG "resultType": "vector", 05:44:28 DEBUG "data": { 05:44:28 DEBUG "result": [ 05:44:28 DEBUG "resultType": "vector", 05:44:28 DEBUG { 05:44:28 DEBUG "result": [ 05:44:28 DEBUG "metric": { 05:44:28 DEBUG { 05:44:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:44:28 DEBUG "metric": { 05:44:28 DEBUG }, 05:44:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:44:28 DEBUG "value": [ 05:44:28 DEBUG }, 05:44:28 DEBUG 05:44:28 DEBUG "value": [ 05:44:28 DEBUG 05:44:28 DEBUG 05:44:28 DEBUG 05:44:28 DEBUG 05:44:28 DEBUG [...] 05:44:28 DEBUG 05:44:28 DEBUG 05:44:28 DEBUG [...] 05:44:28 DEBUG 05:44:28 DEBUG 05:44:28 DEBUG 05:44:28 DEBUG 05:44:28 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:44:28 DEBUG 05:44:28 DEBUG }, 05:44:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:44:28 DEBUG "value": [ 05:44:28 DEBUG }, 05:44:28 DEBUG 1712724268, 05:44:28 DEBUG "value": [ 05:44:28 DEBUG "0" 05:44:28 DEBUG 1712724268, 05:44:28 DEBUG ] 05:44:28 DEBUG "2490064896" 05:44:28 DEBUG } 05:44:28 DEBUG ] 05:44:28 DEBUG ] 05:44:28 DEBUG } 05:44:28 DEBUG } 05:44:28 DEBUG ] 05:44:28 DEBUG } 05:44:28 DEBUG } 05:44:28 DEBUG } 05:44:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724268" 05:44:28 INFO [http_cmd]: http status code OK 05:44:28 DEBUG --- status code --- 05:44:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:28 DEBUG --- http response --- 05:44:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:44:28 DEBUG ----- output ----- 05:44:28 DEBUG { 05:44:28 DEBUG "status": "success", 05:44:28 DEBUG "data": { 05:44:28 DEBUG "resultType": "vector", 05:44:28 DEBUG "result": [ 05:44:28 DEBUG { 05:44:28 DEBUG "metric": { 05:44:28 DEBUG "instance": "10.142.0.25:9100" 05:44:28 DEBUG }, 05:44:28 DEBUG "value": [ 05:44:28 DEBUG 05:44:28 DEBUG 05:44:28 DEBUG 05:44:28 DEBUG [...] 05:44:28 DEBUG 05:44:28 DEBUG 05:44:28 DEBUG 05:44:28 DEBUG "instance": "10.142.0.77:9100" 05:44:28 DEBUG }, 05:44:28 DEBUG "value": [ 05:44:28 DEBUG 1712724268, 05:44:28 DEBUG "8448098.8" 05:44:28 DEBUG ] 05:44:28 DEBUG } 05:44:28 DEBUG ] 05:44:28 DEBUG } 05:44:28 DEBUG } 05:44:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724268" 05:44:28 INFO [http_cmd]: http status code OK 05:44:28 DEBUG --- status code --- 05:44:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:28 DEBUG --- http response --- 05:44:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:44:28 DEBUG ----- output ----- 05:44:28 DEBUG { 05:44:28 DEBUG "status": "success", 05:44:28 DEBUG "data": { 05:44:28 DEBUG "resultType": "vector", 05:44:28 DEBUG "result": [ 05:44:28 DEBUG { 05:44:28 DEBUG "metric": { 05:44:28 DEBUG "instance": "10.142.0.25:9100" 05:44:28 DEBUG }, 05:44:28 DEBUG "value": [ 05:44:28 DEBUG 05:44:28 DEBUG 05:44:28 DEBUG 05:44:28 DEBUG [...] 05:44:28 DEBUG 05:44:28 DEBUG 05:44:28 DEBUG 05:44:28 DEBUG "instance": "10.142.0.77:9100" 05:44:28 DEBUG }, 05:44:28 DEBUG "value": [ 05:44:28 DEBUG 1712724268, 05:44:28 DEBUG "0.02900000000000015" 05:44:28 DEBUG ] 05:44:28 DEBUG } 05:44:28 DEBUG ] 05:44:28 DEBUG } 05:44:28 DEBUG } 05:44:44 INFO 05:44:44 INFO [loop_until]: kubectl --namespace=xlou top pods 05:44:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:44:44 INFO [loop_until]: OK (rc = 0) 05:44:44 DEBUG --- stdout --- 05:44:44 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 1537m 8760Mi am-654c4fcbcf-9scrj 1462m 8390Mi am-654c4fcbcf-v5qb5 1250m 8378Mi ds-cts-0 3273m 12466Mi ds-cts-1 3582m 12930Mi ds-cts-2 3309m 12254Mi ds-idrepo-0 1119m 5622Mi ds-idrepo-1 11m 5671Mi ds-idrepo-2 13m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 8m 1190Mi idm-6bfb7c96cc-vsb58 6m 2213Mi lodemon-b776b7d88-thrqf 5m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 1m 1154Mi 05:44:44 DEBUG --- stderr --- 05:44:44 DEBUG 05:44:51 INFO 05:44:51 INFO [loop_until]: kubectl --namespace=xlou top node 05:44:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:44:51 INFO [loop_until]: OK (rc = 0) 05:44:51 DEBUG --- stdout --- 05:44:51 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 98m 0% 2303Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 1416m 8% 9263Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 77m 0% 3260Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 210m 1% 6527Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 1626m 10% 9285Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 1658m 10% 9614Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 79m 0% 856Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 1155m 7% 6287Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 3416m 21% 13114Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 3499m 22% 13331Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 66m 0% 6376Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 3653m 22% 13581Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 65m 0% 6337Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 66m 0% 4830Mi 8% 05:44:51 DEBUG --- stderr --- 05:44:51 DEBUG 05:45:22 WARNING File /tmp/lodemon_data-20240409_230625/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [10/Apr/2024 05:45:22] "GET /monitoring/average?start_time=20240410_034436&stop_time=20240410_044321 HTTP/1.1" 200 - 05:45:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712724325" 05:45:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724325" 05:45:25 ERROR [http_cmd]: ERROR 05:45:25 ERROR --- status code --- 05:45:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:45:25 ERROR --- http response --- 05:45:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:45:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:45:25 INFO [http_cmd]: http status code OK 05:45:25 DEBUG --- status code --- 05:45:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:25 DEBUG --- http response --- 05:45:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:45:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724325" 05:45:25 INFO [http_cmd]: http status code OK 05:45:25 DEBUG --- status code --- 05:45:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:25 DEBUG --- http response --- 05:45:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:45:25 DEBUG ----- output ----- 05:45:25 DEBUG { 05:45:25 DEBUG "status": "success", 05:45:25 DEBUG "data": { 05:45:25 DEBUG "resultType": "vector", 05:45:25 DEBUG "result": [ 05:45:25 DEBUG { 05:45:25 DEBUG "metric": { 05:45:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:45:25 DEBUG }, 05:45:25 DEBUG "value": [ 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG [...] 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:45:25 DEBUG }, 05:45:25 DEBUG "value": [ 05:45:25 DEBUG 1712724325, 05:45:25 DEBUG "0" 05:45:25 DEBUG ] 05:45:25 DEBUG } 05:45:25 DEBUG ] 05:45:25 DEBUG } 05:45:25 DEBUG } 05:45:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724325" 05:45:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724325" 05:45:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724325" 05:45:25 INFO [http_cmd]: http status code OK 05:45:25 DEBUG --- status code --- 05:45:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:25 DEBUG --- http response --- 05:45:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:45:25 DEBUG ----- output ----- 05:45:25 DEBUG { 05:45:25 DEBUG "status": "success", 05:45:25 DEBUG "data": { 05:45:25 DEBUG "resultType": "vector", 05:45:25 DEBUG "result": [ 05:45:25 DEBUG { 05:45:25 DEBUG "metric": { 05:45:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:45:25 DEBUG }, 05:45:25 DEBUG "value": [ 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG [...] 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:45:25 DEBUG }, 05:45:25 DEBUG "value": [ 05:45:25 DEBUG 1712724325, 05:45:25 DEBUG "0" 05:45:25 DEBUG ] 05:45:25 DEBUG } 05:45:25 DEBUG ] 05:45:25 DEBUG } 05:45:25 DEBUG } 05:45:25 INFO [http_cmd]: http status code OK 05:45:25 DEBUG --- status code --- 05:45:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:25 DEBUG --- http response --- 05:45:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:45:25 DEBUG ----- output ----- 05:45:25 DEBUG { 05:45:25 DEBUG "status": "success", 05:45:25 DEBUG "data": { 05:45:25 DEBUG "resultType": "vector", 05:45:25 DEBUG "result": [ 05:45:25 DEBUG { 05:45:25 DEBUG "metric": { 05:45:25 DEBUG "pod": "ds-idrepo-0" 05:45:25 DEBUG }, 05:45:25 DEBUG "value": [ 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG [...] 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG "pod": "ds-cts-2" 05:45:25 DEBUG }, 05:45:25 DEBUG "value": [ 05:45:25 DEBUG 1712724325, 05:45:25 DEBUG "0" 05:45:25 DEBUG ] 05:45:25 DEBUG } 05:45:25 DEBUG ] 05:45:25 DEBUG } 05:45:25 DEBUG } 05:45:25 INFO [http_cmd]: http status code OK 05:45:25 DEBUG --- status code --- 05:45:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:25 DEBUG --- http response --- 05:45:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:45:25 DEBUG ----- output ----- 05:45:25 DEBUG { 05:45:25 DEBUG "status": "success", 05:45:25 DEBUG "data": { 05:45:25 DEBUG "resultType": "vector", 05:45:25 DEBUG "result": [ 05:45:25 DEBUG { 05:45:25 DEBUG "metric": { 05:45:25 DEBUG "pod": "ds-idrepo-0" 05:45:25 DEBUG }, 05:45:25 DEBUG "value": [ 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG [...] 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG "pod": "ds-cts-2" 05:45:25 DEBUG }, 05:45:25 DEBUG "value": [ 05:45:25 DEBUG 1712724325, 05:45:25 DEBUG "29.933333333333334" 05:45:25 DEBUG ] 05:45:25 DEBUG } 05:45:25 DEBUG ] 05:45:25 DEBUG } 05:45:25 DEBUG } 05:45:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724325" 05:45:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724325" 05:45:25 INFO [http_cmd]: http status code OK 05:45:25 INFO [http_cmd]: http status code OK 05:45:25 DEBUG --- status code --- 05:45:25 DEBUG --- status code --- 05:45:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:25 DEBUG --- http response --- 05:45:25 DEBUG --- http response --- 05:45:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:45:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:45:25 DEBUG ----- output ----- 05:45:25 DEBUG ----- output ----- 05:45:25 DEBUG { 05:45:25 DEBUG { 05:45:25 DEBUG "status": "success", 05:45:25 DEBUG "status": "success", 05:45:25 DEBUG "data": { 05:45:25 DEBUG "data": { 05:45:25 DEBUG "resultType": "vector", 05:45:25 DEBUG "resultType": "vector", 05:45:25 DEBUG "result": [ 05:45:25 DEBUG "result": [ 05:45:25 DEBUG { 05:45:25 DEBUG { 05:45:25 DEBUG "metric": { 05:45:25 DEBUG "metric": { 05:45:25 DEBUG "instance": "10.142.0.25:9100" 05:45:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:45:25 DEBUG }, 05:45:25 DEBUG }, 05:45:25 DEBUG "value": [ 05:45:25 DEBUG "value": [ 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG [...] 05:45:25 DEBUG [...] 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG "instance": "10.142.0.77:9100" 05:45:25 DEBUG "pod": "ds-idrepo-2" 05:45:25 DEBUG }, 05:45:25 DEBUG }, 05:45:25 DEBUG "value": [ 05:45:25 DEBUG "value": [ 05:45:25 DEBUG 1712724325, 05:45:25 DEBUG 1712724325, 05:45:25 DEBUG "313720591360" 05:45:25 DEBUG "0" 05:45:25 DEBUG ] 05:45:25 DEBUG ] 05:45:25 DEBUG } 05:45:25 DEBUG } 05:45:25 DEBUG ] 05:45:25 DEBUG ] 05:45:25 DEBUG } 05:45:25 DEBUG } 05:45:25 DEBUG } 05:45:25 DEBUG } 05:45:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724325" 05:45:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724325" 05:45:25 INFO [http_cmd]: http status code OK 05:45:25 INFO [http_cmd]: http status code OK 05:45:25 DEBUG --- status code --- 05:45:25 DEBUG --- status code --- 05:45:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:25 DEBUG --- http response --- 05:45:25 DEBUG --- http response --- 05:45:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:45:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:45:25 DEBUG ----- output ----- 05:45:25 DEBUG ----- output ----- 05:45:25 DEBUG { 05:45:25 DEBUG { 05:45:25 DEBUG "status": "success", 05:45:25 DEBUG "status": "success", 05:45:25 DEBUG "data": { 05:45:25 DEBUG "data": { 05:45:25 DEBUG "resultType": "vector", 05:45:25 DEBUG "resultType": "vector", 05:45:25 DEBUG "result": [ 05:45:25 DEBUG "result": [ 05:45:25 DEBUG { 05:45:25 DEBUG { 05:45:25 DEBUG "metric": { 05:45:25 DEBUG "metric": { 05:45:25 DEBUG "instance": "10.142.0.25:9100" 05:45:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:45:25 DEBUG }, 05:45:25 DEBUG }, 05:45:25 DEBUG "value": [ 05:45:25 DEBUG "value": [ 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG [...] 05:45:25 DEBUG [...] 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG 05:45:25 DEBUG "instance": "10.142.0.77:9100" 05:45:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:45:25 DEBUG }, 05:45:25 DEBUG }, 05:45:25 DEBUG "value": [ 05:45:25 DEBUG "value": [ 05:45:25 DEBUG 1712724325, 05:45:25 DEBUG 1712724325, 05:45:25 DEBUG "8436296.933333334" 05:45:25 DEBUG "109129.5" 05:45:25 DEBUG ] 05:45:25 DEBUG ] 05:45:25 DEBUG } 05:45:25 DEBUG } 05:45:25 DEBUG ] 05:45:25 DEBUG ] 05:45:25 DEBUG } 05:45:25 DEBUG } 05:45:25 DEBUG } 05:45:25 DEBUG } 05:45:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724325" 05:45:26 INFO [http_cmd]: http status code OK 05:45:26 DEBUG --- status code --- 05:45:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:26 DEBUG --- http response --- 05:45:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:45:26 DEBUG ----- output ----- 05:45:26 DEBUG { 05:45:26 DEBUG "status": "success", 05:45:26 DEBUG "data": { 05:45:26 DEBUG "resultType": "vector", 05:45:26 DEBUG "result": [ 05:45:26 DEBUG { 05:45:26 DEBUG "metric": { 05:45:26 DEBUG "pod": "ds-idrepo-0" 05:45:26 DEBUG }, 05:45:26 DEBUG "value": [ 05:45:26 DEBUG 05:45:26 DEBUG 05:45:26 DEBUG 05:45:26 DEBUG [...] 05:45:26 DEBUG 05:45:26 DEBUG 05:45:26 DEBUG 05:45:26 DEBUG "pod": "ds-idrepo-1" 05:45:26 DEBUG }, 05:45:26 DEBUG "value": [ 05:45:26 DEBUG 1712724325, 05:45:26 DEBUG "0" 05:45:26 DEBUG ] 05:45:26 DEBUG } 05:45:26 DEBUG ] 05:45:26 DEBUG } 05:45:26 DEBUG } 05:45:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724326" 05:45:26 INFO [http_cmd]: http status code OK 05:45:26 DEBUG --- status code --- 05:45:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:26 DEBUG --- http response --- 05:45:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:45:26 DEBUG ----- output ----- 05:45:26 DEBUG { 05:45:26 DEBUG "status": "success", 05:45:26 DEBUG "data": { 05:45:26 DEBUG "resultType": "vector", 05:45:26 DEBUG "result": [ 05:45:26 DEBUG { 05:45:26 DEBUG "metric": { 05:45:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:45:26 DEBUG }, 05:45:26 DEBUG "value": [ 05:45:26 DEBUG 05:45:26 DEBUG 05:45:26 DEBUG 05:45:26 DEBUG [...] 05:45:26 DEBUG 05:45:26 DEBUG 05:45:26 DEBUG 05:45:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:45:26 DEBUG }, 05:45:26 DEBUG "value": [ 05:45:26 DEBUG 1712724326, 05:45:26 DEBUG "0" 05:45:26 DEBUG ] 05:45:26 DEBUG } 05:45:26 DEBUG ] 05:45:26 DEBUG } 05:45:26 DEBUG } 05:45:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724326" 05:45:26 INFO [http_cmd]: http status code OK 05:45:26 DEBUG --- status code --- 05:45:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:26 DEBUG --- http response --- 05:45:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:45:26 DEBUG ----- output ----- 05:45:26 DEBUG { 05:45:26 DEBUG "status": "success", 05:45:26 DEBUG "data": { 05:45:26 DEBUG "resultType": "vector", 05:45:26 DEBUG "result": [ 05:45:26 DEBUG { 05:45:26 DEBUG "metric": { 05:45:26 DEBUG "instance": "10.142.0.25:9100" 05:45:26 DEBUG }, 05:45:26 DEBUG "value": [ 05:45:26 DEBUG 05:45:26 DEBUG 05:45:26 DEBUG 05:45:26 DEBUG [...] 05:45:26 DEBUG 05:45:26 DEBUG 05:45:26 DEBUG 05:45:26 DEBUG "instance": "10.142.0.77:9100" 05:45:26 DEBUG }, 05:45:26 DEBUG "value": [ 05:45:26 DEBUG 1712724326, 05:45:26 DEBUG "0.12416666666666837" 05:45:26 DEBUG ] 05:45:26 DEBUG } 05:45:26 DEBUG ] 05:45:26 DEBUG } 05:45:26 DEBUG } 05:45:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724327" 05:45:27 INFO [http_cmd]: http status code OK 05:45:27 DEBUG --- status code --- 05:45:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:27 DEBUG --- http response --- 05:45:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:45:27 DEBUG ----- output ----- 05:45:27 DEBUG { 05:45:27 DEBUG "status": "success", 05:45:27 DEBUG "data": { 05:45:27 DEBUG "resultType": "vector", 05:45:27 DEBUG "result": [ 05:45:27 DEBUG { 05:45:27 DEBUG "metric": { 05:45:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:45:27 DEBUG }, 05:45:27 DEBUG "value": [ 05:45:27 DEBUG 05:45:27 DEBUG 05:45:27 DEBUG 05:45:27 DEBUG [...] 05:45:27 DEBUG 05:45:27 DEBUG 05:45:27 DEBUG 05:45:27 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:45:27 DEBUG }, 05:45:27 DEBUG "value": [ 05:45:27 DEBUG 1712724327, 05:45:27 DEBUG "0" 05:45:27 DEBUG ] 05:45:27 DEBUG } 05:45:27 DEBUG ] 05:45:27 DEBUG } 05:45:27 DEBUG } 05:45:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712724327" 05:45:27 INFO [http_cmd]: http status code OK 05:45:27 DEBUG --- status code --- 05:45:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:27 DEBUG --- http response --- 05:45:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:45:27 DEBUG ----- output ----- 05:45:27 DEBUG { 05:45:27 DEBUG "status": "success", 05:45:27 DEBUG "data": { 05:45:27 DEBUG "resultType": "vector", 05:45:27 DEBUG "result": [ 05:45:27 DEBUG { 05:45:27 DEBUG "metric": { 05:45:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:45:27 DEBUG }, 05:45:27 DEBUG "value": [ 05:45:27 DEBUG 05:45:27 DEBUG 05:45:27 DEBUG 05:45:27 DEBUG [...] 05:45:27 DEBUG 05:45:27 DEBUG 05:45:27 DEBUG 05:45:27 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:45:27 DEBUG }, 05:45:27 DEBUG "value": [ 05:45:27 DEBUG 1712724327, 05:45:27 DEBUG "0.02491006804062299" 05:45:27 DEBUG ] 05:45:27 DEBUG } 05:45:27 DEBUG ] 05:45:27 DEBUG } 05:45:27 DEBUG } 05:45:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712724327" 05:45:27 INFO [http_cmd]: http status code OK 05:45:27 DEBUG --- status code --- 05:45:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:27 DEBUG --- http response --- 05:45:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:45:27 DEBUG ----- output ----- 05:45:27 DEBUG { 05:45:27 DEBUG "status": "success", 05:45:27 DEBUG "data": { 05:45:27 DEBUG "resultType": "vector", 05:45:27 DEBUG "result": [ 05:45:27 DEBUG { 05:45:27 DEBUG "metric": { 05:45:27 DEBUG "pod": "ds-idrepo-0" 05:45:27 DEBUG }, 05:45:27 DEBUG "value": [ 05:45:27 DEBUG 05:45:27 DEBUG 05:45:27 DEBUG 05:45:27 DEBUG [...] 05:45:27 DEBUG 05:45:27 DEBUG 05:45:27 DEBUG 05:45:27 DEBUG "pod": "ds-cts-2" 05:45:27 DEBUG }, 05:45:27 DEBUG "value": [ 05:45:27 DEBUG 1712724327, 05:45:27 DEBUG "0" 05:45:27 DEBUG ] 05:45:27 DEBUG } 05:45:27 DEBUG ] 05:45:27 DEBUG } 05:45:27 DEBUG } 05:45:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724327" 05:45:27 INFO [http_cmd]: http status code OK 05:45:27 DEBUG --- status code --- 05:45:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:27 DEBUG --- http response --- 05:45:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:45:27 DEBUG ----- output ----- 05:45:27 DEBUG { 05:45:27 DEBUG "status": "success", 05:45:27 DEBUG "data": { 05:45:27 DEBUG "resultType": "vector", 05:45:27 DEBUG "result": [ 05:45:27 DEBUG { 05:45:27 DEBUG "metric": { 05:45:27 DEBUG "pod": "ds-idrepo-2" 05:45:27 DEBUG }, 05:45:27 DEBUG "value": [ 05:45:27 DEBUG 05:45:27 DEBUG 05:45:27 DEBUG 05:45:27 DEBUG [...] 05:45:27 DEBUG 05:45:27 DEBUG 05:45:27 DEBUG 05:45:27 DEBUG "pod": "ds-cts-2" 05:45:27 DEBUG }, 05:45:27 DEBUG "value": [ 05:45:27 DEBUG 1712724327, 05:45:27 DEBUG "0" 05:45:27 DEBUG ] 05:45:27 DEBUG } 05:45:27 DEBUG ] 05:45:27 DEBUG } 05:45:27 DEBUG } 05:45:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724327" 05:45:27 INFO [http_cmd]: http status code OK 05:45:27 DEBUG --- status code --- 05:45:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:27 DEBUG --- http response --- 05:45:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:45:27 DEBUG ----- output ----- 05:45:27 DEBUG { 05:45:27 DEBUG "status": "success", 05:45:27 DEBUG "data": { 05:45:27 DEBUG "resultType": "vector", 05:45:27 DEBUG "result": [ 05:45:27 DEBUG { 05:45:27 DEBUG "metric": { 05:45:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:45:27 DEBUG }, 05:45:27 DEBUG "value": [ 05:45:27 DEBUG 05:45:27 DEBUG 05:45:27 DEBUG 05:45:27 DEBUG [...] 05:45:27 DEBUG 05:45:27 DEBUG 05:45:27 DEBUG 05:45:27 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:45:27 DEBUG }, 05:45:27 DEBUG "value": [ 05:45:27 DEBUG 1712724327, 05:45:27 DEBUG "0" 05:45:27 DEBUG ] 05:45:27 DEBUG } 05:45:27 DEBUG ] 05:45:27 DEBUG } 05:45:27 DEBUG } 05:45:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724327" 05:45:27 INFO [http_cmd]: http status code OK 05:45:27 DEBUG --- status code --- 05:45:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:27 DEBUG --- http response --- 05:45:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:45:27 DEBUG ----- output ----- 05:45:27 DEBUG { 05:45:27 DEBUG "status": "success", 05:45:27 DEBUG "data": { 05:45:27 DEBUG "resultType": "vector", 05:45:27 DEBUG "result": [ 05:45:27 DEBUG { 05:45:27 DEBUG "metric": { 05:45:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:45:27 DEBUG }, 05:45:27 DEBUG "value": [ 05:45:27 DEBUG 05:45:27 DEBUG 05:45:27 DEBUG 05:45:27 DEBUG [...] 05:45:27 DEBUG 05:45:27 DEBUG 05:45:27 DEBUG 05:45:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:45:27 DEBUG }, 05:45:27 DEBUG "value": [ 05:45:27 DEBUG 1712724327, 05:45:27 DEBUG "0" 05:45:27 DEBUG ] 05:45:27 DEBUG } 05:45:27 DEBUG ] 05:45:27 DEBUG } 05:45:27 DEBUG } 05:45:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712724327" 05:45:27 INFO [http_cmd]: http status code OK 05:45:27 DEBUG --- status code --- 05:45:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:27 DEBUG --- http response --- 05:45:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:45:27 DEBUG ----- output ----- 05:45:27 DEBUG { 05:45:27 DEBUG "status": "success", 05:45:27 DEBUG "data": { 05:45:27 DEBUG "resultType": "vector", 05:45:27 DEBUG "result": [ 05:45:27 DEBUG { 05:45:27 DEBUG "metric": { 05:45:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:45:27 DEBUG }, 05:45:27 DEBUG "value": [ 05:45:27 DEBUG 05:45:27 DEBUG 05:45:27 DEBUG 05:45:27 DEBUG [...] 05:45:27 DEBUG 05:45:27 DEBUG 05:45:27 DEBUG 05:45:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:45:27 DEBUG }, 05:45:27 DEBUG "value": [ 05:45:27 DEBUG 1712724327, 05:45:27 DEBUG "0.043" 05:45:27 DEBUG ] 05:45:27 DEBUG } 05:45:27 DEBUG ] 05:45:27 DEBUG } 05:45:27 DEBUG } 05:45:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724328" 05:45:28 INFO [http_cmd]: http status code OK 05:45:28 DEBUG --- status code --- 05:45:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:28 DEBUG --- http response --- 05:45:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:45:28 DEBUG ----- output ----- 05:45:28 DEBUG { 05:45:28 DEBUG "status": "success", 05:45:28 DEBUG "data": { 05:45:28 DEBUG "resultType": "vector", 05:45:28 DEBUG "result": [ 05:45:28 DEBUG { 05:45:28 DEBUG "metric": { 05:45:28 DEBUG "pod": "ds-idrepo-0" 05:45:28 DEBUG }, 05:45:28 DEBUG "value": [ 05:45:28 DEBUG 05:45:28 DEBUG 05:45:28 DEBUG 05:45:28 DEBUG [...] 05:45:28 DEBUG 05:45:28 DEBUG 05:45:28 DEBUG 05:45:28 DEBUG "pod": "ds-idrepo-1" 05:45:28 DEBUG }, 05:45:28 DEBUG "value": [ 05:45:28 DEBUG 1712724328, 05:45:28 DEBUG "0" 05:45:28 DEBUG ] 05:45:28 DEBUG } 05:45:28 DEBUG ] 05:45:28 DEBUG } 05:45:28 DEBUG } 05:45:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724328" 05:45:28 INFO [http_cmd]: http status code OK 05:45:28 DEBUG --- status code --- 05:45:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:28 DEBUG --- http response --- 05:45:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:45:28 DEBUG ----- output ----- 05:45:28 DEBUG { 05:45:28 DEBUG "status": "success", 05:45:28 DEBUG "data": { 05:45:28 DEBUG "resultType": "vector", 05:45:28 DEBUG "result": [ 05:45:28 DEBUG { 05:45:28 DEBUG "metric": { 05:45:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:45:28 DEBUG }, 05:45:28 DEBUG "value": [ 05:45:28 DEBUG 05:45:28 DEBUG 05:45:28 DEBUG 05:45:28 DEBUG [...] 05:45:28 DEBUG 05:45:28 DEBUG 05:45:28 DEBUG 05:45:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:45:28 DEBUG }, 05:45:28 DEBUG "value": [ 05:45:28 DEBUG 1712724328, 05:45:28 DEBUG "1255.3666666666666" 05:45:28 DEBUG ] 05:45:28 DEBUG } 05:45:28 DEBUG ] 05:45:28 DEBUG } 05:45:28 DEBUG } 05:45:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724328" 05:45:28 INFO [http_cmd]: http status code OK 05:45:28 DEBUG --- status code --- 05:45:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:28 DEBUG --- http response --- 05:45:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:45:28 DEBUG ----- output ----- 05:45:28 DEBUG { 05:45:28 DEBUG "status": "success", 05:45:28 DEBUG "data": { 05:45:28 DEBUG "resultType": "vector", 05:45:28 DEBUG "result": [ 05:45:28 DEBUG { 05:45:28 DEBUG "metric": { 05:45:28 DEBUG "pod": "ds-idrepo-0" 05:45:28 DEBUG }, 05:45:28 DEBUG "value": [ 05:45:28 DEBUG 05:45:28 DEBUG 05:45:28 DEBUG 05:45:28 DEBUG [...] 05:45:28 DEBUG 05:45:28 DEBUG 05:45:28 DEBUG 05:45:28 DEBUG "pod": "ds-idrepo-1" 05:45:28 DEBUG }, 05:45:28 DEBUG "value": [ 05:45:28 DEBUG 1712724328, 05:45:28 DEBUG "0" 05:45:28 DEBUG ] 05:45:28 DEBUG } 05:45:28 DEBUG ] 05:45:28 DEBUG } 05:45:28 DEBUG } 05:45:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712724328" 05:45:28 INFO [http_cmd]: http status code OK 05:45:28 DEBUG --- status code --- 05:45:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:28 DEBUG --- http response --- 05:45:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:45:28 DEBUG ----- output ----- 05:45:28 DEBUG { 05:45:28 DEBUG "status": "success", 05:45:28 DEBUG "data": { 05:45:28 DEBUG "resultType": "vector", 05:45:28 DEBUG "result": [ 05:45:28 DEBUG { 05:45:28 DEBUG "metric": { 05:45:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:45:28 DEBUG }, 05:45:28 DEBUG "value": [ 05:45:28 DEBUG 05:45:28 DEBUG 05:45:28 DEBUG 05:45:28 DEBUG [...] 05:45:28 DEBUG 05:45:28 DEBUG 05:45:28 DEBUG 05:45:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:45:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724328" 05:45:28 DEBUG }, 05:45:28 DEBUG "value": [ 05:45:28 DEBUG 1712724328, 05:45:28 DEBUG "2491400192" 05:45:28 DEBUG ] 05:45:28 DEBUG } 05:45:28 DEBUG ] 05:45:28 DEBUG } 05:45:28 DEBUG } 05:45:28 INFO [http_cmd]: http status code OK 05:45:28 DEBUG --- status code --- 05:45:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:28 DEBUG --- http response --- 05:45:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:45:28 DEBUG ----- output ----- 05:45:28 DEBUG { 05:45:28 DEBUG "status": "success", 05:45:28 DEBUG "data": { 05:45:28 DEBUG "resultType": "vector", 05:45:28 DEBUG "result": [ 05:45:28 DEBUG { 05:45:28 DEBUG "metric": { 05:45:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:45:28 DEBUG }, 05:45:28 DEBUG "value": [ 05:45:28 DEBUG 05:45:28 DEBUG 05:45:28 DEBUG 05:45:28 DEBUG [...] 05:45:28 DEBUG 05:45:28 DEBUG 05:45:28 DEBUG 05:45:28 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 05:45:28 DEBUG }, 05:45:28 DEBUG "value": [ 05:45:28 DEBUG 1712724328, 05:45:28 DEBUG "0" 05:45:28 DEBUG ] 05:45:28 DEBUG } 05:45:28 DEBUG ] 05:45:28 DEBUG } 05:45:28 DEBUG } 05:45:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724328" 05:45:28 INFO [http_cmd]: http status code OK 05:45:28 DEBUG --- status code --- 05:45:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:28 DEBUG --- http response --- 05:45:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:45:28 DEBUG ----- output ----- 05:45:28 DEBUG { 05:45:28 DEBUG "status": "success", 05:45:28 DEBUG "data": { 05:45:28 DEBUG "resultType": "vector", 05:45:28 DEBUG "result": [ 05:45:28 DEBUG { 05:45:28 DEBUG "metric": { 05:45:28 DEBUG "instance": "10.142.0.25:9100" 05:45:28 DEBUG }, 05:45:28 DEBUG "value": [ 05:45:28 DEBUG 05:45:28 DEBUG 05:45:28 DEBUG 05:45:28 DEBUG [...] 05:45:28 DEBUG 05:45:28 DEBUG 05:45:28 DEBUG 05:45:28 DEBUG "instance": "10.142.0.77:9100" 05:45:28 DEBUG }, 05:45:28 DEBUG "value": [ 05:45:28 DEBUG 1712724328, 05:45:28 DEBUG "8437103.833333334" 05:45:28 DEBUG ] 05:45:28 DEBUG } 05:45:28 DEBUG ] 05:45:28 DEBUG } 05:45:28 DEBUG } 05:45:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724328" 05:45:28 INFO [http_cmd]: http status code OK 05:45:28 DEBUG --- status code --- 05:45:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:28 DEBUG --- http response --- 05:45:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:45:28 DEBUG ----- output ----- 05:45:28 DEBUG { 05:45:28 DEBUG "status": "success", 05:45:28 DEBUG "data": { 05:45:28 DEBUG "resultType": "vector", 05:45:28 DEBUG "result": [ 05:45:28 DEBUG { 05:45:28 DEBUG "metric": { 05:45:28 DEBUG "instance": "10.142.0.25:9100" 05:45:28 DEBUG }, 05:45:28 DEBUG "value": [ 05:45:28 DEBUG 05:45:28 DEBUG 05:45:28 DEBUG 05:45:28 DEBUG [...] 05:45:28 DEBUG 05:45:28 DEBUG 05:45:28 DEBUG 05:45:28 DEBUG "instance": "10.142.0.77:9100" 05:45:28 DEBUG }, 05:45:28 DEBUG "value": [ 05:45:28 DEBUG 1712724328, 05:45:28 DEBUG "0.02933333333333301" 05:45:28 DEBUG ] 05:45:28 DEBUG } 05:45:28 DEBUG ] 05:45:28 DEBUG } 05:45:28 DEBUG } 05:45:44 INFO 05:45:44 INFO [loop_until]: kubectl --namespace=xlou top pods 05:45:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:45:44 INFO [loop_until]: OK (rc = 0) 05:45:44 DEBUG --- stdout --- 05:45:44 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 1458m 8753Mi am-654c4fcbcf-9scrj 1446m 8383Mi am-654c4fcbcf-v5qb5 1219m 8375Mi ds-cts-0 3227m 12791Mi ds-cts-1 3344m 12969Mi ds-cts-2 3089m 12055Mi ds-idrepo-0 1168m 5622Mi ds-idrepo-1 11m 5671Mi ds-idrepo-2 12m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 7m 2213Mi lodemon-b776b7d88-thrqf 10m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 481m 1528Mi 05:45:44 DEBUG --- stderr --- 05:45:44 DEBUG 05:45:51 INFO 05:45:51 INFO [loop_until]: kubectl --namespace=xlou top node 05:45:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:45:51 INFO [loop_until]: OK (rc = 0) 05:45:51 DEBUG --- stdout --- 05:45:51 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 97m 0% 2307Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 1397m 8% 9263Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 83m 0% 3261Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 214m 1% 6510Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 1660m 10% 9283Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 1584m 9% 9607Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 81m 0% 851Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 1156m 7% 6287Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 3344m 21% 13445Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 3226m 20% 12707Mi 21% gke-xlou-cdm-ds-32e4dcb1-gxzj 65m 0% 6372Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 3408m 21% 13612Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 67m 0% 6337Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 665m 4% 5212Mi 8% 05:45:51 DEBUG --- stderr --- 05:45:51 DEBUG 05:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712724385" 05:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724385" 05:46:25 ERROR [http_cmd]: ERROR 05:46:25 INFO [http_cmd]: http status code OK 05:46:25 DEBUG --- status code --- 05:46:25 ERROR --- status code --- 05:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:46:25 DEBUG --- http response --- 05:46:25 ERROR --- http response --- 05:46:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:46:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:46:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724385" 05:46:25 INFO [http_cmd]: http status code OK 05:46:25 DEBUG --- status code --- 05:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:25 DEBUG --- http response --- 05:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:46:25 DEBUG ----- output ----- 05:46:25 DEBUG { 05:46:25 DEBUG "status": "success", 05:46:25 DEBUG "data": { 05:46:25 DEBUG "resultType": "vector", 05:46:25 DEBUG "result": [ 05:46:25 DEBUG { 05:46:25 DEBUG "metric": { 05:46:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:46:25 DEBUG }, 05:46:25 DEBUG "value": [ 05:46:25 DEBUG 05:46:25 DEBUG 05:46:25 DEBUG 05:46:25 DEBUG [...] 05:46:25 DEBUG 05:46:25 DEBUG 05:46:25 DEBUG 05:46:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:46:25 DEBUG }, 05:46:25 DEBUG "value": [ 05:46:25 DEBUG 1712724385, 05:46:25 DEBUG "0" 05:46:25 DEBUG ] 05:46:25 DEBUG } 05:46:25 DEBUG ] 05:46:25 DEBUG } 05:46:25 DEBUG } 05:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724385" 05:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724385" 05:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724385" 05:46:25 INFO [http_cmd]: http status code OK 05:46:25 DEBUG --- status code --- 05:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:25 DEBUG --- http response --- 05:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:46:25 DEBUG ----- output ----- 05:46:25 DEBUG { 05:46:25 INFO [http_cmd]: http status code OK 05:46:25 INFO [http_cmd]: http status code OK 05:46:25 DEBUG "status": "success", 05:46:25 DEBUG --- status code --- 05:46:25 DEBUG --- status code --- 05:46:25 DEBUG "data": { 05:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:25 DEBUG "resultType": "vector", 05:46:25 DEBUG --- http response --- 05:46:25 DEBUG --- http response --- 05:46:25 DEBUG "result": [ 05:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:46:25 DEBUG ----- output ----- 05:46:25 DEBUG ----- output ----- 05:46:25 DEBUG { 05:46:25 DEBUG { 05:46:25 DEBUG "status": "success", 05:46:25 DEBUG { 05:46:25 DEBUG "status": "success", 05:46:25 DEBUG "data": { 05:46:25 DEBUG "metric": { 05:46:25 DEBUG "pod": "ds-idrepo-0" 05:46:25 DEBUG "resultType": "vector", 05:46:25 DEBUG }, 05:46:25 DEBUG "value": [ 05:46:25 DEBUG "result": [ 05:46:25 DEBUG 05:46:25 DEBUG 05:46:25 DEBUG { 05:46:25 DEBUG 05:46:25 DEBUG [...] 05:46:25 DEBUG "metric": { 05:46:25 DEBUG "pod": "ds-idrepo-0" 05:46:25 DEBUG "data": { 05:46:25 DEBUG "resultType": "vector", 05:46:25 DEBUG }, 05:46:25 DEBUG "result": [ 05:46:25 DEBUG { 05:46:25 DEBUG "metric": { 05:46:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:46:25 DEBUG "value": [ 05:46:25 DEBUG 05:46:25 DEBUG 05:46:25 DEBUG 05:46:25 DEBUG [...] 05:46:25 DEBUG 05:46:25 DEBUG 05:46:25 DEBUG 05:46:25 DEBUG "pod": "ds-cts-2" 05:46:25 DEBUG }, 05:46:25 DEBUG "value": [ 05:46:25 DEBUG 1712724385, 05:46:25 DEBUG "30" 05:46:25 DEBUG ] 05:46:25 DEBUG } 05:46:25 DEBUG ] 05:46:25 DEBUG } 05:46:25 DEBUG } 05:46:25 DEBUG 05:46:25 DEBUG 05:46:25 DEBUG 05:46:25 DEBUG "pod": "ds-cts-2" 05:46:25 DEBUG }, 05:46:25 DEBUG }, 05:46:25 DEBUG "value": [ 05:46:25 DEBUG 05:46:25 DEBUG "value": [ 05:46:25 DEBUG 05:46:25 DEBUG 1712724385, 05:46:25 DEBUG 05:46:25 DEBUG "0" 05:46:25 DEBUG [...] 05:46:25 DEBUG 05:46:25 DEBUG ] 05:46:25 DEBUG } 05:46:25 DEBUG 05:46:25 DEBUG 05:46:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:46:25 DEBUG ] 05:46:25 DEBUG } 05:46:25 DEBUG } 05:46:25 DEBUG }, 05:46:25 DEBUG "value": [ 05:46:25 DEBUG 1712724385, 05:46:25 DEBUG "830.4333333333333" 05:46:25 DEBUG ] 05:46:25 DEBUG } 05:46:25 DEBUG ] 05:46:25 DEBUG } 05:46:25 DEBUG } 05:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724385" 05:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724385" 05:46:25 INFO [http_cmd]: http status code OK 05:46:25 INFO [http_cmd]: http status code OK 05:46:25 DEBUG --- status code --- 05:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:25 DEBUG --- http response --- 05:46:25 DEBUG --- status code --- 05:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:25 DEBUG ----- output ----- 05:46:25 DEBUG --- http response --- 05:46:25 DEBUG { 05:46:25 DEBUG "status": "success", 05:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:46:25 DEBUG "data": { 05:46:25 DEBUG ----- output ----- 05:46:25 DEBUG "resultType": "vector", 05:46:25 DEBUG { 05:46:25 DEBUG "result": [ 05:46:25 DEBUG "status": "success", 05:46:25 DEBUG { 05:46:25 DEBUG "data": { 05:46:25 DEBUG "metric": { 05:46:25 DEBUG "resultType": "vector", 05:46:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:46:25 DEBUG "result": [ 05:46:25 DEBUG { 05:46:25 DEBUG "metric": { 05:46:25 DEBUG "instance": "10.142.0.25:9100" 05:46:25 DEBUG }, 05:46:25 DEBUG "value": [ 05:46:25 DEBUG 05:46:25 DEBUG 05:46:25 DEBUG 05:46:25 DEBUG [...] 05:46:25 DEBUG 05:46:25 DEBUG 05:46:25 DEBUG 05:46:25 DEBUG "instance": "10.142.0.77:9100" 05:46:25 DEBUG }, 05:46:25 DEBUG "value": [ 05:46:25 DEBUG 1712724385, 05:46:25 DEBUG "314383842304" 05:46:25 DEBUG ] 05:46:25 DEBUG } 05:46:25 DEBUG ] 05:46:25 DEBUG } 05:46:25 DEBUG } 05:46:25 DEBUG }, 05:46:25 DEBUG "value": [ 05:46:25 DEBUG 05:46:25 DEBUG 05:46:25 DEBUG 05:46:25 DEBUG [...] 05:46:25 DEBUG 05:46:25 DEBUG 05:46:25 DEBUG 05:46:25 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:46:25 DEBUG }, 05:46:25 DEBUG "value": [ 05:46:25 DEBUG 1712724385, 05:46:25 DEBUG "0" 05:46:25 DEBUG ] 05:46:25 DEBUG } 05:46:25 DEBUG ] 05:46:25 DEBUG } 05:46:25 DEBUG } 05:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724385" 05:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724385" 05:46:25 INFO [http_cmd]: http status code OK 05:46:25 DEBUG --- status code --- 05:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:25 INFO [http_cmd]: http status code OK 05:46:25 DEBUG --- http response --- 05:46:25 DEBUG --- status code --- 05:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:25 DEBUG ----- output ----- 05:46:25 DEBUG --- http response --- 05:46:25 DEBUG { 05:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:46:25 DEBUG "status": "success", 05:46:25 DEBUG ----- output ----- 05:46:25 DEBUG "data": { 05:46:25 DEBUG { 05:46:25 DEBUG "status": "success", 05:46:25 DEBUG "resultType": "vector", 05:46:25 DEBUG "data": { 05:46:25 DEBUG "result": [ 05:46:25 DEBUG "resultType": "vector", 05:46:25 DEBUG { 05:46:25 DEBUG "result": [ 05:46:25 DEBUG "metric": { 05:46:25 DEBUG { 05:46:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:46:25 DEBUG "metric": { 05:46:25 DEBUG }, 05:46:25 DEBUG "instance": "10.142.0.25:9100" 05:46:25 DEBUG "value": [ 05:46:25 DEBUG }, 05:46:25 DEBUG 05:46:25 DEBUG "value": [ 05:46:25 DEBUG 05:46:25 DEBUG 05:46:25 DEBUG 05:46:25 DEBUG 05:46:25 DEBUG [...] 05:46:25 DEBUG 05:46:25 DEBUG 05:46:25 DEBUG [...] 05:46:25 DEBUG 05:46:25 DEBUG 05:46:25 DEBUG 05:46:25 DEBUG 05:46:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:46:25 DEBUG 05:46:25 DEBUG }, 05:46:25 DEBUG "instance": "10.142.0.77:9100" 05:46:25 DEBUG "value": [ 05:46:25 DEBUG }, 05:46:25 DEBUG 1712724385, 05:46:25 DEBUG "value": [ 05:46:25 DEBUG "46292" 05:46:25 DEBUG 1712724385, 05:46:25 DEBUG ] 05:46:25 DEBUG "31866195.733333334" 05:46:25 DEBUG } 05:46:25 DEBUG ] 05:46:25 DEBUG ] 05:46:25 DEBUG } 05:46:25 DEBUG } 05:46:25 DEBUG } 05:46:25 DEBUG ] 05:46:25 DEBUG } 05:46:25 DEBUG } 05:46:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724386" 05:46:26 INFO [http_cmd]: http status code OK 05:46:26 DEBUG --- status code --- 05:46:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:26 DEBUG --- http response --- 05:46:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:46:26 DEBUG ----- output ----- 05:46:26 DEBUG { 05:46:26 DEBUG "status": "success", 05:46:26 DEBUG "data": { 05:46:26 DEBUG "resultType": "vector", 05:46:26 DEBUG "result": [ 05:46:26 DEBUG { 05:46:26 DEBUG "metric": { 05:46:26 DEBUG "pod": "ds-idrepo-0" 05:46:26 DEBUG }, 05:46:26 DEBUG "value": [ 05:46:26 DEBUG 05:46:26 DEBUG 05:46:26 DEBUG 05:46:26 DEBUG [...] 05:46:26 DEBUG 05:46:26 DEBUG 05:46:26 DEBUG 05:46:26 DEBUG "pod": "ds-idrepo-1" 05:46:26 DEBUG }, 05:46:26 DEBUG "value": [ 05:46:26 DEBUG 1712724386, 05:46:26 DEBUG "0" 05:46:26 DEBUG ] 05:46:26 DEBUG } 05:46:26 DEBUG ] 05:46:26 DEBUG } 05:46:26 DEBUG } 05:46:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724386" 05:46:26 INFO [http_cmd]: http status code OK 05:46:26 DEBUG --- status code --- 05:46:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:26 DEBUG --- http response --- 05:46:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:46:26 DEBUG ----- output ----- 05:46:26 DEBUG { 05:46:26 DEBUG "status": "success", 05:46:26 DEBUG "data": { 05:46:26 DEBUG "resultType": "vector", 05:46:26 DEBUG "result": [ 05:46:26 DEBUG { 05:46:26 DEBUG "metric": { 05:46:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:46:26 DEBUG }, 05:46:26 DEBUG "value": [ 05:46:26 DEBUG 05:46:26 DEBUG 05:46:26 DEBUG 05:46:26 DEBUG [...] 05:46:26 DEBUG 05:46:26 DEBUG 05:46:26 DEBUG 05:46:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:46:26 DEBUG }, 05:46:26 DEBUG "value": [ 05:46:26 DEBUG 1712724386, 05:46:26 DEBUG "820.1333333333333" 05:46:26 DEBUG ] 05:46:26 DEBUG } 05:46:26 DEBUG ] 05:46:26 DEBUG } 05:46:26 DEBUG } 05:46:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724386" 05:46:26 INFO [http_cmd]: http status code OK 05:46:26 DEBUG --- status code --- 05:46:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:26 DEBUG --- http response --- 05:46:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:46:26 DEBUG ----- output ----- 05:46:26 DEBUG { 05:46:26 DEBUG "status": "success", 05:46:26 DEBUG "data": { 05:46:26 DEBUG "resultType": "vector", 05:46:26 DEBUG "result": [ 05:46:26 DEBUG { 05:46:26 DEBUG "metric": { 05:46:26 DEBUG "instance": "10.142.0.25:9100" 05:46:26 DEBUG }, 05:46:26 DEBUG "value": [ 05:46:26 DEBUG 05:46:26 DEBUG 05:46:26 DEBUG 05:46:26 DEBUG [...] 05:46:26 DEBUG 05:46:26 DEBUG 05:46:26 DEBUG 05:46:26 DEBUG "instance": "10.142.0.77:9100" 05:46:26 DEBUG }, 05:46:26 DEBUG "value": [ 05:46:26 DEBUG 1712724386, 05:46:26 DEBUG "0.13296666666666007" 05:46:26 DEBUG ] 05:46:26 DEBUG } 05:46:26 DEBUG ] 05:46:26 DEBUG } 05:46:26 DEBUG } 05:46:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724387" 05:46:27 INFO [http_cmd]: http status code OK 05:46:27 DEBUG --- status code --- 05:46:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:27 DEBUG --- http response --- 05:46:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:46:27 DEBUG ----- output ----- 05:46:27 DEBUG { 05:46:27 DEBUG "status": "success", 05:46:27 DEBUG "data": { 05:46:27 DEBUG "resultType": "vector", 05:46:27 DEBUG "result": [ 05:46:27 DEBUG { 05:46:27 DEBUG "metric": { 05:46:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:46:27 DEBUG }, 05:46:27 DEBUG "value": [ 05:46:27 DEBUG 05:46:27 DEBUG 05:46:27 DEBUG 05:46:27 DEBUG [...] 05:46:27 DEBUG 05:46:27 DEBUG 05:46:27 DEBUG 05:46:27 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:46:27 DEBUG }, 05:46:27 DEBUG "value": [ 05:46:27 DEBUG 1712724387, 05:46:27 DEBUG "6164348.431708044" 05:46:27 DEBUG ] 05:46:27 DEBUG } 05:46:27 DEBUG ] 05:46:27 DEBUG } 05:46:27 DEBUG } 05:46:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712724387" 05:46:27 INFO [http_cmd]: http status code OK 05:46:27 DEBUG --- status code --- 05:46:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:27 DEBUG --- http response --- 05:46:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:46:27 DEBUG ----- output ----- 05:46:27 DEBUG { 05:46:27 DEBUG "status": "success", 05:46:27 DEBUG "data": { 05:46:27 DEBUG "resultType": "vector", 05:46:27 DEBUG "result": [ 05:46:27 DEBUG { 05:46:27 DEBUG "metric": { 05:46:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:46:27 DEBUG }, 05:46:27 DEBUG "value": [ 05:46:27 DEBUG 05:46:27 DEBUG 05:46:27 DEBUG 05:46:27 DEBUG [...] 05:46:27 DEBUG 05:46:27 DEBUG 05:46:27 DEBUG 05:46:27 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:46:27 DEBUG }, 05:46:27 DEBUG "value": [ 05:46:27 DEBUG 1712724387, 05:46:27 DEBUG "1.8313413195375903" 05:46:27 DEBUG ] 05:46:27 DEBUG } 05:46:27 DEBUG ] 05:46:27 DEBUG } 05:46:27 DEBUG } 05:46:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712724387" 05:46:27 INFO [http_cmd]: http status code OK 05:46:27 DEBUG --- status code --- 05:46:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:27 DEBUG --- http response --- 05:46:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:46:27 DEBUG ----- output ----- 05:46:27 DEBUG { 05:46:27 DEBUG "status": "success", 05:46:27 DEBUG "data": { 05:46:27 DEBUG "resultType": "vector", 05:46:27 DEBUG "result": [ 05:46:27 DEBUG { 05:46:27 DEBUG "metric": { 05:46:27 DEBUG "pod": "ds-idrepo-0" 05:46:27 DEBUG }, 05:46:27 DEBUG "value": [ 05:46:27 DEBUG 05:46:27 DEBUG 05:46:27 DEBUG 05:46:27 DEBUG [...] 05:46:27 DEBUG 05:46:27 DEBUG 05:46:27 DEBUG 05:46:27 DEBUG "pod": "ds-cts-2" 05:46:27 DEBUG }, 05:46:27 DEBUG "value": [ 05:46:27 DEBUG 1712724387, 05:46:27 DEBUG "0" 05:46:27 DEBUG ] 05:46:27 DEBUG } 05:46:27 DEBUG ] 05:46:27 DEBUG } 05:46:27 DEBUG } 05:46:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724387" 05:46:27 INFO [http_cmd]: http status code OK 05:46:27 DEBUG --- status code --- 05:46:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:27 DEBUG --- http response --- 05:46:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:46:27 DEBUG ----- output ----- 05:46:27 DEBUG { 05:46:27 DEBUG "status": "success", 05:46:27 DEBUG "data": { 05:46:27 DEBUG "resultType": "vector", 05:46:27 DEBUG "result": [ 05:46:27 DEBUG { 05:46:27 DEBUG "metric": { 05:46:27 DEBUG "pod": "ds-idrepo-2" 05:46:27 DEBUG }, 05:46:27 DEBUG "value": [ 05:46:27 DEBUG 05:46:27 DEBUG 05:46:27 DEBUG 05:46:27 DEBUG [...] 05:46:27 DEBUG 05:46:27 DEBUG 05:46:27 DEBUG 05:46:27 DEBUG "pod": "ds-cts-2" 05:46:27 DEBUG }, 05:46:27 DEBUG "value": [ 05:46:27 DEBUG 1712724387, 05:46:27 DEBUG "0.000021218888888888892" 05:46:27 DEBUG ] 05:46:27 DEBUG } 05:46:27 DEBUG ] 05:46:27 DEBUG } 05:46:27 DEBUG } 05:46:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724387" 05:46:27 INFO [http_cmd]: http status code OK 05:46:27 DEBUG --- status code --- 05:46:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:27 DEBUG --- http response --- 05:46:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:46:27 DEBUG ----- output ----- 05:46:27 DEBUG { 05:46:27 DEBUG "status": "success", 05:46:27 DEBUG "data": { 05:46:27 DEBUG "resultType": "vector", 05:46:27 DEBUG "result": [ 05:46:27 DEBUG { 05:46:27 DEBUG "metric": { 05:46:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:46:27 DEBUG }, 05:46:27 DEBUG "value": [ 05:46:27 DEBUG 05:46:27 DEBUG 05:46:27 DEBUG 05:46:27 DEBUG [...] 05:46:27 DEBUG 05:46:27 DEBUG 05:46:27 DEBUG 05:46:27 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:46:27 DEBUG }, 05:46:27 DEBUG "value": [ 05:46:27 DEBUG 1712724387, 05:46:27 DEBUG "1.5413636515524978" 05:46:27 DEBUG ] 05:46:27 DEBUG } 05:46:27 DEBUG ] 05:46:27 DEBUG } 05:46:27 DEBUG } 05:46:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724387" 05:46:27 INFO [http_cmd]: http status code OK 05:46:27 DEBUG --- status code --- 05:46:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:27 DEBUG --- http response --- 05:46:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:46:27 DEBUG ----- output ----- 05:46:27 DEBUG { 05:46:27 DEBUG "status": "success", 05:46:27 DEBUG "data": { 05:46:27 DEBUG "resultType": "vector", 05:46:27 DEBUG "result": [ 05:46:27 DEBUG { 05:46:27 DEBUG "metric": { 05:46:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:46:27 DEBUG }, 05:46:27 DEBUG "value": [ 05:46:27 DEBUG 05:46:27 DEBUG 05:46:27 DEBUG 05:46:27 DEBUG [...] 05:46:27 DEBUG 05:46:27 DEBUG 05:46:27 DEBUG 05:46:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:46:27 DEBUG }, 05:46:27 DEBUG "value": [ 05:46:27 DEBUG 1712724387, 05:46:27 DEBUG "819.3333333333334" 05:46:27 DEBUG ] 05:46:27 DEBUG } 05:46:27 DEBUG ] 05:46:27 DEBUG } 05:46:27 DEBUG } 05:46:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712724388" 05:46:28 INFO [http_cmd]: http status code OK 05:46:28 DEBUG --- status code --- 05:46:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:28 DEBUG --- http response --- 05:46:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:46:28 DEBUG ----- output ----- 05:46:28 DEBUG { 05:46:28 DEBUG "status": "success", 05:46:28 DEBUG "data": { 05:46:28 DEBUG "resultType": "vector", 05:46:28 DEBUG "result": [ 05:46:28 DEBUG { 05:46:28 DEBUG "metric": { 05:46:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:46:28 DEBUG }, 05:46:28 DEBUG "value": [ 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG [...] 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:46:28 DEBUG }, 05:46:28 DEBUG "value": [ 05:46:28 DEBUG 1712724388, 05:46:28 DEBUG "0.043" 05:46:28 DEBUG ] 05:46:28 DEBUG } 05:46:28 DEBUG ] 05:46:28 DEBUG } 05:46:28 DEBUG } 05:46:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724388" 05:46:28 INFO [http_cmd]: http status code OK 05:46:28 DEBUG --- status code --- 05:46:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:28 DEBUG --- http response --- 05:46:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:46:28 DEBUG ----- output ----- 05:46:28 DEBUG { 05:46:28 DEBUG "status": "success", 05:46:28 DEBUG "data": { 05:46:28 DEBUG "resultType": "vector", 05:46:28 DEBUG "result": [ 05:46:28 DEBUG { 05:46:28 DEBUG "metric": { 05:46:28 DEBUG "pod": "ds-idrepo-0" 05:46:28 DEBUG }, 05:46:28 DEBUG "value": [ 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG [...] 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG "pod": "ds-idrepo-1" 05:46:28 DEBUG }, 05:46:28 DEBUG "value": [ 05:46:28 DEBUG 1712724388, 05:46:28 DEBUG "0" 05:46:28 DEBUG ] 05:46:28 DEBUG } 05:46:28 DEBUG ] 05:46:28 DEBUG } 05:46:28 DEBUG } 05:46:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724388" 05:46:28 INFO [http_cmd]: http status code OK 05:46:28 DEBUG --- status code --- 05:46:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:28 DEBUG --- http response --- 05:46:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:46:28 DEBUG ----- output ----- 05:46:28 DEBUG { 05:46:28 DEBUG "status": "success", 05:46:28 DEBUG "data": { 05:46:28 DEBUG "resultType": "vector", 05:46:28 DEBUG "result": [ 05:46:28 DEBUG { 05:46:28 DEBUG "metric": { 05:46:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:46:28 DEBUG }, 05:46:28 DEBUG "value": [ 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG [...] 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:46:28 DEBUG }, 05:46:28 DEBUG "value": [ 05:46:28 DEBUG 1712724388, 05:46:28 DEBUG "1250.1333333333332" 05:46:28 DEBUG ] 05:46:28 DEBUG } 05:46:28 DEBUG ] 05:46:28 DEBUG } 05:46:28 DEBUG } 05:46:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724388" 05:46:28 INFO [http_cmd]: http status code OK 05:46:28 DEBUG --- status code --- 05:46:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:28 DEBUG --- http response --- 05:46:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:46:28 DEBUG ----- output ----- 05:46:28 DEBUG { 05:46:28 DEBUG "status": "success", 05:46:28 DEBUG "data": { 05:46:28 DEBUG "resultType": "vector", 05:46:28 DEBUG "result": [ 05:46:28 DEBUG { 05:46:28 DEBUG "metric": { 05:46:28 DEBUG "pod": "ds-idrepo-0" 05:46:28 DEBUG }, 05:46:28 DEBUG "value": [ 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG [...] 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG "pod": "ds-idrepo-1" 05:46:28 DEBUG }, 05:46:28 DEBUG "value": [ 05:46:28 DEBUG 1712724388, 05:46:28 DEBUG "0" 05:46:28 DEBUG ] 05:46:28 DEBUG } 05:46:28 DEBUG ] 05:46:28 DEBUG } 05:46:28 DEBUG } 05:46:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712724388" 05:46:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724388" 05:46:28 INFO [http_cmd]: http status code OK 05:46:28 INFO [http_cmd]: http status code OK 05:46:28 DEBUG --- status code --- 05:46:28 DEBUG --- status code --- 05:46:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:28 DEBUG --- http response --- 05:46:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:46:28 DEBUG ----- output ----- 05:46:28 DEBUG --- http response --- 05:46:28 DEBUG { 05:46:28 DEBUG "status": "success", 05:46:28 DEBUG "data": { 05:46:28 DEBUG "resultType": "vector", 05:46:28 DEBUG "result": [ 05:46:28 DEBUG { 05:46:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:46:28 DEBUG "metric": { 05:46:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:46:28 DEBUG }, 05:46:28 DEBUG ----- output ----- 05:46:28 DEBUG "value": [ 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG [...] 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG { 05:46:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:46:28 DEBUG "status": "success", 05:46:28 DEBUG }, 05:46:28 DEBUG "value": [ 05:46:28 DEBUG 1712724388, 05:46:28 DEBUG "data": { 05:46:28 DEBUG "2488586240" 05:46:28 DEBUG "resultType": "vector", 05:46:28 DEBUG ] 05:46:28 DEBUG } 05:46:28 DEBUG "result": [ 05:46:28 DEBUG ] 05:46:28 DEBUG } 05:46:28 DEBUG } 05:46:28 DEBUG { 05:46:28 DEBUG "metric": { 05:46:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:46:28 DEBUG }, 05:46:28 DEBUG "value": [ 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG [...] 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:46:28 DEBUG }, 05:46:28 DEBUG "value": [ 05:46:28 DEBUG 1712724388, 05:46:28 DEBUG "73100.18086172735" 05:46:28 DEBUG ] 05:46:28 DEBUG } 05:46:28 DEBUG ] 05:46:28 DEBUG } 05:46:28 DEBUG } 05:46:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724388" 05:46:28 INFO [http_cmd]: http status code OK 05:46:28 DEBUG --- status code --- 05:46:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:28 DEBUG --- http response --- 05:46:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:46:28 DEBUG ----- output ----- 05:46:28 DEBUG { 05:46:28 DEBUG "status": "success", 05:46:28 DEBUG "data": { 05:46:28 DEBUG "resultType": "vector", 05:46:28 DEBUG "result": [ 05:46:28 DEBUG { 05:46:28 DEBUG "metric": { 05:46:28 DEBUG "instance": "10.142.0.25:9100" 05:46:28 DEBUG }, 05:46:28 DEBUG "value": [ 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG [...] 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG "instance": "10.142.0.77:9100" 05:46:28 DEBUG }, 05:46:28 DEBUG "value": [ 05:46:28 DEBUG 1712724388, 05:46:28 DEBUG "31867669.43333333" 05:46:28 DEBUG ] 05:46:28 DEBUG } 05:46:28 DEBUG ] 05:46:28 DEBUG } 05:46:28 DEBUG } 05:46:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724388" 05:46:28 INFO [http_cmd]: http status code OK 05:46:28 DEBUG --- status code --- 05:46:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:28 DEBUG --- http response --- 05:46:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:46:28 DEBUG ----- output ----- 05:46:28 DEBUG { 05:46:28 DEBUG "status": "success", 05:46:28 DEBUG "data": { 05:46:28 DEBUG "resultType": "vector", 05:46:28 DEBUG "result": [ 05:46:28 DEBUG { 05:46:28 DEBUG "metric": { 05:46:28 DEBUG "instance": "10.142.0.25:9100" 05:46:28 DEBUG }, 05:46:28 DEBUG "value": [ 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG [...] 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG 05:46:28 DEBUG "instance": "10.142.0.77:9100" 05:46:28 DEBUG }, 05:46:28 DEBUG "value": [ 05:46:28 DEBUG 1712724388, 05:46:28 DEBUG "0.04733333333333362" 05:46:28 DEBUG ] 05:46:28 DEBUG } 05:46:28 DEBUG ] 05:46:28 DEBUG } 05:46:28 DEBUG } 05:46:44 INFO 05:46:44 INFO [loop_until]: kubectl --namespace=xlou top pods 05:46:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:46:44 INFO [loop_until]: OK (rc = 0) 05:46:44 DEBUG --- stdout --- 05:46:44 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 9128m 8791Mi am-654c4fcbcf-9scrj 9036m 8402Mi am-654c4fcbcf-v5qb5 7272m 8380Mi ds-cts-0 5868m 12183Mi ds-cts-1 6348m 12196Mi ds-cts-2 5143m 12359Mi ds-idrepo-0 9899m 5622Mi ds-idrepo-1 355m 5671Mi ds-idrepo-2 373m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 2m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4097m 1658Mi 05:46:44 DEBUG --- stderr --- 05:46:44 DEBUG 05:46:51 INFO 05:46:51 INFO [loop_until]: kubectl --namespace=xlou top node 05:46:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:46:51 INFO [loop_until]: OK (rc = 0) 05:46:51 DEBUG --- stdout --- 05:46:51 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 95m 0% 2304Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8385m 52% 9263Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 73m 0% 3260Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 12898m 81% 6508Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9620m 60% 9287Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 10375m 65% 9626Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 79m 0% 851Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10221m 64% 6291Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 5560m 34% 12849Mi 21% gke-xlou-cdm-ds-32e4dcb1-4hls 5673m 35% 12919Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 443m 2% 6375Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 5376m 33% 12948Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 394m 2% 6340Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4257m 26% 5327Mi 9% 05:46:51 DEBUG --- stderr --- 05:46:51 DEBUG 05:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712724445" 05:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724445" 05:47:25 ERROR [http_cmd]: ERROR 05:47:25 ERROR --- status code --- 05:47:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:47:25 ERROR --- http response --- 05:47:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:47:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:47:25 INFO [http_cmd]: http status code OK 05:47:25 DEBUG --- status code --- 05:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:25 DEBUG --- http response --- 05:47:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724445" 05:47:25 INFO [http_cmd]: http status code OK 05:47:25 DEBUG --- status code --- 05:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:25 DEBUG --- http response --- 05:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:47:25 DEBUG ----- output ----- 05:47:25 DEBUG { 05:47:25 DEBUG "status": "success", 05:47:25 DEBUG "data": { 05:47:25 DEBUG "resultType": "vector", 05:47:25 DEBUG "result": [ 05:47:25 DEBUG { 05:47:25 DEBUG "metric": { 05:47:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:47:25 DEBUG }, 05:47:25 DEBUG "value": [ 05:47:25 DEBUG 05:47:25 DEBUG 05:47:25 DEBUG 05:47:25 DEBUG [...] 05:47:25 DEBUG 05:47:25 DEBUG 05:47:25 DEBUG 05:47:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:47:25 DEBUG }, 05:47:25 DEBUG "value": [ 05:47:25 DEBUG 1712724445, 05:47:25 DEBUG "0" 05:47:25 DEBUG ] 05:47:25 DEBUG } 05:47:25 DEBUG ] 05:47:25 DEBUG } 05:47:25 DEBUG } 05:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724445" 05:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724445" 05:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724445" 05:47:25 INFO [http_cmd]: http status code OK 05:47:25 DEBUG --- status code --- 05:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:25 DEBUG --- http response --- 05:47:25 INFO [http_cmd]: http status code OK 05:47:25 DEBUG --- status code --- 05:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:25 DEBUG --- http response --- 05:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:47:25 DEBUG ----- output ----- 05:47:25 DEBUG { 05:47:25 DEBUG "status": "success", 05:47:25 DEBUG "data": { 05:47:25 DEBUG "resultType": "vector", 05:47:25 DEBUG "result": [ 05:47:25 DEBUG { 05:47:25 DEBUG "metric": { 05:47:25 DEBUG "pod": "ds-idrepo-0" 05:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:47:25 DEBUG ----- output ----- 05:47:25 DEBUG { 05:47:25 DEBUG "status": "success", 05:47:25 DEBUG "data": { 05:47:25 DEBUG "resultType": "vector", 05:47:25 DEBUG "result": [ 05:47:25 DEBUG { 05:47:25 DEBUG "metric": { 05:47:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:47:25 DEBUG }, 05:47:25 DEBUG "value": [ 05:47:25 DEBUG 05:47:25 DEBUG 05:47:25 DEBUG 05:47:25 DEBUG [...] 05:47:25 DEBUG 05:47:25 DEBUG 05:47:25 DEBUG 05:47:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:47:25 DEBUG }, 05:47:25 DEBUG "value": [ 05:47:25 DEBUG 1712724445, 05:47:25 DEBUG "1443.4333333333334" 05:47:25 DEBUG ] 05:47:25 DEBUG } 05:47:25 DEBUG ] 05:47:25 DEBUG } 05:47:25 DEBUG } 05:47:25 DEBUG }, 05:47:25 DEBUG "value": [ 05:47:25 DEBUG 05:47:25 DEBUG 05:47:25 DEBUG 05:47:25 DEBUG [...] 05:47:25 DEBUG 05:47:25 DEBUG 05:47:25 DEBUG 05:47:25 DEBUG "pod": "ds-cts-2" 05:47:25 DEBUG }, 05:47:25 DEBUG "value": [ 05:47:25 DEBUG 1712724445, 05:47:25 DEBUG "30" 05:47:25 DEBUG ] 05:47:25 DEBUG } 05:47:25 DEBUG ] 05:47:25 DEBUG } 05:47:25 DEBUG } 05:47:25 INFO [http_cmd]: http status code OK 05:47:25 DEBUG --- status code --- 05:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:25 DEBUG --- http response --- 05:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:47:25 DEBUG ----- output ----- 05:47:25 DEBUG { 05:47:25 DEBUG "status": "success", 05:47:25 DEBUG "data": { 05:47:25 DEBUG "resultType": "vector", 05:47:25 DEBUG "result": [ 05:47:25 DEBUG { 05:47:25 DEBUG "metric": { 05:47:25 DEBUG "pod": "ds-idrepo-0" 05:47:25 DEBUG }, 05:47:25 DEBUG "value": [ 05:47:25 DEBUG 05:47:25 DEBUG 05:47:25 DEBUG 05:47:25 DEBUG [...] 05:47:25 DEBUG 05:47:25 DEBUG 05:47:25 DEBUG 05:47:25 DEBUG "pod": "ds-cts-2" 05:47:25 DEBUG }, 05:47:25 DEBUG "value": [ 05:47:25 DEBUG 1712724445, 05:47:25 DEBUG "0" 05:47:25 DEBUG ] 05:47:25 DEBUG } 05:47:25 DEBUG ] 05:47:25 DEBUG } 05:47:25 DEBUG } 05:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724445" 05:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724445" 05:47:25 INFO [http_cmd]: http status code OK 05:47:25 DEBUG --- status code --- 05:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:25 DEBUG --- http response --- 05:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:47:25 INFO [http_cmd]: http status code OK 05:47:25 DEBUG ----- output ----- 05:47:25 DEBUG --- status code --- 05:47:25 DEBUG { 05:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:25 DEBUG "status": "success", 05:47:25 DEBUG --- http response --- 05:47:25 DEBUG "data": { 05:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:47:25 DEBUG "resultType": "vector", 05:47:25 DEBUG "result": [ 05:47:25 DEBUG { 05:47:25 DEBUG "metric": { 05:47:25 DEBUG "instance": "10.142.0.25:9100" 05:47:25 DEBUG }, 05:47:25 DEBUG "value": [ 05:47:25 DEBUG 05:47:25 DEBUG 05:47:25 DEBUG 05:47:25 DEBUG [...] 05:47:25 DEBUG 05:47:25 DEBUG 05:47:25 DEBUG 05:47:25 DEBUG "instance": "10.142.0.77:9100" 05:47:25 DEBUG }, 05:47:25 DEBUG "value": [ 05:47:25 DEBUG 1712724445, 05:47:25 DEBUG "315697333248" 05:47:25 DEBUG ] 05:47:25 DEBUG ----- output ----- 05:47:25 DEBUG } 05:47:25 DEBUG { 05:47:25 DEBUG ] 05:47:25 DEBUG "status": "success", 05:47:25 DEBUG } 05:47:25 DEBUG "data": { 05:47:25 DEBUG } 05:47:25 DEBUG "resultType": "vector", 05:47:25 DEBUG "result": [ 05:47:25 DEBUG { 05:47:25 DEBUG "metric": { 05:47:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:47:25 DEBUG }, 05:47:25 DEBUG "value": [ 05:47:25 DEBUG 05:47:25 DEBUG 05:47:25 DEBUG 05:47:25 DEBUG [...] 05:47:25 DEBUG 05:47:25 DEBUG 05:47:25 DEBUG 05:47:25 DEBUG "pod": "ds-idrepo-2" 05:47:25 DEBUG }, 05:47:25 DEBUG "value": [ 05:47:25 DEBUG 1712724445, 05:47:25 DEBUG "0" 05:47:25 DEBUG ] 05:47:25 DEBUG } 05:47:25 DEBUG ] 05:47:25 DEBUG } 05:47:25 DEBUG } 05:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724445" 05:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724445" 05:47:25 INFO [http_cmd]: http status code OK 05:47:25 DEBUG --- status code --- 05:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:25 DEBUG --- http response --- 05:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:47:25 DEBUG ----- output ----- 05:47:25 DEBUG { 05:47:25 DEBUG "status": "success", 05:47:25 INFO [http_cmd]: http status code OK 05:47:25 DEBUG "data": { 05:47:25 DEBUG --- status code --- 05:47:25 DEBUG "resultType": "vector", 05:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:25 DEBUG "result": [ 05:47:25 DEBUG --- http response --- 05:47:25 DEBUG { 05:47:25 DEBUG "metric": { 05:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:47:25 DEBUG "instance": "10.142.0.25:9100" 05:47:25 DEBUG ----- output ----- 05:47:25 DEBUG }, 05:47:25 DEBUG { 05:47:25 DEBUG "value": [ 05:47:25 DEBUG "status": "success", 05:47:25 DEBUG 05:47:25 DEBUG "data": { 05:47:25 DEBUG 05:47:25 DEBUG "resultType": "vector", 05:47:25 DEBUG 05:47:25 DEBUG "result": [ 05:47:25 DEBUG [...] 05:47:25 DEBUG { 05:47:25 DEBUG 05:47:25 DEBUG "metric": { 05:47:25 DEBUG 05:47:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:47:25 DEBUG 05:47:25 DEBUG }, 05:47:25 DEBUG "instance": "10.142.0.77:9100" 05:47:25 DEBUG "value": [ 05:47:25 DEBUG }, 05:47:25 DEBUG "value": [ 05:47:25 DEBUG 1712724445, 05:47:25 DEBUG 05:47:25 DEBUG "33949250.8" 05:47:25 DEBUG ] 05:47:25 DEBUG 05:47:25 DEBUG 05:47:25 DEBUG } 05:47:25 DEBUG ] 05:47:25 DEBUG } 05:47:25 DEBUG } 05:47:25 DEBUG [...] 05:47:25 DEBUG 05:47:25 DEBUG 05:47:25 DEBUG 05:47:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:47:25 DEBUG }, 05:47:25 DEBUG "value": [ 05:47:25 DEBUG 1712724445, 05:47:25 DEBUG "89774" 05:47:25 DEBUG ] 05:47:25 DEBUG } 05:47:25 DEBUG ] 05:47:25 DEBUG } 05:47:25 DEBUG } 05:47:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724446" 05:47:26 INFO [http_cmd]: http status code OK 05:47:26 DEBUG --- status code --- 05:47:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:26 DEBUG --- http response --- 05:47:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:47:26 DEBUG ----- output ----- 05:47:26 DEBUG { 05:47:26 DEBUG "status": "success", 05:47:26 DEBUG "data": { 05:47:26 DEBUG "resultType": "vector", 05:47:26 DEBUG "result": [ 05:47:26 DEBUG { 05:47:26 DEBUG "metric": { 05:47:26 DEBUG "pod": "ds-idrepo-0" 05:47:26 DEBUG }, 05:47:26 DEBUG "value": [ 05:47:26 DEBUG 05:47:26 DEBUG 05:47:26 DEBUG 05:47:26 DEBUG [...] 05:47:26 DEBUG 05:47:26 DEBUG 05:47:26 DEBUG 05:47:26 DEBUG "pod": "ds-idrepo-1" 05:47:26 DEBUG }, 05:47:26 DEBUG "value": [ 05:47:26 DEBUG 1712724446, 05:47:26 DEBUG "0" 05:47:26 DEBUG ] 05:47:26 DEBUG } 05:47:26 DEBUG ] 05:47:26 DEBUG } 05:47:26 DEBUG } 05:47:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724446" 05:47:26 INFO [http_cmd]: http status code OK 05:47:26 DEBUG --- status code --- 05:47:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:26 DEBUG --- http response --- 05:47:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:47:26 DEBUG ----- output ----- 05:47:26 DEBUG { 05:47:26 DEBUG "status": "success", 05:47:26 DEBUG "data": { 05:47:26 DEBUG "resultType": "vector", 05:47:26 DEBUG "result": [ 05:47:26 DEBUG { 05:47:26 DEBUG "metric": { 05:47:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:47:26 DEBUG }, 05:47:26 DEBUG "value": [ 05:47:26 DEBUG 05:47:26 DEBUG 05:47:26 DEBUG 05:47:26 DEBUG [...] 05:47:26 DEBUG 05:47:26 DEBUG 05:47:26 DEBUG 05:47:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:47:26 DEBUG }, 05:47:26 DEBUG "value": [ 05:47:26 DEBUG 1712724446, 05:47:26 DEBUG "1436.2" 05:47:26 DEBUG ] 05:47:26 DEBUG } 05:47:26 DEBUG ] 05:47:26 DEBUG } 05:47:26 DEBUG } 05:47:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724446" 05:47:26 INFO [http_cmd]: http status code OK 05:47:26 DEBUG --- status code --- 05:47:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:26 DEBUG --- http response --- 05:47:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:47:26 DEBUG ----- output ----- 05:47:26 DEBUG { 05:47:26 DEBUG "status": "success", 05:47:26 DEBUG "data": { 05:47:26 DEBUG "resultType": "vector", 05:47:26 DEBUG "result": [ 05:47:26 DEBUG { 05:47:26 DEBUG "metric": { 05:47:26 DEBUG "instance": "10.142.0.25:9100" 05:47:26 DEBUG }, 05:47:26 DEBUG "value": [ 05:47:26 DEBUG 05:47:26 DEBUG 05:47:26 DEBUG 05:47:26 DEBUG [...] 05:47:26 DEBUG 05:47:26 DEBUG 05:47:26 DEBUG 05:47:26 DEBUG "instance": "10.142.0.77:9100" 05:47:26 DEBUG }, 05:47:26 DEBUG "value": [ 05:47:26 DEBUG 1712724446, 05:47:26 DEBUG "0.032933333333327396" 05:47:26 DEBUG ] 05:47:26 DEBUG } 05:47:26 DEBUG ] 05:47:26 DEBUG } 05:47:26 DEBUG } 05:47:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724447" 05:47:27 INFO [http_cmd]: http status code OK 05:47:27 DEBUG --- status code --- 05:47:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:27 DEBUG --- http response --- 05:47:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:47:27 DEBUG ----- output ----- 05:47:27 DEBUG { 05:47:27 DEBUG "status": "success", 05:47:27 DEBUG "data": { 05:47:27 DEBUG "resultType": "vector", 05:47:27 DEBUG "result": [ 05:47:27 DEBUG { 05:47:27 DEBUG "metric": { 05:47:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:47:27 DEBUG }, 05:47:27 DEBUG "value": [ 05:47:27 DEBUG 05:47:27 DEBUG 05:47:27 DEBUG 05:47:27 DEBUG [...] 05:47:27 DEBUG 05:47:27 DEBUG 05:47:27 DEBUG 05:47:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:47:27 DEBUG }, 05:47:27 DEBUG "value": [ 05:47:27 DEBUG 1712724447, 05:47:27 DEBUG "339529063.74101835" 05:47:27 DEBUG ] 05:47:27 DEBUG } 05:47:27 DEBUG ] 05:47:27 DEBUG } 05:47:27 DEBUG } 05:47:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712724447" 05:47:27 INFO [http_cmd]: http status code OK 05:47:27 DEBUG --- status code --- 05:47:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:27 DEBUG --- http response --- 05:47:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:47:27 DEBUG ----- output ----- 05:47:27 DEBUG { 05:47:27 DEBUG "status": "success", 05:47:27 DEBUG "data": { 05:47:27 DEBUG "resultType": "vector", 05:47:27 DEBUG "result": [ 05:47:27 DEBUG { 05:47:27 DEBUG "metric": { 05:47:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:47:27 DEBUG }, 05:47:27 DEBUG "value": [ 05:47:27 DEBUG 05:47:27 DEBUG 05:47:27 DEBUG 05:47:27 DEBUG [...] 05:47:27 DEBUG 05:47:27 DEBUG 05:47:27 DEBUG 05:47:27 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:47:27 DEBUG }, 05:47:27 DEBUG "value": [ 05:47:27 DEBUG 1712724447, 05:47:27 DEBUG "4.1457340108463" 05:47:27 DEBUG ] 05:47:27 DEBUG } 05:47:27 DEBUG ] 05:47:27 DEBUG } 05:47:27 DEBUG } 05:47:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712724447" 05:47:27 INFO [http_cmd]: http status code OK 05:47:27 DEBUG --- status code --- 05:47:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:27 DEBUG --- http response --- 05:47:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:47:27 DEBUG ----- output ----- 05:47:27 DEBUG { 05:47:27 DEBUG "status": "success", 05:47:27 DEBUG "data": { 05:47:27 DEBUG "resultType": "vector", 05:47:27 DEBUG "result": [ 05:47:27 DEBUG { 05:47:27 DEBUG "metric": { 05:47:27 DEBUG "pod": "ds-idrepo-0" 05:47:27 DEBUG }, 05:47:27 DEBUG "value": [ 05:47:27 DEBUG 05:47:27 DEBUG 05:47:27 DEBUG 05:47:27 DEBUG [...] 05:47:27 DEBUG 05:47:27 DEBUG 05:47:27 DEBUG 05:47:27 DEBUG "pod": "ds-cts-2" 05:47:27 DEBUG }, 05:47:27 DEBUG "value": [ 05:47:27 DEBUG 1712724447, 05:47:27 DEBUG "0" 05:47:27 DEBUG ] 05:47:27 DEBUG } 05:47:27 DEBUG ] 05:47:27 DEBUG } 05:47:27 DEBUG } 05:47:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724447" 05:47:27 INFO [http_cmd]: http status code OK 05:47:27 DEBUG --- status code --- 05:47:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:27 DEBUG --- http response --- 05:47:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:47:27 DEBUG ----- output ----- 05:47:27 DEBUG { 05:47:27 DEBUG "status": "success", 05:47:27 DEBUG "data": { 05:47:27 DEBUG "resultType": "vector", 05:47:27 DEBUG "result": [ 05:47:27 DEBUG { 05:47:27 DEBUG "metric": { 05:47:27 DEBUG "pod": "ds-idrepo-2" 05:47:27 DEBUG }, 05:47:27 DEBUG "value": [ 05:47:27 DEBUG 05:47:27 DEBUG 05:47:27 DEBUG 05:47:27 DEBUG [...] 05:47:27 DEBUG 05:47:27 DEBUG 05:47:27 DEBUG 05:47:27 DEBUG "pod": "ds-cts-2" 05:47:27 DEBUG }, 05:47:27 DEBUG "value": [ 05:47:27 DEBUG 1712724447, 05:47:27 DEBUG "0" 05:47:27 DEBUG ] 05:47:27 DEBUG } 05:47:27 DEBUG ] 05:47:27 DEBUG } 05:47:27 DEBUG } 05:47:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724447" 05:47:27 INFO [http_cmd]: http status code OK 05:47:27 DEBUG --- status code --- 05:47:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:27 DEBUG --- http response --- 05:47:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:47:27 DEBUG ----- output ----- 05:47:27 DEBUG { 05:47:27 DEBUG "status": "success", 05:47:27 DEBUG "data": { 05:47:27 DEBUG "resultType": "vector", 05:47:27 DEBUG "result": [ 05:47:27 DEBUG { 05:47:27 DEBUG "metric": { 05:47:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:47:27 DEBUG }, 05:47:27 DEBUG "value": [ 05:47:27 DEBUG 05:47:27 DEBUG 05:47:27 DEBUG 05:47:27 DEBUG [...] 05:47:27 DEBUG 05:47:27 DEBUG 05:47:27 DEBUG 05:47:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:47:27 DEBUG }, 05:47:27 DEBUG "value": [ 05:47:27 DEBUG 1712724447, 05:47:27 DEBUG "7.001234823997754" 05:47:27 DEBUG ] 05:47:27 DEBUG } 05:47:27 DEBUG ] 05:47:27 DEBUG } 05:47:27 DEBUG } 05:47:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724448" 05:47:28 INFO [http_cmd]: http status code OK 05:47:28 DEBUG --- status code --- 05:47:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:28 DEBUG --- http response --- 05:47:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:47:28 DEBUG ----- output ----- 05:47:28 DEBUG { 05:47:28 DEBUG "status": "success", 05:47:28 DEBUG "data": { 05:47:28 DEBUG "resultType": "vector", 05:47:28 DEBUG "result": [ 05:47:28 DEBUG { 05:47:28 DEBUG "metric": { 05:47:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:47:28 DEBUG }, 05:47:28 DEBUG "value": [ 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG [...] 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:47:28 DEBUG }, 05:47:28 DEBUG "value": [ 05:47:28 DEBUG 1712724448, 05:47:28 DEBUG "1436.4333333333334" 05:47:28 DEBUG ] 05:47:28 DEBUG } 05:47:28 DEBUG ] 05:47:28 DEBUG } 05:47:28 DEBUG } 05:47:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712724448" 05:47:28 INFO [http_cmd]: http status code OK 05:47:28 DEBUG --- status code --- 05:47:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:28 DEBUG --- http response --- 05:47:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:47:28 DEBUG ----- output ----- 05:47:28 DEBUG { 05:47:28 DEBUG "status": "success", 05:47:28 DEBUG "data": { 05:47:28 DEBUG "resultType": "vector", 05:47:28 DEBUG "result": [ 05:47:28 DEBUG { 05:47:28 DEBUG "metric": { 05:47:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:47:28 DEBUG }, 05:47:28 DEBUG "value": [ 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG [...] 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:47:28 DEBUG }, 05:47:28 DEBUG "value": [ 05:47:28 DEBUG 1712724448, 05:47:28 DEBUG "0.043" 05:47:28 DEBUG ] 05:47:28 DEBUG } 05:47:28 DEBUG ] 05:47:28 DEBUG } 05:47:28 DEBUG } 05:47:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724448" 05:47:28 INFO [http_cmd]: http status code OK 05:47:28 DEBUG --- status code --- 05:47:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:28 DEBUG --- http response --- 05:47:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:47:28 DEBUG ----- output ----- 05:47:28 DEBUG { 05:47:28 DEBUG "status": "success", 05:47:28 DEBUG "data": { 05:47:28 DEBUG "resultType": "vector", 05:47:28 DEBUG "result": [ 05:47:28 DEBUG { 05:47:28 DEBUG "metric": { 05:47:28 DEBUG "pod": "ds-idrepo-0" 05:47:28 DEBUG }, 05:47:28 DEBUG "value": [ 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG [...] 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG "pod": "ds-idrepo-1" 05:47:28 DEBUG }, 05:47:28 DEBUG "value": [ 05:47:28 DEBUG 1712724448, 05:47:28 DEBUG "0" 05:47:28 DEBUG ] 05:47:28 DEBUG } 05:47:28 DEBUG ] 05:47:28 DEBUG } 05:47:28 DEBUG } 05:47:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724448" 05:47:28 INFO [http_cmd]: http status code OK 05:47:28 DEBUG --- status code --- 05:47:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:28 DEBUG --- http response --- 05:47:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:47:28 DEBUG ----- output ----- 05:47:28 DEBUG { 05:47:28 DEBUG "status": "success", 05:47:28 DEBUG "data": { 05:47:28 DEBUG "resultType": "vector", 05:47:28 DEBUG "result": [ 05:47:28 DEBUG { 05:47:28 DEBUG "metric": { 05:47:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:47:28 DEBUG }, 05:47:28 DEBUG "value": [ 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG [...] 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:47:28 DEBUG }, 05:47:28 DEBUG "value": [ 05:47:28 DEBUG 1712724448, 05:47:28 DEBUG "0" 05:47:28 DEBUG ] 05:47:28 DEBUG } 05:47:28 DEBUG ] 05:47:28 DEBUG } 05:47:28 DEBUG } 05:47:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724448" 05:47:28 INFO [http_cmd]: http status code OK 05:47:28 DEBUG --- status code --- 05:47:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:28 DEBUG --- http response --- 05:47:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:47:28 DEBUG ----- output ----- 05:47:28 DEBUG { 05:47:28 DEBUG "status": "success", 05:47:28 DEBUG "data": { 05:47:28 DEBUG "resultType": "vector", 05:47:28 DEBUG "result": [ 05:47:28 DEBUG { 05:47:28 DEBUG "metric": { 05:47:28 DEBUG "pod": "ds-idrepo-0" 05:47:28 DEBUG }, 05:47:28 DEBUG "value": [ 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG [...] 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG "pod": "ds-idrepo-1" 05:47:28 DEBUG }, 05:47:28 DEBUG "value": [ 05:47:28 DEBUG 1712724448, 05:47:28 DEBUG "0" 05:47:28 DEBUG ] 05:47:28 DEBUG } 05:47:28 DEBUG ] 05:47:28 DEBUG } 05:47:28 DEBUG } 05:47:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712724448" 05:47:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724448" 05:47:28 INFO [http_cmd]: http status code OK 05:47:28 DEBUG --- status code --- 05:47:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:28 DEBUG --- http response --- 05:47:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:47:28 INFO [http_cmd]: http status code OK 05:47:28 DEBUG ----- output ----- 05:47:28 DEBUG --- status code --- 05:47:28 DEBUG { 05:47:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:28 DEBUG "status": "success", 05:47:28 DEBUG --- http response --- 05:47:28 DEBUG "data": { 05:47:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:47:28 DEBUG "resultType": "vector", 05:47:28 DEBUG ----- output ----- 05:47:28 DEBUG { 05:47:28 DEBUG "result": [ 05:47:28 DEBUG "status": "success", 05:47:28 DEBUG { 05:47:28 DEBUG "data": { 05:47:28 DEBUG "metric": { 05:47:28 DEBUG "resultType": "vector", 05:47:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:47:28 DEBUG "result": [ 05:47:28 DEBUG }, 05:47:28 DEBUG { 05:47:28 DEBUG "value": [ 05:47:28 DEBUG "metric": { 05:47:28 DEBUG 05:47:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:47:28 DEBUG }, 05:47:28 DEBUG "value": [ 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG [...] 05:47:28 DEBUG [...] 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:47:28 DEBUG }, 05:47:28 DEBUG "value": [ 05:47:28 DEBUG 1712724448, 05:47:28 DEBUG "2486562816" 05:47:28 DEBUG ] 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:47:28 DEBUG } 05:47:28 DEBUG ] 05:47:28 DEBUG } 05:47:28 DEBUG } 05:47:28 DEBUG }, 05:47:28 DEBUG "value": [ 05:47:28 DEBUG 1712724448, 05:47:28 DEBUG "0" 05:47:28 DEBUG ] 05:47:28 DEBUG } 05:47:28 DEBUG ] 05:47:28 DEBUG } 05:47:28 DEBUG } 05:47:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724448" 05:47:28 INFO [http_cmd]: http status code OK 05:47:28 DEBUG --- status code --- 05:47:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:28 DEBUG --- http response --- 05:47:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:47:28 DEBUG ----- output ----- 05:47:28 DEBUG { 05:47:28 DEBUG "status": "success", 05:47:28 DEBUG "data": { 05:47:28 DEBUG "resultType": "vector", 05:47:28 DEBUG "result": [ 05:47:28 DEBUG { 05:47:28 DEBUG "metric": { 05:47:28 DEBUG "instance": "10.142.0.25:9100" 05:47:28 DEBUG }, 05:47:28 DEBUG "value": [ 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG [...] 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG "instance": "10.142.0.77:9100" 05:47:28 DEBUG }, 05:47:28 DEBUG "value": [ 05:47:28 DEBUG 1712724448, 05:47:28 DEBUG "33950453.46666667" 05:47:28 DEBUG ] 05:47:28 DEBUG } 05:47:28 DEBUG ] 05:47:28 DEBUG } 05:47:28 DEBUG } 05:47:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724448" 05:47:28 INFO [http_cmd]: http status code OK 05:47:28 DEBUG --- status code --- 05:47:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:28 DEBUG --- http response --- 05:47:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:47:28 DEBUG ----- output ----- 05:47:28 DEBUG { 05:47:28 DEBUG "status": "success", 05:47:28 DEBUG "data": { 05:47:28 DEBUG "resultType": "vector", 05:47:28 DEBUG "result": [ 05:47:28 DEBUG { 05:47:28 DEBUG "metric": { 05:47:28 DEBUG "instance": "10.142.0.25:9100" 05:47:28 DEBUG }, 05:47:28 DEBUG "value": [ 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG [...] 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG 05:47:28 DEBUG "instance": "10.142.0.77:9100" 05:47:28 DEBUG }, 05:47:28 DEBUG "value": [ 05:47:28 DEBUG 1712724448, 05:47:28 DEBUG "0.01766666666666623" 05:47:28 DEBUG ] 05:47:28 DEBUG } 05:47:28 DEBUG ] 05:47:28 DEBUG } 05:47:28 DEBUG } 05:47:44 INFO 05:47:44 INFO [loop_until]: kubectl --namespace=xlou top pods 05:47:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:47:44 INFO [loop_until]: OK (rc = 0) 05:47:44 DEBUG --- stdout --- 05:47:44 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 8687m 8777Mi am-654c4fcbcf-9scrj 8439m 8393Mi am-654c4fcbcf-v5qb5 6970m 8371Mi ds-cts-0 4019m 12788Mi ds-cts-1 3940m 12934Mi ds-cts-2 3601m 12130Mi ds-idrepo-0 9754m 5622Mi ds-idrepo-1 361m 5671Mi ds-idrepo-2 425m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 2m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4104m 1659Mi 05:47:44 DEBUG --- stderr --- 05:47:44 DEBUG 05:47:51 INFO 05:47:51 INFO [loop_until]: kubectl --namespace=xlou top node 05:47:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:47:52 INFO [loop_until]: OK (rc = 0) 05:47:52 DEBUG --- stdout --- 05:47:52 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 96m 0% 2306Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 7770m 48% 9264Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 78m 0% 3256Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13028m 81% 6529Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9295m 58% 9285Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9497m 59% 9627Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 76m 0% 855Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9852m 62% 6289Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 4127m 25% 13520Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 3676m 23% 12773Mi 21% gke-xlou-cdm-ds-32e4dcb1-gxzj 490m 3% 6374Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 4087m 25% 12681Mi 21% gke-xlou-cdm-ds-32e4dcb1-rr7g 414m 2% 6340Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4232m 26% 5326Mi 9% 05:47:52 DEBUG --- stderr --- 05:47:52 DEBUG 05:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712724505" 05:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724505" 05:48:25 ERROR [http_cmd]: ERROR 05:48:25 ERROR --- status code --- 05:48:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:48:25 ERROR --- http response --- 05:48:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:48:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:48:25 INFO [http_cmd]: http status code OK 05:48:25 DEBUG --- status code --- 05:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:25 DEBUG --- http response --- 05:48:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724505" 05:48:25 INFO [http_cmd]: http status code OK 05:48:25 DEBUG --- status code --- 05:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:25 DEBUG --- http response --- 05:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:48:25 DEBUG ----- output ----- 05:48:25 DEBUG { 05:48:25 DEBUG "status": "success", 05:48:25 DEBUG "data": { 05:48:25 DEBUG "resultType": "vector", 05:48:25 DEBUG "result": [ 05:48:25 DEBUG { 05:48:25 DEBUG "metric": { 05:48:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:48:25 DEBUG }, 05:48:25 DEBUG "value": [ 05:48:25 DEBUG 05:48:25 DEBUG 05:48:25 DEBUG 05:48:25 DEBUG [...] 05:48:25 DEBUG 05:48:25 DEBUG 05:48:25 DEBUG 05:48:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:48:25 DEBUG }, 05:48:25 DEBUG "value": [ 05:48:25 DEBUG 1712724505, 05:48:25 DEBUG "0" 05:48:25 DEBUG ] 05:48:25 DEBUG } 05:48:25 DEBUG ] 05:48:25 DEBUG } 05:48:25 DEBUG } 05:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724505" 05:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724505" 05:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724505" 05:48:25 INFO [http_cmd]: http status code OK 05:48:25 DEBUG --- status code --- 05:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:25 DEBUG --- http response --- 05:48:25 INFO [http_cmd]: http status code OK 05:48:25 INFO [http_cmd]: http status code OK 05:48:25 DEBUG --- status code --- 05:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:48:25 DEBUG --- status code --- 05:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:25 DEBUG ----- output ----- 05:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:25 DEBUG --- http response --- 05:48:25 DEBUG { 05:48:25 DEBUG --- http response --- 05:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:48:25 DEBUG "status": "success", 05:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:48:25 DEBUG ----- output ----- 05:48:25 DEBUG "data": { 05:48:25 DEBUG ----- output ----- 05:48:25 DEBUG { 05:48:25 DEBUG "resultType": "vector", 05:48:25 DEBUG { 05:48:25 DEBUG "status": "success", 05:48:25 DEBUG "result": [ 05:48:25 DEBUG "data": { 05:48:25 DEBUG "resultType": "vector", 05:48:25 DEBUG { 05:48:25 DEBUG "result": [ 05:48:25 DEBUG { 05:48:25 DEBUG "metric": { 05:48:25 DEBUG "metric": { 05:48:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:48:25 DEBUG "pod": "ds-idrepo-0" 05:48:25 DEBUG }, 05:48:25 DEBUG "value": [ 05:48:25 DEBUG 05:48:25 DEBUG }, 05:48:25 DEBUG "value": [ 05:48:25 DEBUG "status": "success", 05:48:25 DEBUG 05:48:25 DEBUG 05:48:25 DEBUG 05:48:25 DEBUG 05:48:25 DEBUG [...] 05:48:25 DEBUG "data": { 05:48:25 DEBUG 05:48:25 DEBUG 05:48:25 DEBUG 05:48:25 DEBUG [...] 05:48:25 DEBUG 05:48:25 DEBUG "resultType": "vector", 05:48:25 DEBUG 05:48:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:48:25 DEBUG "result": [ 05:48:25 DEBUG 05:48:25 DEBUG }, 05:48:25 DEBUG { 05:48:25 DEBUG 05:48:25 DEBUG "value": [ 05:48:25 DEBUG "metric": { 05:48:25 DEBUG "pod": "ds-cts-2" 05:48:25 DEBUG 1712724505, 05:48:25 DEBUG "pod": "ds-idrepo-0" 05:48:25 DEBUG }, 05:48:25 DEBUG "1448.8333333333333" 05:48:25 DEBUG ] 05:48:25 DEBUG } 05:48:25 DEBUG "value": [ 05:48:25 DEBUG 1712724505, 05:48:25 DEBUG ] 05:48:25 DEBUG "29.833333333333332" 05:48:25 DEBUG ] 05:48:25 DEBUG } 05:48:25 DEBUG } 05:48:25 DEBUG }, 05:48:25 DEBUG "value": [ 05:48:25 DEBUG ] 05:48:25 DEBUG 05:48:25 DEBUG 05:48:25 DEBUG 05:48:25 DEBUG } 05:48:25 DEBUG [...] 05:48:25 DEBUG 05:48:25 DEBUG 05:48:25 DEBUG 05:48:25 DEBUG } 05:48:25 DEBUG "pod": "ds-cts-2" 05:48:25 DEBUG }, 05:48:25 DEBUG "value": [ 05:48:25 DEBUG 1712724505, 05:48:25 DEBUG "0" 05:48:25 DEBUG ] 05:48:25 DEBUG } 05:48:25 DEBUG } 05:48:25 DEBUG ] 05:48:25 DEBUG } 05:48:25 DEBUG } 05:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724505" 05:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724505" 05:48:25 INFO [http_cmd]: http status code OK 05:48:25 DEBUG --- status code --- 05:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:25 DEBUG --- http response --- 05:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:48:25 INFO [http_cmd]: http status code OK 05:48:25 DEBUG ----- output ----- 05:48:25 DEBUG --- status code --- 05:48:25 DEBUG { 05:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:25 DEBUG --- http response --- 05:48:25 DEBUG "status": "success", 05:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:48:25 DEBUG "data": { 05:48:25 DEBUG ----- output ----- 05:48:25 DEBUG "resultType": "vector", 05:48:25 DEBUG { 05:48:25 DEBUG "result": [ 05:48:25 DEBUG { 05:48:25 DEBUG "status": "success", 05:48:25 DEBUG "metric": { 05:48:25 DEBUG "data": { 05:48:25 DEBUG "instance": "10.142.0.25:9100" 05:48:25 DEBUG "resultType": "vector", 05:48:25 DEBUG }, 05:48:25 DEBUG "result": [ 05:48:25 DEBUG "value": [ 05:48:25 DEBUG { 05:48:25 DEBUG 05:48:25 DEBUG "metric": { 05:48:25 DEBUG 05:48:25 DEBUG 05:48:25 DEBUG [...] 05:48:25 DEBUG 05:48:25 DEBUG 05:48:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:48:25 DEBUG 05:48:25 DEBUG }, 05:48:25 DEBUG "instance": "10.142.0.77:9100" 05:48:25 DEBUG "value": [ 05:48:25 DEBUG }, 05:48:25 DEBUG 05:48:25 DEBUG "value": [ 05:48:25 DEBUG 05:48:25 DEBUG 1712724505, 05:48:25 DEBUG 05:48:25 DEBUG "317155103744" 05:48:25 DEBUG [...] 05:48:25 DEBUG ] 05:48:25 DEBUG 05:48:25 DEBUG } 05:48:25 DEBUG ] 05:48:25 DEBUG 05:48:25 DEBUG } 05:48:25 DEBUG 05:48:25 DEBUG } 05:48:25 DEBUG "pod": "ds-idrepo-2" 05:48:25 DEBUG }, 05:48:25 DEBUG "value": [ 05:48:25 DEBUG 1712724505, 05:48:25 DEBUG "0" 05:48:25 DEBUG ] 05:48:25 DEBUG } 05:48:25 DEBUG ] 05:48:25 DEBUG } 05:48:25 DEBUG } 05:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724505" 05:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724505" 05:48:25 INFO [http_cmd]: http status code OK 05:48:25 INFO [http_cmd]: http status code OK 05:48:25 DEBUG --- status code --- 05:48:25 DEBUG --- status code --- 05:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:25 DEBUG --- http response --- 05:48:25 DEBUG --- http response --- 05:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:48:25 DEBUG ----- output ----- 05:48:25 DEBUG ----- output ----- 05:48:25 DEBUG { 05:48:25 DEBUG { 05:48:25 DEBUG "status": "success", 05:48:25 DEBUG "data": { 05:48:25 DEBUG "resultType": "vector", 05:48:25 DEBUG "status": "success", 05:48:25 DEBUG "result": [ 05:48:25 DEBUG "data": { 05:48:25 DEBUG { 05:48:25 DEBUG "resultType": "vector", 05:48:25 DEBUG "metric": { 05:48:25 DEBUG "result": [ 05:48:25 DEBUG "instance": "10.142.0.25:9100" 05:48:25 DEBUG { 05:48:25 DEBUG }, 05:48:25 DEBUG "metric": { 05:48:25 DEBUG "value": [ 05:48:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:48:25 DEBUG 05:48:25 DEBUG }, 05:48:25 DEBUG 05:48:25 DEBUG "value": [ 05:48:25 DEBUG 05:48:25 DEBUG 05:48:25 DEBUG [...] 05:48:25 DEBUG 05:48:25 DEBUG 05:48:25 DEBUG 05:48:25 DEBUG 05:48:25 DEBUG [...] 05:48:25 DEBUG 05:48:25 DEBUG 05:48:25 DEBUG 05:48:25 DEBUG 05:48:25 DEBUG "instance": "10.142.0.77:9100" 05:48:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:48:25 DEBUG }, 05:48:25 DEBUG }, 05:48:25 DEBUG "value": [ 05:48:25 DEBUG "value": [ 05:48:25 DEBUG 1712724505, 05:48:25 DEBUG 1712724505, 05:48:25 DEBUG "34076790.93333334" 05:48:25 DEBUG ] 05:48:25 DEBUG } 05:48:25 DEBUG ] 05:48:25 DEBUG } 05:48:25 DEBUG } 05:48:25 DEBUG "176328" 05:48:25 DEBUG ] 05:48:25 DEBUG } 05:48:25 DEBUG ] 05:48:25 DEBUG } 05:48:25 DEBUG } 05:48:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724506" 05:48:26 INFO [http_cmd]: http status code OK 05:48:26 DEBUG --- status code --- 05:48:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:26 DEBUG --- http response --- 05:48:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:48:26 DEBUG ----- output ----- 05:48:26 DEBUG { 05:48:26 DEBUG "status": "success", 05:48:26 DEBUG "data": { 05:48:26 DEBUG "resultType": "vector", 05:48:26 DEBUG "result": [ 05:48:26 DEBUG { 05:48:26 DEBUG "metric": { 05:48:26 DEBUG "pod": "ds-idrepo-0" 05:48:26 DEBUG }, 05:48:26 DEBUG "value": [ 05:48:26 DEBUG 05:48:26 DEBUG 05:48:26 DEBUG 05:48:26 DEBUG [...] 05:48:26 DEBUG 05:48:26 DEBUG 05:48:26 DEBUG 05:48:26 DEBUG "pod": "ds-idrepo-1" 05:48:26 DEBUG }, 05:48:26 DEBUG "value": [ 05:48:26 DEBUG 1712724506, 05:48:26 DEBUG "0" 05:48:26 DEBUG ] 05:48:26 DEBUG } 05:48:26 DEBUG ] 05:48:26 DEBUG } 05:48:26 DEBUG } 05:48:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724506" 05:48:26 INFO [http_cmd]: http status code OK 05:48:26 DEBUG --- status code --- 05:48:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:26 DEBUG --- http response --- 05:48:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:48:26 DEBUG ----- output ----- 05:48:26 DEBUG { 05:48:26 DEBUG "status": "success", 05:48:26 DEBUG "data": { 05:48:26 DEBUG "resultType": "vector", 05:48:26 DEBUG "result": [ 05:48:26 DEBUG { 05:48:26 DEBUG "metric": { 05:48:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:48:26 DEBUG }, 05:48:26 DEBUG "value": [ 05:48:26 DEBUG 05:48:26 DEBUG 05:48:26 DEBUG 05:48:26 DEBUG [...] 05:48:26 DEBUG 05:48:26 DEBUG 05:48:26 DEBUG 05:48:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:48:26 DEBUG }, 05:48:26 DEBUG "value": [ 05:48:26 DEBUG 1712724506, 05:48:26 DEBUG "1448.9666666666667" 05:48:26 DEBUG ] 05:48:26 DEBUG } 05:48:26 DEBUG ] 05:48:26 DEBUG } 05:48:26 DEBUG } 05:48:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724506" 05:48:26 INFO [http_cmd]: http status code OK 05:48:26 DEBUG --- status code --- 05:48:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:26 DEBUG --- http response --- 05:48:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:48:26 DEBUG ----- output ----- 05:48:26 DEBUG { 05:48:26 DEBUG "status": "success", 05:48:26 DEBUG "data": { 05:48:26 DEBUG "resultType": "vector", 05:48:26 DEBUG "result": [ 05:48:26 DEBUG { 05:48:26 DEBUG "metric": { 05:48:26 DEBUG "instance": "10.142.0.25:9100" 05:48:26 DEBUG }, 05:48:26 DEBUG "value": [ 05:48:26 DEBUG 05:48:26 DEBUG 05:48:26 DEBUG 05:48:26 DEBUG [...] 05:48:26 DEBUG 05:48:26 DEBUG 05:48:26 DEBUG 05:48:26 DEBUG "instance": "10.142.0.77:9100" 05:48:26 DEBUG }, 05:48:26 DEBUG "value": [ 05:48:26 DEBUG 1712724506, 05:48:26 DEBUG "0.03256666666666833" 05:48:26 DEBUG ] 05:48:26 DEBUG } 05:48:26 DEBUG ] 05:48:26 DEBUG } 05:48:26 DEBUG } 05:48:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724507" 05:48:27 INFO [http_cmd]: http status code OK 05:48:27 DEBUG --- status code --- 05:48:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:27 DEBUG --- http response --- 05:48:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:48:27 DEBUG ----- output ----- 05:48:27 DEBUG { 05:48:27 DEBUG "status": "success", 05:48:27 DEBUG "data": { 05:48:27 DEBUG "resultType": "vector", 05:48:27 DEBUG "result": [ 05:48:27 DEBUG { 05:48:27 DEBUG "metric": { 05:48:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:48:27 DEBUG }, 05:48:27 DEBUG "value": [ 05:48:27 DEBUG 05:48:27 DEBUG 05:48:27 DEBUG 05:48:27 DEBUG [...] 05:48:27 DEBUG 05:48:27 DEBUG 05:48:27 DEBUG 05:48:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:48:27 DEBUG }, 05:48:27 DEBUG "value": [ 05:48:27 DEBUG 1712724507, 05:48:27 DEBUG "0" 05:48:27 DEBUG ] 05:48:27 DEBUG } 05:48:27 DEBUG ] 05:48:27 DEBUG } 05:48:27 DEBUG } 05:48:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712724507" 05:48:27 INFO [http_cmd]: http status code OK 05:48:27 DEBUG --- status code --- 05:48:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:27 DEBUG --- http response --- 05:48:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:48:27 DEBUG ----- output ----- 05:48:27 DEBUG { 05:48:27 DEBUG "status": "success", 05:48:27 DEBUG "data": { 05:48:27 DEBUG "resultType": "vector", 05:48:27 DEBUG "result": [ 05:48:27 DEBUG { 05:48:27 DEBUG "metric": { 05:48:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:48:27 DEBUG }, 05:48:27 DEBUG "value": [ 05:48:27 DEBUG 05:48:27 DEBUG 05:48:27 DEBUG 05:48:27 DEBUG [...] 05:48:27 DEBUG 05:48:27 DEBUG 05:48:27 DEBUG 05:48:27 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:48:27 DEBUG }, 05:48:27 DEBUG "value": [ 05:48:27 DEBUG 1712724507, 05:48:27 DEBUG "4.235738679029822" 05:48:27 DEBUG ] 05:48:27 DEBUG } 05:48:27 DEBUG ] 05:48:27 DEBUG } 05:48:27 DEBUG } 05:48:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712724507" 05:48:27 INFO [http_cmd]: http status code OK 05:48:27 DEBUG --- status code --- 05:48:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:27 DEBUG --- http response --- 05:48:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:48:27 DEBUG ----- output ----- 05:48:27 DEBUG { 05:48:27 DEBUG "status": "success", 05:48:27 DEBUG "data": { 05:48:27 DEBUG "resultType": "vector", 05:48:27 DEBUG "result": [ 05:48:27 DEBUG { 05:48:27 DEBUG "metric": { 05:48:27 DEBUG "pod": "ds-idrepo-0" 05:48:27 DEBUG }, 05:48:27 DEBUG "value": [ 05:48:27 DEBUG 05:48:27 DEBUG 05:48:27 DEBUG 05:48:27 DEBUG [...] 05:48:27 DEBUG 05:48:27 DEBUG 05:48:27 DEBUG 05:48:27 DEBUG "pod": "ds-cts-2" 05:48:27 DEBUG }, 05:48:27 DEBUG "value": [ 05:48:27 DEBUG 1712724507, 05:48:27 DEBUG "0" 05:48:27 DEBUG ] 05:48:27 DEBUG } 05:48:27 DEBUG ] 05:48:27 DEBUG } 05:48:27 DEBUG } 05:48:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724507" 05:48:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724507" 05:48:27 INFO [http_cmd]: http status code OK 05:48:27 DEBUG --- status code --- 05:48:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:27 DEBUG --- http response --- 05:48:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:48:27 DEBUG ----- output ----- 05:48:27 DEBUG { 05:48:27 DEBUG "status": "success", 05:48:27 DEBUG "data": { 05:48:27 DEBUG "resultType": "vector", 05:48:27 DEBUG "result": [ 05:48:27 DEBUG { 05:48:27 DEBUG "metric": { 05:48:27 DEBUG "pod": "ds-idrepo-2" 05:48:27 DEBUG }, 05:48:27 DEBUG "value": [ 05:48:27 DEBUG 05:48:27 DEBUG 05:48:27 DEBUG 05:48:27 DEBUG [...] 05:48:27 DEBUG 05:48:27 DEBUG 05:48:27 DEBUG 05:48:27 DEBUG "pod": "ds-cts-2" 05:48:27 DEBUG }, 05:48:27 DEBUG "value": [ 05:48:27 DEBUG 1712724507, 05:48:27 DEBUG "0" 05:48:27 DEBUG ] 05:48:27 DEBUG } 05:48:27 DEBUG ] 05:48:27 DEBUG } 05:48:27 DEBUG } 05:48:27 INFO [http_cmd]: http status code OK 05:48:27 DEBUG --- status code --- 05:48:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:27 DEBUG --- http response --- 05:48:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:48:27 DEBUG ----- output ----- 05:48:27 DEBUG { 05:48:27 DEBUG "status": "success", 05:48:27 DEBUG "data": { 05:48:27 DEBUG "resultType": "vector", 05:48:27 DEBUG "result": [ 05:48:27 DEBUG { 05:48:27 DEBUG "metric": { 05:48:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:48:27 DEBUG }, 05:48:27 DEBUG "value": [ 05:48:27 DEBUG 05:48:27 DEBUG 05:48:27 DEBUG 05:48:27 DEBUG [...] 05:48:27 DEBUG 05:48:27 DEBUG 05:48:27 DEBUG 05:48:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:48:27 DEBUG }, 05:48:27 DEBUG "value": [ 05:48:27 DEBUG 1712724507, 05:48:27 DEBUG "0" 05:48:27 DEBUG ] 05:48:27 DEBUG } 05:48:27 DEBUG ] 05:48:27 DEBUG } 05:48:27 DEBUG } 05:48:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724508" 05:48:28 INFO [http_cmd]: http status code OK 05:48:28 DEBUG --- status code --- 05:48:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:28 DEBUG --- http response --- 05:48:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:48:28 DEBUG ----- output ----- 05:48:28 DEBUG { 05:48:28 DEBUG "status": "success", 05:48:28 DEBUG "data": { 05:48:28 DEBUG "resultType": "vector", 05:48:28 DEBUG "result": [ 05:48:28 DEBUG { 05:48:28 DEBUG "metric": { 05:48:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:48:28 DEBUG }, 05:48:28 DEBUG "value": [ 05:48:28 DEBUG 05:48:28 DEBUG 05:48:28 DEBUG 05:48:28 DEBUG [...] 05:48:28 DEBUG 05:48:28 DEBUG 05:48:28 DEBUG 05:48:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:48:28 DEBUG }, 05:48:28 DEBUG "value": [ 05:48:28 DEBUG 1712724508, 05:48:28 DEBUG "1448.8333333333333" 05:48:28 DEBUG ] 05:48:28 DEBUG } 05:48:28 DEBUG ] 05:48:28 DEBUG } 05:48:28 DEBUG } 05:48:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712724508" 05:48:28 INFO [http_cmd]: http status code OK 05:48:28 DEBUG --- status code --- 05:48:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:28 DEBUG --- http response --- 05:48:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:48:28 DEBUG ----- output ----- 05:48:28 DEBUG { 05:48:28 DEBUG "status": "success", 05:48:28 DEBUG "data": { 05:48:28 DEBUG "resultType": "vector", 05:48:28 DEBUG "result": [ 05:48:28 DEBUG { 05:48:28 DEBUG "metric": { 05:48:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:48:28 DEBUG }, 05:48:28 DEBUG "value": [ 05:48:28 DEBUG 05:48:28 DEBUG 05:48:28 DEBUG 05:48:28 DEBUG [...] 05:48:28 DEBUG 05:48:28 DEBUG 05:48:28 DEBUG 05:48:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:48:28 DEBUG }, 05:48:28 DEBUG "value": [ 05:48:28 DEBUG 1712724508, 05:48:28 DEBUG "0.043" 05:48:28 DEBUG ] 05:48:28 DEBUG } 05:48:28 DEBUG ] 05:48:28 DEBUG } 05:48:28 DEBUG } 05:48:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724508" 05:48:28 INFO [http_cmd]: http status code OK 05:48:28 DEBUG --- status code --- 05:48:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:28 DEBUG --- http response --- 05:48:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:48:28 DEBUG ----- output ----- 05:48:28 DEBUG { 05:48:28 DEBUG "status": "success", 05:48:28 DEBUG "data": { 05:48:28 DEBUG "resultType": "vector", 05:48:28 DEBUG "result": [ 05:48:28 DEBUG { 05:48:28 DEBUG "metric": { 05:48:28 DEBUG "pod": "ds-idrepo-0" 05:48:28 DEBUG }, 05:48:28 DEBUG "value": [ 05:48:28 DEBUG 05:48:28 DEBUG 05:48:28 DEBUG 05:48:28 DEBUG [...] 05:48:28 DEBUG 05:48:28 DEBUG 05:48:28 DEBUG 05:48:28 DEBUG "pod": "ds-idrepo-1" 05:48:28 DEBUG }, 05:48:28 DEBUG "value": [ 05:48:28 DEBUG 1712724508, 05:48:28 DEBUG "0" 05:48:28 DEBUG ] 05:48:28 DEBUG } 05:48:28 DEBUG ] 05:48:28 DEBUG } 05:48:28 DEBUG } 05:48:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724508" 05:48:28 INFO [http_cmd]: http status code OK 05:48:28 DEBUG --- status code --- 05:48:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:28 DEBUG --- http response --- 05:48:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:48:28 DEBUG ----- output ----- 05:48:28 DEBUG { 05:48:28 DEBUG "status": "success", 05:48:28 DEBUG "data": { 05:48:28 DEBUG "resultType": "vector", 05:48:28 DEBUG "result": [ 05:48:28 DEBUG { 05:48:28 DEBUG "metric": { 05:48:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:48:28 DEBUG }, 05:48:28 DEBUG "value": [ 05:48:28 DEBUG 05:48:28 DEBUG 05:48:28 DEBUG 05:48:28 DEBUG [...] 05:48:28 DEBUG 05:48:28 DEBUG 05:48:28 DEBUG 05:48:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:48:28 DEBUG }, 05:48:28 DEBUG "value": [ 05:48:28 DEBUG 1712724508, 05:48:28 DEBUG "0" 05:48:28 DEBUG ] 05:48:28 DEBUG } 05:48:28 DEBUG ] 05:48:28 DEBUG } 05:48:28 DEBUG } 05:48:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724508" 05:48:28 INFO [http_cmd]: http status code OK 05:48:28 DEBUG --- status code --- 05:48:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:28 DEBUG --- http response --- 05:48:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:48:28 DEBUG ----- output ----- 05:48:28 DEBUG { 05:48:28 DEBUG "status": "success", 05:48:28 DEBUG "data": { 05:48:28 DEBUG "resultType": "vector", 05:48:28 DEBUG "result": [ 05:48:28 DEBUG { 05:48:28 DEBUG "metric": { 05:48:28 DEBUG "pod": "ds-idrepo-0" 05:48:28 DEBUG }, 05:48:28 DEBUG "value": [ 05:48:28 DEBUG 05:48:28 DEBUG 05:48:28 DEBUG 05:48:28 DEBUG [...] 05:48:28 DEBUG 05:48:28 DEBUG 05:48:28 DEBUG 05:48:28 DEBUG "pod": "ds-idrepo-1" 05:48:28 DEBUG }, 05:48:28 DEBUG "value": [ 05:48:28 DEBUG 1712724508, 05:48:28 DEBUG "0" 05:48:28 DEBUG ] 05:48:28 DEBUG } 05:48:28 DEBUG ] 05:48:28 DEBUG } 05:48:28 DEBUG } 05:48:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712724508" 05:48:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724508" 05:48:28 INFO [http_cmd]: http status code OK 05:48:28 DEBUG --- status code --- 05:48:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:28 DEBUG --- http response --- 05:48:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:48:28 DEBUG ----- output ----- 05:48:28 DEBUG { 05:48:28 DEBUG "status": "success", 05:48:28 DEBUG "data": { 05:48:28 DEBUG "resultType": "vector", 05:48:28 DEBUG "result": [ 05:48:28 DEBUG { 05:48:28 DEBUG "metric": { 05:48:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:48:28 DEBUG }, 05:48:28 DEBUG "value": [ 05:48:28 DEBUG 05:48:28 DEBUG 05:48:28 DEBUG 05:48:28 DEBUG [...] 05:48:28 DEBUG 05:48:28 DEBUG 05:48:28 DEBUG 05:48:28 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:48:28 DEBUG }, 05:48:28 DEBUG "value": [ 05:48:28 DEBUG 1712724508, 05:48:28 DEBUG "0" 05:48:28 DEBUG ] 05:48:28 DEBUG } 05:48:28 DEBUG ] 05:48:28 DEBUG } 05:48:28 DEBUG } 05:48:28 INFO [http_cmd]: http status code OK 05:48:28 DEBUG --- status code --- 05:48:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:28 DEBUG --- http response --- 05:48:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:48:28 DEBUG ----- output ----- 05:48:28 DEBUG { 05:48:28 DEBUG "status": "success", 05:48:28 DEBUG "data": { 05:48:28 DEBUG "resultType": "vector", 05:48:28 DEBUG "result": [ 05:48:28 DEBUG { 05:48:28 DEBUG "metric": { 05:48:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:48:28 DEBUG }, 05:48:28 DEBUG "value": [ 05:48:28 DEBUG 05:48:28 DEBUG 05:48:28 DEBUG 05:48:28 DEBUG [...] 05:48:28 DEBUG 05:48:28 DEBUG 05:48:28 DEBUG 05:48:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:48:28 DEBUG }, 05:48:28 DEBUG "value": [ 05:48:28 DEBUG 1712724508, 05:48:28 DEBUG "2487844864" 05:48:28 DEBUG ] 05:48:28 DEBUG } 05:48:28 DEBUG ] 05:48:28 DEBUG } 05:48:28 DEBUG } 05:48:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724509" 05:48:29 INFO [http_cmd]: http status code OK 05:48:29 DEBUG --- status code --- 05:48:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:29 DEBUG --- http response --- 05:48:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:48:29 DEBUG ----- output ----- 05:48:29 DEBUG { 05:48:29 DEBUG "status": "success", 05:48:29 DEBUG "data": { 05:48:29 DEBUG "resultType": "vector", 05:48:29 DEBUG "result": [ 05:48:29 DEBUG { 05:48:29 DEBUG "metric": { 05:48:29 DEBUG "instance": "10.142.0.25:9100" 05:48:29 DEBUG }, 05:48:29 DEBUG "value": [ 05:48:29 DEBUG 05:48:29 DEBUG 05:48:29 DEBUG 05:48:29 DEBUG [...] 05:48:29 DEBUG 05:48:29 DEBUG 05:48:29 DEBUG 05:48:29 DEBUG "instance": "10.142.0.77:9100" 05:48:29 DEBUG }, 05:48:29 DEBUG "value": [ 05:48:29 DEBUG 1712724509, 05:48:29 DEBUG "34078403.166666664" 05:48:29 DEBUG ] 05:48:29 DEBUG } 05:48:29 DEBUG ] 05:48:29 DEBUG } 05:48:29 DEBUG } 05:48:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724509" 05:48:29 INFO [http_cmd]: http status code OK 05:48:29 DEBUG --- status code --- 05:48:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:29 DEBUG --- http response --- 05:48:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:48:29 DEBUG ----- output ----- 05:48:29 DEBUG { 05:48:29 DEBUG "status": "success", 05:48:29 DEBUG "data": { 05:48:29 DEBUG "resultType": "vector", 05:48:29 DEBUG "result": [ 05:48:29 DEBUG { 05:48:29 DEBUG "metric": { 05:48:29 DEBUG "instance": "10.142.0.25:9100" 05:48:29 DEBUG }, 05:48:29 DEBUG "value": [ 05:48:29 DEBUG 05:48:29 DEBUG 05:48:29 DEBUG 05:48:29 DEBUG [...] 05:48:29 DEBUG 05:48:29 DEBUG 05:48:29 DEBUG 05:48:29 DEBUG "instance": "10.142.0.77:9100" 05:48:29 DEBUG }, 05:48:29 DEBUG "value": [ 05:48:29 DEBUG 1712724509, 05:48:29 DEBUG "0.016333333333332156" 05:48:29 DEBUG ] 05:48:29 DEBUG } 05:48:29 DEBUG ] 05:48:29 DEBUG } 05:48:29 DEBUG } 05:48:44 INFO 05:48:44 INFO [loop_until]: kubectl --namespace=xlou top pods 05:48:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:48:45 INFO [loop_until]: OK (rc = 0) 05:48:45 DEBUG --- stdout --- 05:48:45 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 8578m 8296Mi am-654c4fcbcf-9scrj 8386m 8396Mi am-654c4fcbcf-v5qb5 6928m 8377Mi ds-cts-0 4010m 12549Mi ds-cts-1 4174m 12567Mi ds-cts-2 3745m 12734Mi ds-idrepo-0 9684m 5622Mi ds-idrepo-1 350m 5671Mi ds-idrepo-2 384m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 2m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4174m 1681Mi 05:48:45 DEBUG --- stderr --- 05:48:45 DEBUG 05:48:52 INFO 05:48:52 INFO [loop_until]: kubectl --namespace=xlou top node 05:48:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:48:52 INFO [loop_until]: OK (rc = 0) 05:48:52 DEBUG --- stdout --- 05:48:52 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 93m 0% 2306Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 7817m 49% 9262Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 78m 0% 3258Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13048m 82% 6539Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 9192m 57% 9288Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 9474m 59% 9138Mi 15% gke-xlou-cdm-default-pool-f05840a3-r9xm 77m 0% 852Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 9866m 62% 6286Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 4098m 25% 13157Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 3819m 24% 13363Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 447m 2% 6378Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 4199m 26% 13268Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 434m 2% 6340Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4242m 26% 5341Mi 9% 05:48:52 DEBUG --- stderr --- 05:48:52 DEBUG 05:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712724565" 05:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724565" 05:49:25 ERROR [http_cmd]: ERROR 05:49:25 INFO [http_cmd]: http status code OK 05:49:25 DEBUG --- status code --- 05:49:25 ERROR --- status code --- 05:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:49:25 DEBUG --- http response --- 05:49:25 ERROR --- http response --- 05:49:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:49:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:49:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724565" 05:49:25 INFO [http_cmd]: http status code OK 05:49:25 DEBUG --- status code --- 05:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:25 DEBUG --- http response --- 05:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:49:25 DEBUG ----- output ----- 05:49:25 DEBUG { 05:49:25 DEBUG "status": "success", 05:49:25 DEBUG "data": { 05:49:25 DEBUG "resultType": "vector", 05:49:25 DEBUG "result": [ 05:49:25 DEBUG { 05:49:25 DEBUG "metric": { 05:49:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:49:25 DEBUG }, 05:49:25 DEBUG "value": [ 05:49:25 DEBUG 05:49:25 DEBUG 05:49:25 DEBUG 05:49:25 DEBUG [...] 05:49:25 DEBUG 05:49:25 DEBUG 05:49:25 DEBUG 05:49:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:49:25 DEBUG }, 05:49:25 DEBUG "value": [ 05:49:25 DEBUG 1712724565, 05:49:25 DEBUG "0" 05:49:25 DEBUG ] 05:49:25 DEBUG } 05:49:25 DEBUG ] 05:49:25 DEBUG } 05:49:25 DEBUG } 05:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724565" 05:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724565" 05:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724565" 05:49:25 INFO [http_cmd]: http status code OK 05:49:25 DEBUG --- status code --- 05:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:25 DEBUG --- http response --- 05:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:49:25 DEBUG ----- output ----- 05:49:25 DEBUG { 05:49:25 DEBUG "status": "success", 05:49:25 DEBUG "data": { 05:49:25 DEBUG "resultType": "vector", 05:49:25 DEBUG "result": [ 05:49:25 DEBUG { 05:49:25 DEBUG "metric": { 05:49:25 DEBUG "pod": "ds-idrepo-0" 05:49:25 DEBUG }, 05:49:25 DEBUG "value": [ 05:49:25 DEBUG 05:49:25 DEBUG 05:49:25 DEBUG 05:49:25 DEBUG [...] 05:49:25 DEBUG 05:49:25 DEBUG 05:49:25 DEBUG 05:49:25 DEBUG "pod": "ds-cts-2" 05:49:25 DEBUG }, 05:49:25 DEBUG "value": [ 05:49:25 DEBUG 1712724565, 05:49:25 DEBUG "0" 05:49:25 DEBUG ] 05:49:25 DEBUG } 05:49:25 DEBUG ] 05:49:25 DEBUG } 05:49:25 DEBUG } 05:49:25 INFO [http_cmd]: http status code OK 05:49:25 DEBUG --- status code --- 05:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:25 DEBUG --- http response --- 05:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:49:25 DEBUG ----- output ----- 05:49:25 DEBUG { 05:49:25 DEBUG "status": "success", 05:49:25 DEBUG "data": { 05:49:25 INFO [http_cmd]: http status code OK 05:49:25 DEBUG "resultType": "vector", 05:49:25 DEBUG --- status code --- 05:49:25 DEBUG "result": [ 05:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:25 DEBUG { 05:49:25 DEBUG --- http response --- 05:49:25 DEBUG "metric": { 05:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:49:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:49:25 DEBUG ----- output ----- 05:49:25 DEBUG }, 05:49:25 DEBUG { 05:49:25 DEBUG "value": [ 05:49:25 DEBUG "status": "success", 05:49:25 DEBUG 05:49:25 DEBUG "data": { 05:49:25 DEBUG 05:49:25 DEBUG "resultType": "vector", 05:49:25 DEBUG 05:49:25 DEBUG "result": [ 05:49:25 DEBUG [...] 05:49:25 DEBUG { 05:49:25 DEBUG 05:49:25 DEBUG "metric": { 05:49:25 DEBUG 05:49:25 DEBUG "pod": "ds-idrepo-0" 05:49:25 DEBUG 05:49:25 DEBUG }, 05:49:25 DEBUG "value": [ 05:49:25 DEBUG 05:49:25 DEBUG 05:49:25 DEBUG 05:49:25 DEBUG [...] 05:49:25 DEBUG 05:49:25 DEBUG 05:49:25 DEBUG 05:49:25 DEBUG "pod": "ds-cts-2" 05:49:25 DEBUG }, 05:49:25 DEBUG "value": [ 05:49:25 DEBUG 1712724565, 05:49:25 DEBUG "30" 05:49:25 DEBUG ] 05:49:25 DEBUG } 05:49:25 DEBUG ] 05:49:25 DEBUG } 05:49:25 DEBUG } 05:49:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:49:25 DEBUG }, 05:49:25 DEBUG "value": [ 05:49:25 DEBUG 1712724565, 05:49:25 DEBUG "1444.3333333333333" 05:49:25 DEBUG ] 05:49:25 DEBUG } 05:49:25 DEBUG ] 05:49:25 DEBUG } 05:49:25 DEBUG } 05:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724565" 05:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724565" 05:49:25 INFO [http_cmd]: http status code OK 05:49:25 DEBUG --- status code --- 05:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:25 DEBUG --- http response --- 05:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:49:25 DEBUG ----- output ----- 05:49:25 DEBUG { 05:49:25 DEBUG "status": "success", 05:49:25 DEBUG "data": { 05:49:25 DEBUG "resultType": "vector", 05:49:25 DEBUG "result": [ 05:49:25 DEBUG { 05:49:25 DEBUG "metric": { 05:49:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:49:25 DEBUG }, 05:49:25 DEBUG "value": [ 05:49:25 DEBUG 05:49:25 DEBUG 05:49:25 DEBUG 05:49:25 DEBUG [...] 05:49:25 DEBUG 05:49:25 DEBUG 05:49:25 DEBUG 05:49:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:49:25 DEBUG }, 05:49:25 DEBUG "value": [ 05:49:25 DEBUG 1712724565, 05:49:25 DEBUG "0" 05:49:25 DEBUG ] 05:49:25 DEBUG } 05:49:25 DEBUG ] 05:49:25 DEBUG } 05:49:25 DEBUG } 05:49:25 INFO [http_cmd]: http status code OK 05:49:25 DEBUG --- status code --- 05:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:25 DEBUG --- http response --- 05:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:49:25 DEBUG ----- output ----- 05:49:25 DEBUG { 05:49:25 DEBUG "status": "success", 05:49:25 DEBUG "data": { 05:49:25 DEBUG "resultType": "vector", 05:49:25 DEBUG "result": [ 05:49:25 DEBUG { 05:49:25 DEBUG "metric": { 05:49:25 DEBUG "instance": "10.142.0.25:9100" 05:49:25 DEBUG }, 05:49:25 DEBUG "value": [ 05:49:25 DEBUG 05:49:25 DEBUG 05:49:25 DEBUG 05:49:25 DEBUG [...] 05:49:25 DEBUG 05:49:25 DEBUG 05:49:25 DEBUG 05:49:25 DEBUG "instance": "10.142.0.77:9100" 05:49:25 DEBUG }, 05:49:25 DEBUG "value": [ 05:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724565" 05:49:25 DEBUG 1712724565, 05:49:25 DEBUG "318608546816" 05:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724565" 05:49:25 DEBUG ] 05:49:25 DEBUG } 05:49:25 DEBUG ] 05:49:25 DEBUG } 05:49:25 DEBUG } 05:49:25 INFO [http_cmd]: http status code OK 05:49:25 DEBUG --- status code --- 05:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:25 DEBUG --- http response --- 05:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:49:25 DEBUG ----- output ----- 05:49:25 DEBUG { 05:49:25 DEBUG "status": "success", 05:49:25 DEBUG "data": { 05:49:25 DEBUG "resultType": "vector", 05:49:25 DEBUG "result": [ 05:49:25 DEBUG { 05:49:25 DEBUG "metric": { 05:49:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:49:25 DEBUG }, 05:49:25 DEBUG "value": [ 05:49:25 DEBUG 05:49:25 DEBUG 05:49:25 DEBUG 05:49:25 DEBUG [...] 05:49:25 DEBUG 05:49:25 DEBUG 05:49:25 DEBUG 05:49:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:49:25 DEBUG }, 05:49:25 DEBUG "value": [ 05:49:25 DEBUG 1712724565, 05:49:25 DEBUG "250548.5" 05:49:25 DEBUG ] 05:49:25 DEBUG } 05:49:25 DEBUG ] 05:49:25 DEBUG } 05:49:25 DEBUG } 05:49:25 INFO [http_cmd]: http status code OK 05:49:25 DEBUG --- status code --- 05:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:25 DEBUG --- http response --- 05:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:49:25 DEBUG ----- output ----- 05:49:25 DEBUG { 05:49:25 DEBUG "status": "success", 05:49:25 DEBUG "data": { 05:49:25 DEBUG "resultType": "vector", 05:49:25 DEBUG "result": [ 05:49:25 DEBUG { 05:49:25 DEBUG "metric": { 05:49:25 DEBUG "instance": "10.142.0.25:9100" 05:49:25 DEBUG }, 05:49:25 DEBUG "value": [ 05:49:25 DEBUG 05:49:25 DEBUG 05:49:25 DEBUG 05:49:25 DEBUG [...] 05:49:25 DEBUG 05:49:25 DEBUG 05:49:25 DEBUG 05:49:25 DEBUG "instance": "10.142.0.77:9100" 05:49:25 DEBUG }, 05:49:25 DEBUG "value": [ 05:49:25 DEBUG 1712724565, 05:49:25 DEBUG "40100709.3" 05:49:25 DEBUG ] 05:49:25 DEBUG } 05:49:25 DEBUG ] 05:49:25 DEBUG } 05:49:25 DEBUG } 05:49:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724566" 05:49:26 INFO [http_cmd]: http status code OK 05:49:26 DEBUG --- status code --- 05:49:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:26 DEBUG --- http response --- 05:49:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:49:26 DEBUG ----- output ----- 05:49:26 DEBUG { 05:49:26 DEBUG "status": "success", 05:49:26 DEBUG "data": { 05:49:26 DEBUG "resultType": "vector", 05:49:26 DEBUG "result": [ 05:49:26 DEBUG { 05:49:26 DEBUG "metric": { 05:49:26 DEBUG "pod": "ds-idrepo-0" 05:49:26 DEBUG }, 05:49:26 DEBUG "value": [ 05:49:26 DEBUG 05:49:26 DEBUG 05:49:26 DEBUG 05:49:26 DEBUG [...] 05:49:26 DEBUG 05:49:26 DEBUG 05:49:26 DEBUG 05:49:26 DEBUG "pod": "ds-idrepo-1" 05:49:26 DEBUG }, 05:49:26 DEBUG "value": [ 05:49:26 DEBUG 1712724566, 05:49:26 DEBUG "0" 05:49:26 DEBUG ] 05:49:26 DEBUG } 05:49:26 DEBUG ] 05:49:26 DEBUG } 05:49:26 DEBUG } 05:49:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724566" 05:49:26 INFO [http_cmd]: http status code OK 05:49:26 DEBUG --- status code --- 05:49:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:26 DEBUG --- http response --- 05:49:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:49:26 DEBUG ----- output ----- 05:49:26 DEBUG { 05:49:26 DEBUG "status": "success", 05:49:26 DEBUG "data": { 05:49:26 DEBUG "resultType": "vector", 05:49:26 DEBUG "result": [ 05:49:26 DEBUG { 05:49:26 DEBUG "metric": { 05:49:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:49:26 DEBUG }, 05:49:26 DEBUG "value": [ 05:49:26 DEBUG 05:49:26 DEBUG 05:49:26 DEBUG 05:49:26 DEBUG [...] 05:49:26 DEBUG 05:49:26 DEBUG 05:49:26 DEBUG 05:49:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:49:26 DEBUG }, 05:49:26 DEBUG "value": [ 05:49:26 DEBUG 1712724566, 05:49:26 DEBUG "1440.5" 05:49:26 DEBUG ] 05:49:26 DEBUG } 05:49:26 DEBUG ] 05:49:26 DEBUG } 05:49:26 DEBUG } 05:49:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724566" 05:49:26 INFO [http_cmd]: http status code OK 05:49:26 DEBUG --- status code --- 05:49:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:26 DEBUG --- http response --- 05:49:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:49:26 DEBUG ----- output ----- 05:49:26 DEBUG { 05:49:26 DEBUG "status": "success", 05:49:26 DEBUG "data": { 05:49:26 DEBUG "resultType": "vector", 05:49:26 DEBUG "result": [ 05:49:26 DEBUG { 05:49:26 DEBUG "metric": { 05:49:26 DEBUG "instance": "10.142.0.25:9100" 05:49:26 DEBUG }, 05:49:26 DEBUG "value": [ 05:49:26 DEBUG 05:49:26 DEBUG 05:49:26 DEBUG 05:49:26 DEBUG [...] 05:49:26 DEBUG 05:49:26 DEBUG 05:49:26 DEBUG 05:49:26 DEBUG "instance": "10.142.0.77:9100" 05:49:26 DEBUG }, 05:49:26 DEBUG "value": [ 05:49:26 DEBUG 1712724566, 05:49:26 DEBUG "0.03839999999999778" 05:49:26 DEBUG ] 05:49:26 DEBUG } 05:49:26 DEBUG ] 05:49:26 DEBUG } 05:49:26 DEBUG } 05:49:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724567" 05:49:27 INFO [http_cmd]: http status code OK 05:49:27 DEBUG --- status code --- 05:49:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:27 DEBUG --- http response --- 05:49:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 05:49:27 DEBUG ----- output ----- 05:49:27 DEBUG { 05:49:27 DEBUG "status": "success", 05:49:27 DEBUG "data": { 05:49:27 DEBUG "resultType": "vector", 05:49:27 DEBUG "result": [ 05:49:27 DEBUG { 05:49:27 DEBUG "metric": { 05:49:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:49:27 DEBUG }, 05:49:27 DEBUG "value": [ 05:49:27 DEBUG 05:49:27 DEBUG 05:49:27 DEBUG 05:49:27 DEBUG [...] 05:49:27 DEBUG 05:49:27 DEBUG 05:49:27 DEBUG 05:49:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:49:27 DEBUG }, 05:49:27 DEBUG "value": [ 05:49:27 DEBUG 1712724567, 05:49:27 DEBUG "174885.39325842695" 05:49:27 DEBUG ] 05:49:27 DEBUG } 05:49:27 DEBUG ] 05:49:27 DEBUG } 05:49:27 DEBUG } 05:49:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712724567" 05:49:27 INFO [http_cmd]: http status code OK 05:49:27 DEBUG --- status code --- 05:49:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:27 DEBUG --- http response --- 05:49:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:49:27 DEBUG ----- output ----- 05:49:27 DEBUG { 05:49:27 DEBUG "status": "success", 05:49:27 DEBUG "data": { 05:49:27 DEBUG "resultType": "vector", 05:49:27 DEBUG "result": [ 05:49:27 DEBUG { 05:49:27 DEBUG "metric": { 05:49:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:49:27 DEBUG }, 05:49:27 DEBUG "value": [ 05:49:27 DEBUG 05:49:27 DEBUG 05:49:27 DEBUG 05:49:27 DEBUG [...] 05:49:27 DEBUG 05:49:27 DEBUG 05:49:27 DEBUG 05:49:27 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:49:27 DEBUG }, 05:49:27 DEBUG "value": [ 05:49:27 DEBUG 1712724567, 05:49:27 DEBUG "4.150933850268206" 05:49:27 DEBUG ] 05:49:27 DEBUG } 05:49:27 DEBUG ] 05:49:27 DEBUG } 05:49:27 DEBUG } 05:49:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712724567" 05:49:27 INFO [http_cmd]: http status code OK 05:49:27 DEBUG --- status code --- 05:49:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:27 DEBUG --- http response --- 05:49:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:49:27 DEBUG ----- output ----- 05:49:27 DEBUG { 05:49:27 DEBUG "status": "success", 05:49:27 DEBUG "data": { 05:49:27 DEBUG "resultType": "vector", 05:49:27 DEBUG "result": [ 05:49:27 DEBUG { 05:49:27 DEBUG "metric": { 05:49:27 DEBUG "pod": "ds-idrepo-0" 05:49:27 DEBUG }, 05:49:27 DEBUG "value": [ 05:49:27 DEBUG 05:49:27 DEBUG 05:49:27 DEBUG 05:49:27 DEBUG [...] 05:49:27 DEBUG 05:49:27 DEBUG 05:49:27 DEBUG 05:49:27 DEBUG "pod": "ds-cts-2" 05:49:27 DEBUG }, 05:49:27 DEBUG "value": [ 05:49:27 DEBUG 1712724567, 05:49:27 DEBUG "0" 05:49:27 DEBUG ] 05:49:27 DEBUG } 05:49:27 DEBUG ] 05:49:27 DEBUG } 05:49:27 DEBUG } 05:49:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724568" 05:49:28 INFO [http_cmd]: http status code OK 05:49:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724568" 05:49:28 DEBUG --- status code --- 05:49:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:28 DEBUG --- http response --- 05:49:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:49:28 DEBUG ----- output ----- 05:49:28 DEBUG { 05:49:28 DEBUG "status": "success", 05:49:28 DEBUG "data": { 05:49:28 DEBUG "resultType": "vector", 05:49:28 DEBUG "result": [ 05:49:28 DEBUG { 05:49:28 DEBUG "metric": { 05:49:28 DEBUG "pod": "ds-idrepo-2" 05:49:28 DEBUG }, 05:49:28 DEBUG "value": [ 05:49:28 DEBUG 05:49:28 DEBUG 05:49:28 DEBUG 05:49:28 DEBUG [...] 05:49:28 DEBUG 05:49:28 DEBUG 05:49:28 DEBUG 05:49:28 DEBUG "pod": "ds-cts-2" 05:49:28 DEBUG }, 05:49:28 DEBUG "value": [ 05:49:28 DEBUG 1712724568, 05:49:28 DEBUG "0.00014731333333333334" 05:49:28 DEBUG ] 05:49:28 DEBUG } 05:49:28 DEBUG ] 05:49:28 DEBUG } 05:49:28 DEBUG } 05:49:28 INFO [http_cmd]: http status code OK 05:49:28 DEBUG --- status code --- 05:49:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:28 DEBUG --- http response --- 05:49:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 05:49:28 DEBUG ----- output ----- 05:49:28 DEBUG { 05:49:28 DEBUG "status": "success", 05:49:28 DEBUG "data": { 05:49:28 DEBUG "resultType": "vector", 05:49:28 DEBUG "result": [ 05:49:28 DEBUG { 05:49:28 DEBUG "metric": { 05:49:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:49:28 DEBUG }, 05:49:28 DEBUG "value": [ 05:49:28 DEBUG 05:49:28 DEBUG 05:49:28 DEBUG 05:49:28 DEBUG [...] 05:49:28 DEBUG 05:49:28 DEBUG 05:49:28 DEBUG 05:49:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:49:28 DEBUG }, 05:49:28 DEBUG "value": [ 05:49:28 DEBUG 1712724568, 05:49:28 DEBUG "5.786279868890236" 05:49:28 DEBUG ] 05:49:28 DEBUG } 05:49:28 DEBUG ] 05:49:28 DEBUG } 05:49:28 DEBUG } 05:49:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724568" 05:49:28 INFO [http_cmd]: http status code OK 05:49:28 DEBUG --- status code --- 05:49:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:28 DEBUG --- http response --- 05:49:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:49:28 DEBUG ----- output ----- 05:49:28 DEBUG { 05:49:28 DEBUG "status": "success", 05:49:28 DEBUG "data": { 05:49:28 DEBUG "resultType": "vector", 05:49:28 DEBUG "result": [ 05:49:28 DEBUG { 05:49:28 DEBUG "metric": { 05:49:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:49:28 DEBUG }, 05:49:28 DEBUG "value": [ 05:49:28 DEBUG 05:49:28 DEBUG 05:49:28 DEBUG 05:49:28 DEBUG [...] 05:49:28 DEBUG 05:49:28 DEBUG 05:49:28 DEBUG 05:49:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:49:28 DEBUG }, 05:49:28 DEBUG "value": [ 05:49:28 DEBUG 1712724568, 05:49:28 DEBUG "1440.0666666666666" 05:49:28 DEBUG ] 05:49:28 DEBUG } 05:49:28 DEBUG ] 05:49:28 DEBUG } 05:49:28 DEBUG } 05:49:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712724568" 05:49:28 INFO [http_cmd]: http status code OK 05:49:28 DEBUG --- status code --- 05:49:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:28 DEBUG --- http response --- 05:49:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:49:28 DEBUG ----- output ----- 05:49:28 DEBUG { 05:49:28 DEBUG "status": "success", 05:49:28 DEBUG "data": { 05:49:28 DEBUG "resultType": "vector", 05:49:28 DEBUG "result": [ 05:49:28 DEBUG { 05:49:28 DEBUG "metric": { 05:49:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:49:28 DEBUG }, 05:49:28 DEBUG "value": [ 05:49:28 DEBUG 05:49:28 DEBUG 05:49:28 DEBUG 05:49:28 DEBUG [...] 05:49:28 DEBUG 05:49:28 DEBUG 05:49:28 DEBUG 05:49:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:49:28 DEBUG }, 05:49:28 DEBUG "value": [ 05:49:28 DEBUG 1712724568, 05:49:28 DEBUG "0.043" 05:49:28 DEBUG ] 05:49:28 DEBUG } 05:49:28 DEBUG ] 05:49:28 DEBUG } 05:49:28 DEBUG } 05:49:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724568" 05:49:28 INFO [http_cmd]: http status code OK 05:49:28 DEBUG --- status code --- 05:49:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:28 DEBUG --- http response --- 05:49:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:49:28 DEBUG ----- output ----- 05:49:28 DEBUG { 05:49:28 DEBUG "status": "success", 05:49:28 DEBUG "data": { 05:49:28 DEBUG "resultType": "vector", 05:49:28 DEBUG "result": [ 05:49:28 DEBUG { 05:49:28 DEBUG "metric": { 05:49:28 DEBUG "pod": "ds-idrepo-0" 05:49:28 DEBUG }, 05:49:28 DEBUG "value": [ 05:49:28 DEBUG 05:49:28 DEBUG 05:49:28 DEBUG 05:49:28 DEBUG [...] 05:49:28 DEBUG 05:49:28 DEBUG 05:49:28 DEBUG 05:49:28 DEBUG "pod": "ds-idrepo-1" 05:49:28 DEBUG }, 05:49:28 DEBUG "value": [ 05:49:28 DEBUG 1712724568, 05:49:28 DEBUG "0" 05:49:28 DEBUG ] 05:49:28 DEBUG } 05:49:28 DEBUG ] 05:49:28 DEBUG } 05:49:28 DEBUG } 05:49:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724568" 05:49:28 INFO [http_cmd]: http status code OK 05:49:28 DEBUG --- status code --- 05:49:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:28 DEBUG --- http response --- 05:49:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:49:28 DEBUG ----- output ----- 05:49:28 DEBUG { 05:49:28 DEBUG "status": "success", 05:49:28 DEBUG "data": { 05:49:28 DEBUG "resultType": "vector", 05:49:28 DEBUG "result": [ 05:49:28 DEBUG { 05:49:28 DEBUG "metric": { 05:49:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:49:28 DEBUG }, 05:49:28 DEBUG "value": [ 05:49:28 DEBUG 05:49:28 DEBUG 05:49:28 DEBUG 05:49:28 DEBUG [...] 05:49:28 DEBUG 05:49:28 DEBUG 05:49:28 DEBUG 05:49:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:49:28 DEBUG }, 05:49:28 DEBUG "value": [ 05:49:28 DEBUG 1712724568, 05:49:28 DEBUG "814.5" 05:49:28 DEBUG ] 05:49:28 DEBUG } 05:49:28 DEBUG ] 05:49:28 DEBUG } 05:49:28 DEBUG } 05:49:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724568" 05:49:28 INFO [http_cmd]: http status code OK 05:49:28 DEBUG --- status code --- 05:49:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:28 DEBUG --- http response --- 05:49:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:49:28 DEBUG ----- output ----- 05:49:28 DEBUG { 05:49:28 DEBUG "status": "success", 05:49:28 DEBUG "data": { 05:49:28 DEBUG "resultType": "vector", 05:49:28 DEBUG "result": [ 05:49:28 DEBUG { 05:49:28 DEBUG "metric": { 05:49:28 DEBUG "pod": "ds-idrepo-0" 05:49:28 DEBUG }, 05:49:28 DEBUG "value": [ 05:49:28 DEBUG 05:49:28 DEBUG 05:49:28 DEBUG 05:49:28 DEBUG [...] 05:49:28 DEBUG 05:49:28 DEBUG 05:49:28 DEBUG 05:49:28 DEBUG "pod": "ds-idrepo-1" 05:49:28 DEBUG }, 05:49:28 DEBUG "value": [ 05:49:28 DEBUG 1712724568, 05:49:28 DEBUG "0" 05:49:28 DEBUG ] 05:49:28 DEBUG } 05:49:28 DEBUG ] 05:49:28 DEBUG } 05:49:28 DEBUG } 05:49:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724569" 05:49:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712724569" 05:49:29 INFO [http_cmd]: http status code OK 05:49:29 DEBUG --- status code --- 05:49:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:29 DEBUG --- http response --- 05:49:29 INFO [http_cmd]: http status code OK 05:49:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:49:29 DEBUG ----- output ----- 05:49:29 DEBUG { 05:49:29 DEBUG "status": "success", 05:49:29 DEBUG --- status code --- 05:49:29 DEBUG "data": { 05:49:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:29 DEBUG "resultType": "vector", 05:49:29 DEBUG --- http response --- 05:49:29 DEBUG "result": [ 05:49:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:49:29 DEBUG ----- output ----- 05:49:29 DEBUG { 05:49:29 DEBUG "status": "success", 05:49:29 DEBUG "data": { 05:49:29 DEBUG "resultType": "vector", 05:49:29 DEBUG "result": [ 05:49:29 DEBUG { 05:49:29 DEBUG "metric": { 05:49:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:49:29 DEBUG }, 05:49:29 DEBUG "value": [ 05:49:29 DEBUG 05:49:29 DEBUG 05:49:29 DEBUG 05:49:29 DEBUG [...] 05:49:29 DEBUG 05:49:29 DEBUG { 05:49:29 DEBUG 05:49:29 DEBUG "metric": { 05:49:29 DEBUG 05:49:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:49:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:49:29 DEBUG }, 05:49:29 DEBUG }, 05:49:29 DEBUG "value": [ 05:49:29 DEBUG 1712724569, 05:49:29 DEBUG "2486259712" 05:49:29 DEBUG ] 05:49:29 DEBUG } 05:49:29 DEBUG ] 05:49:29 DEBUG } 05:49:29 DEBUG } 05:49:29 DEBUG "value": [ 05:49:29 DEBUG 05:49:29 DEBUG 05:49:29 DEBUG 05:49:29 DEBUG [...] 05:49:29 DEBUG 05:49:29 DEBUG 05:49:29 DEBUG 05:49:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:49:29 DEBUG }, 05:49:29 DEBUG "value": [ 05:49:29 DEBUG 1712724569, 05:49:29 DEBUG "0" 05:49:29 DEBUG ] 05:49:29 DEBUG } 05:49:29 DEBUG ] 05:49:29 DEBUG } 05:49:29 DEBUG } 05:49:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724569" 05:49:29 INFO [http_cmd]: http status code OK 05:49:29 DEBUG --- status code --- 05:49:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:29 DEBUG --- http response --- 05:49:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:49:29 DEBUG ----- output ----- 05:49:29 DEBUG { 05:49:29 DEBUG "status": "success", 05:49:29 DEBUG "data": { 05:49:29 DEBUG "resultType": "vector", 05:49:29 DEBUG "result": [ 05:49:29 DEBUG { 05:49:29 DEBUG "metric": { 05:49:29 DEBUG "instance": "10.142.0.25:9100" 05:49:29 DEBUG }, 05:49:29 DEBUG "value": [ 05:49:29 DEBUG 05:49:29 DEBUG 05:49:29 DEBUG 05:49:29 DEBUG [...] 05:49:29 DEBUG 05:49:29 DEBUG 05:49:29 DEBUG 05:49:29 DEBUG "instance": "10.142.0.77:9100" 05:49:29 DEBUG }, 05:49:29 DEBUG "value": [ 05:49:29 DEBUG 1712724569, 05:49:29 DEBUG "40102061.733333334" 05:49:29 DEBUG ] 05:49:29 DEBUG } 05:49:29 DEBUG ] 05:49:29 DEBUG } 05:49:29 DEBUG } 05:49:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724569" 05:49:29 INFO [http_cmd]: http status code OK 05:49:29 DEBUG --- status code --- 05:49:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:29 DEBUG --- http response --- 05:49:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:49:29 DEBUG ----- output ----- 05:49:29 DEBUG { 05:49:29 DEBUG "status": "success", 05:49:29 DEBUG "data": { 05:49:29 DEBUG "resultType": "vector", 05:49:29 DEBUG "result": [ 05:49:29 DEBUG { 05:49:29 DEBUG "metric": { 05:49:29 DEBUG "instance": "10.142.0.25:9100" 05:49:29 DEBUG }, 05:49:29 DEBUG "value": [ 05:49:29 DEBUG 05:49:29 DEBUG 05:49:29 DEBUG 05:49:29 DEBUG [...] 05:49:29 DEBUG 05:49:29 DEBUG 05:49:29 DEBUG 05:49:29 DEBUG "instance": "10.142.0.77:9100" 05:49:29 DEBUG }, 05:49:29 DEBUG "value": [ 05:49:29 DEBUG 1712724569, 05:49:29 DEBUG "0.017000000000000112" 05:49:29 DEBUG ] 05:49:29 DEBUG } 05:49:29 DEBUG ] 05:49:29 DEBUG } 05:49:29 DEBUG } 05:49:45 INFO 05:49:45 INFO [loop_until]: kubectl --namespace=xlou top pods 05:49:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:49:45 INFO [loop_until]: OK (rc = 0) 05:49:45 DEBUG --- stdout --- 05:49:45 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10019m 8312Mi am-654c4fcbcf-9scrj 9568m 8410Mi am-654c4fcbcf-v5qb5 8334m 8392Mi ds-cts-0 8124m 13208Mi ds-cts-1 8263m 13269Mi ds-cts-2 8122m 13356Mi ds-idrepo-0 10845m 5622Mi ds-idrepo-1 410m 5671Mi ds-idrepo-2 429m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 7m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4185m 1684Mi 05:49:45 DEBUG --- stderr --- 05:49:45 DEBUG 05:49:52 INFO 05:49:52 INFO [loop_until]: kubectl --namespace=xlou top node 05:49:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:49:52 INFO [loop_until]: OK (rc = 0) 05:49:52 DEBUG --- stdout --- 05:49:52 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 95m 0% 2302Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9237m 58% 9275Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 88m 0% 3267Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 12739m 80% 6512Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10827m 68% 9295Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11317m 71% 9147Mi 15% gke-xlou-cdm-default-pool-f05840a3-r9xm 79m 0% 857Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10852m 68% 6288Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8337m 52% 13807Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 8048m 50% 13088Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 467m 2% 6377Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8474m 53% 13944Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 468m 2% 6340Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4351m 27% 5347Mi 9% 05:49:52 DEBUG --- stderr --- 05:49:52 DEBUG 05:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724625" 05:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712724625" 05:50:25 INFO [http_cmd]: http status code OK 05:50:25 ERROR [http_cmd]: ERROR 05:50:25 DEBUG --- status code --- 05:50:25 ERROR --- status code --- 05:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:50:25 DEBUG --- http response --- 05:50:25 ERROR --- http response --- 05:50:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:50:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:50:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724625" 05:50:25 INFO [http_cmd]: http status code OK 05:50:25 DEBUG --- status code --- 05:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:25 DEBUG --- http response --- 05:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:50:25 DEBUG ----- output ----- 05:50:25 DEBUG { 05:50:25 DEBUG "status": "success", 05:50:25 DEBUG "data": { 05:50:25 DEBUG "resultType": "vector", 05:50:25 DEBUG "result": [ 05:50:25 DEBUG { 05:50:25 DEBUG "metric": { 05:50:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:50:25 DEBUG }, 05:50:25 DEBUG "value": [ 05:50:25 DEBUG 05:50:25 DEBUG 05:50:25 DEBUG 05:50:25 DEBUG [...] 05:50:25 DEBUG 05:50:25 DEBUG 05:50:25 DEBUG 05:50:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:50:25 DEBUG }, 05:50:25 DEBUG "value": [ 05:50:25 DEBUG 1712724625, 05:50:25 DEBUG "0" 05:50:25 DEBUG ] 05:50:25 DEBUG } 05:50:25 DEBUG ] 05:50:25 DEBUG } 05:50:25 DEBUG } 05:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724625" 05:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724625" 05:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724625" 05:50:25 INFO [http_cmd]: http status code OK 05:50:25 INFO [http_cmd]: http status code OK 05:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724625" 05:50:25 INFO [http_cmd]: http status code OK 05:50:25 DEBUG --- status code --- 05:50:25 DEBUG --- status code --- 05:50:25 DEBUG --- status code --- 05:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:25 DEBUG --- http response --- 05:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:50:25 DEBUG --- http response --- 05:50:25 DEBUG ----- output ----- 05:50:25 DEBUG --- http response --- 05:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:50:25 DEBUG { 05:50:25 DEBUG "status": "success", 05:50:25 DEBUG ----- output ----- 05:50:25 DEBUG "data": { 05:50:25 DEBUG "resultType": "vector", 05:50:25 DEBUG "result": [ 05:50:25 DEBUG { 05:50:25 DEBUG { 05:50:25 DEBUG "metric": { 05:50:25 DEBUG "pod": "ds-idrepo-0" 05:50:25 DEBUG "status": "success", 05:50:25 DEBUG }, 05:50:25 DEBUG "data": { 05:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:50:25 DEBUG ----- output ----- 05:50:25 DEBUG "resultType": "vector", 05:50:25 DEBUG { 05:50:25 DEBUG "status": "success", 05:50:25 DEBUG "result": [ 05:50:25 DEBUG "value": [ 05:50:25 DEBUG 05:50:25 DEBUG { 05:50:25 DEBUG 05:50:25 DEBUG 05:50:25 DEBUG "metric": { 05:50:25 DEBUG [...] 05:50:25 DEBUG 05:50:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:50:25 DEBUG 05:50:25 DEBUG 05:50:25 DEBUG "pod": "ds-cts-2" 05:50:25 DEBUG }, 05:50:25 DEBUG "data": { 05:50:25 DEBUG "resultType": "vector", 05:50:25 DEBUG "value": [ 05:50:25 DEBUG "result": [ 05:50:25 DEBUG }, 05:50:25 DEBUG 05:50:25 DEBUG { 05:50:25 DEBUG "value": [ 05:50:25 DEBUG 05:50:25 DEBUG "metric": { 05:50:25 DEBUG 1712724625, 05:50:25 DEBUG 05:50:25 DEBUG [...] 05:50:25 DEBUG "29.966666666666665" 05:50:25 DEBUG 05:50:25 DEBUG "pod": "ds-idrepo-0" 05:50:25 DEBUG ] 05:50:25 DEBUG 05:50:25 DEBUG }, 05:50:25 DEBUG } 05:50:25 DEBUG 05:50:25 DEBUG "value": [ 05:50:25 DEBUG 05:50:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:50:25 DEBUG ] 05:50:25 DEBUG 05:50:25 DEBUG 05:50:25 DEBUG } 05:50:25 DEBUG } 05:50:25 DEBUG [...] 05:50:25 DEBUG 05:50:25 DEBUG 05:50:25 DEBUG 05:50:25 DEBUG "pod": "ds-cts-2" 05:50:25 DEBUG }, 05:50:25 DEBUG "value": [ 05:50:25 DEBUG 1712724625, 05:50:25 DEBUG "0" 05:50:25 DEBUG ] 05:50:25 DEBUG } 05:50:25 DEBUG ] 05:50:25 DEBUG } 05:50:25 DEBUG } 05:50:25 DEBUG }, 05:50:25 DEBUG "value": [ 05:50:25 DEBUG 1712724625, 05:50:25 DEBUG "1458.6333333333334" 05:50:25 DEBUG ] 05:50:25 DEBUG } 05:50:25 DEBUG ] 05:50:25 DEBUG } 05:50:25 DEBUG } 05:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724625" 05:50:25 INFO [http_cmd]: http status code OK 05:50:25 DEBUG --- status code --- 05:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:25 DEBUG --- http response --- 05:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:50:25 DEBUG ----- output ----- 05:50:25 DEBUG { 05:50:25 DEBUG "status": "success", 05:50:25 DEBUG "data": { 05:50:25 DEBUG "resultType": "vector", 05:50:25 DEBUG "result": [ 05:50:25 DEBUG { 05:50:25 DEBUG "metric": { 05:50:25 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:50:25 DEBUG }, 05:50:25 DEBUG "value": [ 05:50:25 DEBUG 05:50:25 DEBUG 05:50:25 DEBUG 05:50:25 DEBUG [...] 05:50:25 DEBUG 05:50:25 DEBUG 05:50:25 DEBUG 05:50:25 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:50:25 DEBUG }, 05:50:25 DEBUG "value": [ 05:50:25 DEBUG 1712724625, 05:50:25 DEBUG "0" 05:50:25 DEBUG ] 05:50:25 DEBUG } 05:50:25 DEBUG ] 05:50:25 DEBUG } 05:50:25 DEBUG } 05:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724625" 05:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724625" 05:50:25 INFO [http_cmd]: http status code OK 05:50:25 DEBUG --- status code --- 05:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:25 DEBUG --- http response --- 05:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:50:25 DEBUG ----- output ----- 05:50:25 DEBUG { 05:50:25 DEBUG "status": "success", 05:50:25 DEBUG "data": { 05:50:25 DEBUG "resultType": "vector", 05:50:25 DEBUG "result": [ 05:50:25 DEBUG { 05:50:25 DEBUG "metric": { 05:50:25 DEBUG "instance": "10.142.0.25:9100" 05:50:25 DEBUG }, 05:50:25 DEBUG "value": [ 05:50:25 DEBUG 05:50:25 DEBUG 05:50:25 DEBUG 05:50:25 DEBUG [...] 05:50:25 DEBUG 05:50:25 DEBUG 05:50:25 DEBUG 05:50:25 DEBUG "instance": "10.142.0.77:9100" 05:50:25 DEBUG }, 05:50:25 DEBUG "value": [ 05:50:25 DEBUG 1712724625, 05:50:25 DEBUG "320226024448" 05:50:25 DEBUG ] 05:50:25 DEBUG } 05:50:25 DEBUG ] 05:50:25 DEBUG } 05:50:25 DEBUG } 05:50:25 INFO [http_cmd]: http status code OK 05:50:25 DEBUG --- status code --- 05:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:25 INFO [http_cmd]: http status code OK 05:50:25 DEBUG --- status code --- 05:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:25 DEBUG --- http response --- 05:50:25 DEBUG --- http response --- 05:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:50:25 DEBUG ----- output ----- 05:50:25 DEBUG ----- output ----- 05:50:25 DEBUG { 05:50:25 DEBUG { 05:50:25 DEBUG "status": "success", 05:50:25 DEBUG "status": "success", 05:50:25 DEBUG "data": { 05:50:25 DEBUG "data": { 05:50:25 DEBUG "resultType": "vector", 05:50:25 DEBUG "resultType": "vector", 05:50:25 DEBUG "result": [ 05:50:25 DEBUG "result": [ 05:50:25 DEBUG { 05:50:25 DEBUG "metric": { 05:50:25 DEBUG { 05:50:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:50:25 DEBUG "metric": { 05:50:25 DEBUG }, 05:50:25 DEBUG "instance": "10.142.0.25:9100" 05:50:25 DEBUG "value": [ 05:50:25 DEBUG }, 05:50:25 DEBUG 05:50:25 DEBUG "value": [ 05:50:25 DEBUG 05:50:25 DEBUG 05:50:25 DEBUG 05:50:25 DEBUG 05:50:25 DEBUG [...] 05:50:25 DEBUG 05:50:25 DEBUG 05:50:25 DEBUG [...] 05:50:25 DEBUG 05:50:25 DEBUG 05:50:25 DEBUG 05:50:25 DEBUG 05:50:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:50:25 DEBUG 05:50:25 DEBUG }, 05:50:25 DEBUG "instance": "10.142.0.77:9100" 05:50:25 DEBUG "value": [ 05:50:25 DEBUG }, 05:50:25 DEBUG 1712724625, 05:50:25 DEBUG "value": [ 05:50:25 DEBUG 1712724625, 05:50:25 DEBUG "259748" 05:50:25 DEBUG "42825451.266666666" 05:50:25 DEBUG ] 05:50:25 DEBUG ] 05:50:25 DEBUG } 05:50:25 DEBUG } 05:50:25 DEBUG ] 05:50:25 DEBUG ] 05:50:25 DEBUG } 05:50:25 DEBUG } 05:50:25 DEBUG } 05:50:25 DEBUG } 05:50:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724626" 05:50:26 INFO [http_cmd]: http status code OK 05:50:26 DEBUG --- status code --- 05:50:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:26 DEBUG --- http response --- 05:50:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:50:26 DEBUG ----- output ----- 05:50:26 DEBUG { 05:50:26 DEBUG "status": "success", 05:50:26 DEBUG "data": { 05:50:26 DEBUG "resultType": "vector", 05:50:26 DEBUG "result": [ 05:50:26 DEBUG { 05:50:26 DEBUG "metric": { 05:50:26 DEBUG "pod": "ds-idrepo-0" 05:50:26 DEBUG }, 05:50:26 DEBUG "value": [ 05:50:26 DEBUG 05:50:26 DEBUG 05:50:26 DEBUG 05:50:26 DEBUG [...] 05:50:26 DEBUG 05:50:26 DEBUG 05:50:26 DEBUG 05:50:26 DEBUG "pod": "ds-idrepo-1" 05:50:26 DEBUG }, 05:50:26 DEBUG "value": [ 05:50:26 DEBUG 1712724626, 05:50:26 DEBUG "0" 05:50:26 DEBUG ] 05:50:26 DEBUG } 05:50:26 DEBUG ] 05:50:26 DEBUG } 05:50:26 DEBUG } 05:50:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724626" 05:50:26 INFO [http_cmd]: http status code OK 05:50:26 DEBUG --- status code --- 05:50:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:26 DEBUG --- http response --- 05:50:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:50:26 DEBUG ----- output ----- 05:50:26 DEBUG { 05:50:26 DEBUG "status": "success", 05:50:26 DEBUG "data": { 05:50:26 DEBUG "resultType": "vector", 05:50:26 DEBUG "result": [ 05:50:26 DEBUG { 05:50:26 DEBUG "metric": { 05:50:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:50:26 DEBUG }, 05:50:26 DEBUG "value": [ 05:50:26 DEBUG 05:50:26 DEBUG 05:50:26 DEBUG 05:50:26 DEBUG [...] 05:50:26 DEBUG 05:50:26 DEBUG 05:50:26 DEBUG 05:50:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:50:26 DEBUG }, 05:50:26 DEBUG "value": [ 05:50:26 DEBUG 1712724626, 05:50:26 DEBUG "1445.7" 05:50:26 DEBUG ] 05:50:26 DEBUG } 05:50:26 DEBUG ] 05:50:26 DEBUG } 05:50:26 DEBUG } 05:50:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724626" 05:50:26 INFO [http_cmd]: http status code OK 05:50:26 DEBUG --- status code --- 05:50:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:26 DEBUG --- http response --- 05:50:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:50:26 DEBUG ----- output ----- 05:50:26 DEBUG { 05:50:26 DEBUG "status": "success", 05:50:26 DEBUG "data": { 05:50:26 DEBUG "resultType": "vector", 05:50:26 DEBUG "result": [ 05:50:26 DEBUG { 05:50:26 DEBUG "metric": { 05:50:26 DEBUG "instance": "10.142.0.25:9100" 05:50:26 DEBUG }, 05:50:26 DEBUG "value": [ 05:50:26 DEBUG 05:50:26 DEBUG 05:50:26 DEBUG 05:50:26 DEBUG [...] 05:50:26 DEBUG 05:50:26 DEBUG 05:50:26 DEBUG 05:50:26 DEBUG "instance": "10.142.0.77:9100" 05:50:26 DEBUG }, 05:50:26 DEBUG "value": [ 05:50:26 DEBUG 1712724626, 05:50:26 DEBUG "0.15679999999999872" 05:50:26 DEBUG ] 05:50:26 DEBUG } 05:50:26 DEBUG ] 05:50:26 DEBUG } 05:50:26 DEBUG } 05:50:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724627" 05:50:27 INFO [http_cmd]: http status code OK 05:50:27 DEBUG --- status code --- 05:50:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:27 DEBUG --- http response --- 05:50:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:50:27 DEBUG ----- output ----- 05:50:27 DEBUG { 05:50:27 DEBUG "status": "success", 05:50:27 DEBUG "data": { 05:50:27 DEBUG "resultType": "vector", 05:50:27 DEBUG "result": [ 05:50:27 DEBUG { 05:50:27 DEBUG "metric": { 05:50:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:50:27 DEBUG }, 05:50:27 DEBUG "value": [ 05:50:27 DEBUG 05:50:27 DEBUG 05:50:27 DEBUG 05:50:27 DEBUG [...] 05:50:27 DEBUG 05:50:27 DEBUG 05:50:27 DEBUG 05:50:27 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:50:27 DEBUG }, 05:50:27 DEBUG "value": [ 05:50:27 DEBUG 1712724627, 05:50:27 DEBUG "4123234.925886283" 05:50:27 DEBUG ] 05:50:27 DEBUG } 05:50:27 DEBUG ] 05:50:27 DEBUG } 05:50:27 DEBUG } 05:50:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712724627" 05:50:27 INFO [http_cmd]: http status code OK 05:50:27 DEBUG --- status code --- 05:50:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:27 DEBUG --- http response --- 05:50:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:50:27 DEBUG ----- output ----- 05:50:27 DEBUG { 05:50:27 DEBUG "status": "success", 05:50:27 DEBUG "data": { 05:50:27 DEBUG "resultType": "vector", 05:50:27 DEBUG "result": [ 05:50:27 DEBUG { 05:50:27 DEBUG "metric": { 05:50:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:50:27 DEBUG }, 05:50:27 DEBUG "value": [ 05:50:27 DEBUG 05:50:27 DEBUG 05:50:27 DEBUG 05:50:27 DEBUG [...] 05:50:27 DEBUG 05:50:27 DEBUG 05:50:27 DEBUG 05:50:27 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:50:27 DEBUG }, 05:50:27 DEBUG "value": [ 05:50:27 DEBUG 1712724627, 05:50:27 DEBUG "4.23486625180902" 05:50:27 DEBUG ] 05:50:27 DEBUG } 05:50:27 DEBUG ] 05:50:27 DEBUG } 05:50:27 DEBUG } 05:50:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712724627" 05:50:27 INFO [http_cmd]: http status code OK 05:50:27 DEBUG --- status code --- 05:50:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:27 DEBUG --- http response --- 05:50:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:50:27 DEBUG ----- output ----- 05:50:27 DEBUG { 05:50:27 DEBUG "status": "success", 05:50:27 DEBUG "data": { 05:50:27 DEBUG "resultType": "vector", 05:50:27 DEBUG "result": [ 05:50:27 DEBUG { 05:50:27 DEBUG "metric": { 05:50:27 DEBUG "pod": "ds-idrepo-0" 05:50:27 DEBUG }, 05:50:27 DEBUG "value": [ 05:50:27 DEBUG 05:50:27 DEBUG 05:50:27 DEBUG 05:50:27 DEBUG [...] 05:50:27 DEBUG 05:50:27 DEBUG 05:50:27 DEBUG 05:50:27 DEBUG "pod": "ds-cts-2" 05:50:27 DEBUG }, 05:50:27 DEBUG "value": [ 05:50:27 DEBUG 1712724627, 05:50:27 DEBUG "0" 05:50:27 DEBUG ] 05:50:27 DEBUG } 05:50:27 DEBUG ] 05:50:27 DEBUG } 05:50:27 DEBUG } 05:50:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724628" 05:50:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724628" 05:50:28 INFO [http_cmd]: http status code OK 05:50:28 DEBUG --- status code --- 05:50:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:28 DEBUG --- http response --- 05:50:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:50:28 DEBUG ----- output ----- 05:50:28 DEBUG { 05:50:28 DEBUG "status": "success", 05:50:28 DEBUG "data": { 05:50:28 DEBUG "resultType": "vector", 05:50:28 DEBUG "result": [ 05:50:28 DEBUG { 05:50:28 DEBUG "metric": { 05:50:28 DEBUG "pod": "ds-idrepo-2" 05:50:28 DEBUG }, 05:50:28 DEBUG "value": [ 05:50:28 DEBUG 05:50:28 DEBUG 05:50:28 DEBUG 05:50:28 DEBUG [...] 05:50:28 DEBUG 05:50:28 DEBUG 05:50:28 DEBUG 05:50:28 DEBUG "pod": "ds-cts-2" 05:50:28 DEBUG }, 05:50:28 DEBUG "value": [ 05:50:28 DEBUG 1712724628, 05:50:28 DEBUG "0.0001666666666666667" 05:50:28 DEBUG ] 05:50:28 DEBUG } 05:50:28 DEBUG ] 05:50:28 DEBUG } 05:50:28 DEBUG } 05:50:28 INFO [http_cmd]: http status code OK 05:50:28 DEBUG --- status code --- 05:50:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:28 DEBUG --- http response --- 05:50:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:50:28 DEBUG ----- output ----- 05:50:28 DEBUG { 05:50:28 DEBUG "status": "success", 05:50:28 DEBUG "data": { 05:50:28 DEBUG "resultType": "vector", 05:50:28 DEBUG "result": [ 05:50:28 DEBUG { 05:50:28 DEBUG "metric": { 05:50:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:50:28 DEBUG }, 05:50:28 DEBUG "value": [ 05:50:28 DEBUG 05:50:28 DEBUG 05:50:28 DEBUG 05:50:28 DEBUG [...] 05:50:28 DEBUG 05:50:28 DEBUG 05:50:28 DEBUG 05:50:28 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:50:28 DEBUG }, 05:50:28 DEBUG "value": [ 05:50:28 DEBUG 1712724628, 05:50:28 DEBUG "4.199021461937679" 05:50:28 DEBUG ] 05:50:28 DEBUG } 05:50:28 DEBUG ] 05:50:28 DEBUG } 05:50:28 DEBUG } 05:50:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724628" 05:50:28 INFO [http_cmd]: http status code OK 05:50:28 DEBUG --- status code --- 05:50:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:28 DEBUG --- http response --- 05:50:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:50:28 DEBUG ----- output ----- 05:50:28 DEBUG { 05:50:28 DEBUG "status": "success", 05:50:28 DEBUG "data": { 05:50:28 DEBUG "resultType": "vector", 05:50:28 DEBUG "result": [ 05:50:28 DEBUG { 05:50:28 DEBUG "metric": { 05:50:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:50:28 DEBUG }, 05:50:28 DEBUG "value": [ 05:50:28 DEBUG 05:50:28 DEBUG 05:50:28 DEBUG 05:50:28 DEBUG [...] 05:50:28 DEBUG 05:50:28 DEBUG 05:50:28 DEBUG 05:50:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:50:28 DEBUG }, 05:50:28 DEBUG "value": [ 05:50:28 DEBUG 1712724628, 05:50:28 DEBUG "1446.2666666666667" 05:50:28 DEBUG ] 05:50:28 DEBUG } 05:50:28 DEBUG ] 05:50:28 DEBUG } 05:50:28 DEBUG } 05:50:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712724628" 05:50:28 INFO [http_cmd]: http status code OK 05:50:28 DEBUG --- status code --- 05:50:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:28 DEBUG --- http response --- 05:50:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:50:28 DEBUG ----- output ----- 05:50:28 DEBUG { 05:50:28 DEBUG "status": "success", 05:50:28 DEBUG "data": { 05:50:28 DEBUG "resultType": "vector", 05:50:28 DEBUG "result": [ 05:50:28 DEBUG { 05:50:28 DEBUG "metric": { 05:50:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:50:28 DEBUG }, 05:50:28 DEBUG "value": [ 05:50:28 DEBUG 05:50:28 DEBUG 05:50:28 DEBUG 05:50:28 DEBUG [...] 05:50:28 DEBUG 05:50:28 DEBUG 05:50:28 DEBUG 05:50:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:50:28 DEBUG }, 05:50:28 DEBUG "value": [ 05:50:28 DEBUG 1712724628, 05:50:28 DEBUG "0.0435" 05:50:28 DEBUG ] 05:50:28 DEBUG } 05:50:28 DEBUG ] 05:50:28 DEBUG } 05:50:28 DEBUG } 05:50:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724628" 05:50:28 INFO [http_cmd]: http status code OK 05:50:28 DEBUG --- status code --- 05:50:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:28 DEBUG --- http response --- 05:50:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:50:28 DEBUG ----- output ----- 05:50:28 DEBUG { 05:50:28 DEBUG "status": "success", 05:50:28 DEBUG "data": { 05:50:28 DEBUG "resultType": "vector", 05:50:28 DEBUG "result": [ 05:50:28 DEBUG { 05:50:28 DEBUG "metric": { 05:50:28 DEBUG "pod": "ds-idrepo-0" 05:50:28 DEBUG }, 05:50:28 DEBUG "value": [ 05:50:28 DEBUG 05:50:28 DEBUG 05:50:28 DEBUG 05:50:28 DEBUG [...] 05:50:28 DEBUG 05:50:28 DEBUG 05:50:28 DEBUG 05:50:28 DEBUG "pod": "ds-idrepo-1" 05:50:28 DEBUG }, 05:50:28 DEBUG "value": [ 05:50:28 DEBUG 1712724628, 05:50:28 DEBUG "0" 05:50:28 DEBUG ] 05:50:28 DEBUG } 05:50:28 DEBUG ] 05:50:28 DEBUG } 05:50:28 DEBUG } 05:50:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724628" 05:50:28 INFO [http_cmd]: http status code OK 05:50:28 DEBUG --- status code --- 05:50:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:28 DEBUG --- http response --- 05:50:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:50:28 DEBUG ----- output ----- 05:50:28 DEBUG { 05:50:28 DEBUG "status": "success", 05:50:28 DEBUG "data": { 05:50:28 DEBUG "resultType": "vector", 05:50:28 DEBUG "result": [ 05:50:28 DEBUG { 05:50:28 DEBUG "metric": { 05:50:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:50:28 DEBUG }, 05:50:28 DEBUG "value": [ 05:50:28 DEBUG 05:50:28 DEBUG 05:50:28 DEBUG 05:50:28 DEBUG [...] 05:50:28 DEBUG 05:50:28 DEBUG 05:50:28 DEBUG 05:50:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:50:28 DEBUG }, 05:50:28 DEBUG "value": [ 05:50:28 DEBUG 1712724628, 05:50:28 DEBUG "1436.9333333333334" 05:50:28 DEBUG ] 05:50:28 DEBUG } 05:50:28 DEBUG ] 05:50:28 DEBUG } 05:50:28 DEBUG } 05:50:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724628" 05:50:28 INFO [http_cmd]: http status code OK 05:50:28 DEBUG --- status code --- 05:50:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:28 DEBUG --- http response --- 05:50:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:50:28 DEBUG ----- output ----- 05:50:28 DEBUG { 05:50:28 DEBUG "status": "success", 05:50:28 DEBUG "data": { 05:50:28 DEBUG "resultType": "vector", 05:50:28 DEBUG "result": [ 05:50:28 DEBUG { 05:50:28 DEBUG "metric": { 05:50:28 DEBUG "pod": "ds-idrepo-0" 05:50:28 DEBUG }, 05:50:28 DEBUG "value": [ 05:50:28 DEBUG 05:50:28 DEBUG 05:50:28 DEBUG 05:50:28 DEBUG [...] 05:50:28 DEBUG 05:50:28 DEBUG 05:50:28 DEBUG 05:50:28 DEBUG "pod": "ds-idrepo-1" 05:50:28 DEBUG }, 05:50:28 DEBUG "value": [ 05:50:28 DEBUG 1712724628, 05:50:28 DEBUG "0" 05:50:28 DEBUG ] 05:50:28 DEBUG } 05:50:28 DEBUG ] 05:50:28 DEBUG } 05:50:28 DEBUG } 05:50:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712724629" 05:50:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724629" 05:50:29 INFO [http_cmd]: http status code OK 05:50:29 DEBUG --- status code --- 05:50:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:29 DEBUG --- http response --- 05:50:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:50:29 INFO [http_cmd]: http status code OK 05:50:29 DEBUG --- status code --- 05:50:29 DEBUG ----- output ----- 05:50:29 DEBUG { 05:50:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:29 DEBUG "status": "success", 05:50:29 DEBUG --- http response --- 05:50:29 DEBUG "data": { 05:50:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:50:29 DEBUG "resultType": "vector", 05:50:29 DEBUG ----- output ----- 05:50:29 DEBUG "result": [ 05:50:29 DEBUG { 05:50:29 DEBUG { 05:50:29 DEBUG "metric": { 05:50:29 DEBUG "status": "success", 05:50:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:50:29 DEBUG "data": { 05:50:29 DEBUG }, 05:50:29 DEBUG "resultType": "vector", 05:50:29 DEBUG "value": [ 05:50:29 DEBUG "result": [ 05:50:29 DEBUG 05:50:29 DEBUG { 05:50:29 DEBUG 05:50:29 DEBUG "metric": { 05:50:29 DEBUG 05:50:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:50:29 DEBUG [...] 05:50:29 DEBUG }, 05:50:29 DEBUG 05:50:29 DEBUG "value": [ 05:50:29 DEBUG 05:50:29 DEBUG 05:50:29 DEBUG 05:50:29 DEBUG 05:50:29 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:50:29 DEBUG 05:50:29 DEBUG }, 05:50:29 DEBUG [...] 05:50:29 DEBUG "value": [ 05:50:29 DEBUG 05:50:29 DEBUG 1712724629, 05:50:29 DEBUG 05:50:29 DEBUG "0" 05:50:29 DEBUG ] 05:50:29 DEBUG 05:50:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:50:29 DEBUG }, 05:50:29 DEBUG "value": [ 05:50:29 DEBUG } 05:50:29 DEBUG 1712724629, 05:50:29 DEBUG ] 05:50:29 DEBUG "2488287232" 05:50:29 DEBUG } 05:50:29 DEBUG ] 05:50:29 DEBUG } 05:50:29 DEBUG } 05:50:29 DEBUG ] 05:50:29 DEBUG } 05:50:29 DEBUG } 05:50:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724629" 05:50:29 INFO [http_cmd]: http status code OK 05:50:29 DEBUG --- status code --- 05:50:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:29 DEBUG --- http response --- 05:50:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:50:29 DEBUG ----- output ----- 05:50:29 DEBUG { 05:50:29 DEBUG "status": "success", 05:50:29 DEBUG "data": { 05:50:29 DEBUG "resultType": "vector", 05:50:29 DEBUG "result": [ 05:50:29 DEBUG { 05:50:29 DEBUG "metric": { 05:50:29 DEBUG "instance": "10.142.0.25:9100" 05:50:29 DEBUG }, 05:50:29 DEBUG "value": [ 05:50:29 DEBUG 05:50:29 DEBUG 05:50:29 DEBUG 05:50:29 DEBUG [...] 05:50:29 DEBUG 05:50:29 DEBUG 05:50:29 DEBUG 05:50:29 DEBUG "instance": "10.142.0.77:9100" 05:50:29 DEBUG }, 05:50:29 DEBUG "value": [ 05:50:29 DEBUG 1712724629, 05:50:29 DEBUG "42827131.63333333" 05:50:29 DEBUG ] 05:50:29 DEBUG } 05:50:29 DEBUG ] 05:50:29 DEBUG } 05:50:29 DEBUG } 05:50:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724629" 05:50:29 INFO [http_cmd]: http status code OK 05:50:29 DEBUG --- status code --- 05:50:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:29 DEBUG --- http response --- 05:50:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:50:29 DEBUG ----- output ----- 05:50:29 DEBUG { 05:50:29 DEBUG "status": "success", 05:50:29 DEBUG "data": { 05:50:29 DEBUG "resultType": "vector", 05:50:29 DEBUG "result": [ 05:50:29 DEBUG { 05:50:29 DEBUG "metric": { 05:50:29 DEBUG "instance": "10.142.0.25:9100" 05:50:29 DEBUG }, 05:50:29 DEBUG "value": [ 05:50:29 DEBUG 05:50:29 DEBUG 05:50:29 DEBUG 05:50:29 DEBUG [...] 05:50:29 DEBUG 05:50:29 DEBUG 05:50:29 DEBUG 05:50:29 DEBUG "instance": "10.142.0.77:9100" 05:50:29 DEBUG }, 05:50:29 DEBUG "value": [ 05:50:29 DEBUG 1712724629, 05:50:29 DEBUG "0.045666666666666765" 05:50:29 DEBUG ] 05:50:29 DEBUG } 05:50:29 DEBUG ] 05:50:29 DEBUG } 05:50:29 DEBUG } 05:50:45 INFO 05:50:45 INFO [loop_until]: kubectl --namespace=xlou top pods 05:50:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:50:45 INFO [loop_until]: OK (rc = 0) 05:50:45 DEBUG --- stdout --- 05:50:45 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10098m 8345Mi am-654c4fcbcf-9scrj 9840m 8429Mi am-654c4fcbcf-v5qb5 8305m 8401Mi ds-cts-0 8395m 13355Mi ds-cts-1 8350m 12477Mi ds-cts-2 8052m 13360Mi ds-idrepo-0 10619m 5622Mi ds-idrepo-1 377m 5671Mi ds-idrepo-2 377m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 5Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 5m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4049m 1805Mi 05:50:45 DEBUG --- stderr --- 05:50:45 DEBUG 05:50:52 INFO 05:50:52 INFO [loop_until]: kubectl --namespace=xlou top node 05:50:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:50:52 INFO [loop_until]: OK (rc = 0) 05:50:52 DEBUG --- stdout --- 05:50:52 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 92m 0% 2305Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9267m 58% 9279Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 77m 0% 3251Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13142m 82% 6547Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10841m 68% 9312Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11402m 71% 9169Mi 15% gke-xlou-cdm-default-pool-f05840a3-r9xm 77m 0% 853Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10522m 66% 6291Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8463m 53% 14003Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 7826m 49% 13958Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 431m 2% 6377Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8511m 53% 13060Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 430m 2% 6338Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4124m 25% 5468Mi 9% 05:50:52 DEBUG --- stderr --- 05:50:52 DEBUG 05:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724685" 05:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712724685" 05:51:25 ERROR [http_cmd]: ERROR 05:51:25 INFO [http_cmd]: http status code OK 05:51:25 ERROR --- status code --- 05:51:25 DEBUG --- status code --- 05:51:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:25 ERROR --- http response --- 05:51:25 DEBUG --- http response --- 05:51:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:51:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:51:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724685" 05:51:25 INFO [http_cmd]: http status code OK 05:51:25 DEBUG --- status code --- 05:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:25 DEBUG --- http response --- 05:51:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:51:25 DEBUG ----- output ----- 05:51:25 DEBUG { 05:51:25 DEBUG "status": "success", 05:51:25 DEBUG "data": { 05:51:25 DEBUG "resultType": "vector", 05:51:25 DEBUG "result": [ 05:51:25 DEBUG { 05:51:25 DEBUG "metric": { 05:51:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:51:25 DEBUG }, 05:51:25 DEBUG "value": [ 05:51:25 DEBUG 05:51:25 DEBUG 05:51:25 DEBUG 05:51:25 DEBUG [...] 05:51:25 DEBUG 05:51:25 DEBUG 05:51:25 DEBUG 05:51:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:51:25 DEBUG }, 05:51:25 DEBUG "value": [ 05:51:25 DEBUG 1712724685, 05:51:25 DEBUG "0" 05:51:25 DEBUG ] 05:51:25 DEBUG } 05:51:25 DEBUG ] 05:51:25 DEBUG } 05:51:25 DEBUG } 05:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724685" 05:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724685" 05:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724685" 05:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724685" 05:51:25 INFO [http_cmd]: http status code OK 05:51:25 DEBUG --- status code --- 05:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:25 INFO [http_cmd]: http status code OK 05:51:25 DEBUG --- http response --- 05:51:25 DEBUG --- status code --- 05:51:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:25 DEBUG ----- output ----- 05:51:25 DEBUG --- http response --- 05:51:25 INFO [http_cmd]: http status code OK 05:51:25 DEBUG { 05:51:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:51:25 DEBUG --- status code --- 05:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:25 DEBUG ----- output ----- 05:51:25 DEBUG "status": "success", 05:51:25 DEBUG --- http response --- 05:51:25 DEBUG { 05:51:25 DEBUG "data": { 05:51:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:51:25 DEBUG "status": "success", 05:51:25 DEBUG "data": { 05:51:25 DEBUG ----- output ----- 05:51:25 DEBUG { 05:51:25 DEBUG "resultType": "vector", 05:51:25 DEBUG "resultType": "vector", 05:51:25 DEBUG "status": "success", 05:51:25 DEBUG "result": [ 05:51:25 DEBUG { 05:51:25 DEBUG "metric": { 05:51:25 DEBUG "data": { 05:51:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:51:25 DEBUG }, 05:51:25 DEBUG "resultType": "vector", 05:51:25 DEBUG "value": [ 05:51:25 DEBUG 05:51:25 DEBUG "result": [ 05:51:25 DEBUG 05:51:25 DEBUG 05:51:25 DEBUG [...] 05:51:25 DEBUG 05:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724685" 05:51:25 DEBUG { 05:51:25 DEBUG "result": [ 05:51:25 DEBUG 05:51:25 DEBUG "metric": { 05:51:25 DEBUG { 05:51:25 DEBUG "metric": { 05:51:25 DEBUG "pod": "ds-idrepo-0" 05:51:25 DEBUG }, 05:51:25 DEBUG "pod": "ds-idrepo-0" 05:51:25 DEBUG "value": [ 05:51:25 DEBUG 05:51:25 DEBUG }, 05:51:25 DEBUG "value": [ 05:51:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:51:25 DEBUG }, 05:51:25 DEBUG "value": [ 05:51:25 DEBUG 05:51:25 DEBUG 05:51:25 DEBUG 1712724685, 05:51:25 DEBUG 05:51:25 DEBUG 05:51:25 DEBUG "1433.5333333333333" 05:51:25 DEBUG ] 05:51:25 DEBUG 05:51:25 DEBUG [...] 05:51:25 DEBUG 05:51:25 DEBUG 05:51:25 DEBUG [...] 05:51:25 DEBUG 05:51:25 DEBUG 05:51:25 DEBUG 05:51:25 DEBUG } 05:51:25 DEBUG 05:51:25 DEBUG 05:51:25 DEBUG ] 05:51:25 DEBUG "pod": "ds-cts-2" 05:51:25 DEBUG "pod": "ds-cts-2" 05:51:25 DEBUG }, 05:51:25 DEBUG }, 05:51:25 DEBUG } 05:51:25 DEBUG "value": [ 05:51:25 DEBUG "value": [ 05:51:25 DEBUG } 05:51:25 DEBUG 1712724685, 05:51:25 DEBUG 1712724685, 05:51:25 DEBUG "29.966666666666665" 05:51:25 DEBUG "0" 05:51:25 DEBUG ] 05:51:25 DEBUG ] 05:51:25 DEBUG } 05:51:25 DEBUG } 05:51:25 DEBUG ] 05:51:25 DEBUG ] 05:51:25 DEBUG } 05:51:25 DEBUG } 05:51:25 DEBUG } 05:51:25 DEBUG } 05:51:25 INFO [http_cmd]: http status code OK 05:51:25 DEBUG --- status code --- 05:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724685" 05:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724685" 05:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:26 DEBUG --- http response --- 05:51:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:51:26 DEBUG ----- output ----- 05:51:26 DEBUG { 05:51:26 DEBUG "status": "success", 05:51:26 DEBUG "data": { 05:51:26 DEBUG "resultType": "vector", 05:51:26 DEBUG "result": [ 05:51:26 DEBUG { 05:51:26 DEBUG "metric": { 05:51:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:51:26 DEBUG }, 05:51:26 DEBUG "value": [ 05:51:26 DEBUG 05:51:26 DEBUG 05:51:26 DEBUG 05:51:26 DEBUG [...] 05:51:26 DEBUG 05:51:26 DEBUG 05:51:26 DEBUG 05:51:26 DEBUG "pod": "ds-idrepo-2" 05:51:26 INFO [http_cmd]: http status code OK 05:51:26 DEBUG }, 05:51:26 DEBUG "value": [ 05:51:26 DEBUG 1712724685, 05:51:26 DEBUG "0" 05:51:26 DEBUG ] 05:51:26 DEBUG } 05:51:26 DEBUG ] 05:51:26 DEBUG } 05:51:26 DEBUG } 05:51:26 DEBUG --- status code --- 05:51:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:26 DEBUG --- http response --- 05:51:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:51:26 DEBUG ----- output ----- 05:51:26 DEBUG { 05:51:26 DEBUG "status": "success", 05:51:26 DEBUG "data": { 05:51:26 DEBUG "resultType": "vector", 05:51:26 DEBUG "result": [ 05:51:26 DEBUG { 05:51:26 DEBUG "metric": { 05:51:26 DEBUG "instance": "10.142.0.25:9100" 05:51:26 DEBUG }, 05:51:26 DEBUG "value": [ 05:51:26 DEBUG 05:51:26 DEBUG 05:51:26 DEBUG 05:51:26 DEBUG [...] 05:51:26 DEBUG 05:51:26 DEBUG 05:51:26 DEBUG 05:51:26 DEBUG "instance": "10.142.0.77:9100" 05:51:26 DEBUG }, 05:51:26 DEBUG "value": [ 05:51:26 DEBUG 1712724685, 05:51:26 DEBUG "321927865344" 05:51:26 DEBUG ] 05:51:26 DEBUG } 05:51:26 DEBUG ] 05:51:26 DEBUG } 05:51:26 DEBUG } 05:51:26 INFO [http_cmd]: http status code OK 05:51:26 DEBUG --- status code --- 05:51:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:26 DEBUG --- http response --- 05:51:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:51:26 DEBUG ----- output ----- 05:51:26 DEBUG { 05:51:26 DEBUG "status": "success", 05:51:26 DEBUG "data": { 05:51:26 DEBUG "resultType": "vector", 05:51:26 DEBUG "result": [ 05:51:26 DEBUG { 05:51:26 DEBUG "metric": { 05:51:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:51:26 DEBUG }, 05:51:26 DEBUG "value": [ 05:51:26 DEBUG 05:51:26 DEBUG 05:51:26 DEBUG 05:51:26 DEBUG [...] 05:51:26 DEBUG 05:51:26 DEBUG 05:51:26 DEBUG 05:51:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:51:26 DEBUG }, 05:51:26 DEBUG "value": [ 05:51:26 DEBUG 1712724685, 05:51:26 DEBUG "259486.5" 05:51:26 DEBUG ] 05:51:26 DEBUG } 05:51:26 DEBUG ] 05:51:26 DEBUG } 05:51:26 DEBUG } 05:51:26 INFO [http_cmd]: http status code OK 05:51:26 DEBUG --- status code --- 05:51:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:26 DEBUG --- http response --- 05:51:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:51:26 DEBUG ----- output ----- 05:51:26 DEBUG { 05:51:26 DEBUG "status": "success", 05:51:26 DEBUG "data": { 05:51:26 DEBUG "resultType": "vector", 05:51:26 DEBUG "result": [ 05:51:26 DEBUG { 05:51:26 DEBUG "metric": { 05:51:26 DEBUG "instance": "10.142.0.25:9100" 05:51:26 DEBUG }, 05:51:26 DEBUG "value": [ 05:51:26 DEBUG 05:51:26 DEBUG 05:51:26 DEBUG 05:51:26 DEBUG [...] 05:51:26 DEBUG 05:51:26 DEBUG 05:51:26 DEBUG 05:51:26 DEBUG "instance": "10.142.0.77:9100" 05:51:26 DEBUG }, 05:51:26 DEBUG "value": [ 05:51:26 DEBUG 1712724685, 05:51:26 DEBUG "42557004.06666666" 05:51:26 DEBUG ] 05:51:26 DEBUG } 05:51:26 DEBUG ] 05:51:26 DEBUG } 05:51:26 DEBUG } 05:51:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724686" 05:51:26 INFO [http_cmd]: http status code OK 05:51:26 DEBUG --- status code --- 05:51:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:26 DEBUG --- http response --- 05:51:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:51:26 DEBUG ----- output ----- 05:51:26 DEBUG { 05:51:26 DEBUG "status": "success", 05:51:26 DEBUG "data": { 05:51:26 DEBUG "resultType": "vector", 05:51:26 DEBUG "result": [ 05:51:26 DEBUG { 05:51:26 DEBUG "metric": { 05:51:26 DEBUG "pod": "ds-idrepo-0" 05:51:26 DEBUG }, 05:51:26 DEBUG "value": [ 05:51:26 DEBUG 05:51:26 DEBUG 05:51:26 DEBUG 05:51:26 DEBUG [...] 05:51:26 DEBUG 05:51:26 DEBUG 05:51:26 DEBUG 05:51:26 DEBUG "pod": "ds-idrepo-1" 05:51:26 DEBUG }, 05:51:26 DEBUG "value": [ 05:51:26 DEBUG 1712724686, 05:51:26 DEBUG "0" 05:51:26 DEBUG ] 05:51:26 DEBUG } 05:51:26 DEBUG ] 05:51:26 DEBUG } 05:51:26 DEBUG } 05:51:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724686" 05:51:26 INFO [http_cmd]: http status code OK 05:51:26 DEBUG --- status code --- 05:51:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:26 DEBUG --- http response --- 05:51:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:51:26 DEBUG ----- output ----- 05:51:26 DEBUG { 05:51:26 DEBUG "status": "success", 05:51:26 DEBUG "data": { 05:51:26 DEBUG "resultType": "vector", 05:51:26 DEBUG "result": [ 05:51:26 DEBUG { 05:51:26 DEBUG "metric": { 05:51:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:51:26 DEBUG }, 05:51:26 DEBUG "value": [ 05:51:26 DEBUG 05:51:26 DEBUG 05:51:26 DEBUG 05:51:26 DEBUG [...] 05:51:26 DEBUG 05:51:26 DEBUG 05:51:26 DEBUG 05:51:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:51:26 DEBUG }, 05:51:26 DEBUG "value": [ 05:51:26 DEBUG 1712724686, 05:51:26 DEBUG "1428.4666666666667" 05:51:26 DEBUG ] 05:51:26 DEBUG } 05:51:26 DEBUG ] 05:51:26 DEBUG } 05:51:26 DEBUG } 05:51:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724686" 05:51:26 INFO [http_cmd]: http status code OK 05:51:26 DEBUG --- status code --- 05:51:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:26 DEBUG --- http response --- 05:51:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:51:26 DEBUG ----- output ----- 05:51:26 DEBUG { 05:51:26 DEBUG "status": "success", 05:51:26 DEBUG "data": { 05:51:26 DEBUG "resultType": "vector", 05:51:26 DEBUG "result": [ 05:51:26 DEBUG { 05:51:26 DEBUG "metric": { 05:51:26 DEBUG "instance": "10.142.0.25:9100" 05:51:26 DEBUG }, 05:51:26 DEBUG "value": [ 05:51:26 DEBUG 05:51:26 DEBUG 05:51:26 DEBUG 05:51:26 DEBUG [...] 05:51:26 DEBUG 05:51:26 DEBUG 05:51:26 DEBUG 05:51:26 DEBUG "instance": "10.142.0.77:9100" 05:51:26 DEBUG }, 05:51:26 DEBUG "value": [ 05:51:26 DEBUG 1712724686, 05:51:26 DEBUG "0.14936666666666007" 05:51:26 DEBUG ] 05:51:26 DEBUG } 05:51:26 DEBUG ] 05:51:26 DEBUG } 05:51:26 DEBUG } 05:51:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724687" 05:51:27 INFO [http_cmd]: http status code OK 05:51:27 DEBUG --- status code --- 05:51:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:27 DEBUG --- http response --- 05:51:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:51:27 DEBUG ----- output ----- 05:51:27 DEBUG { 05:51:27 DEBUG "status": "success", 05:51:27 DEBUG "data": { 05:51:27 DEBUG "resultType": "vector", 05:51:27 DEBUG "result": [ 05:51:27 DEBUG { 05:51:27 DEBUG "metric": { 05:51:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:51:27 DEBUG }, 05:51:27 DEBUG "value": [ 05:51:27 DEBUG 05:51:27 DEBUG 05:51:27 DEBUG 05:51:27 DEBUG [...] 05:51:27 DEBUG 05:51:27 DEBUG 05:51:27 DEBUG 05:51:27 DEBUG "pod": "ds-idrepo-2" 05:51:27 DEBUG }, 05:51:27 DEBUG "value": [ 05:51:27 DEBUG 1712724687, 05:51:27 DEBUG "151503564.4389724" 05:51:27 DEBUG ] 05:51:27 DEBUG } 05:51:27 DEBUG ] 05:51:27 DEBUG } 05:51:27 DEBUG } 05:51:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712724687" 05:51:27 INFO [http_cmd]: http status code OK 05:51:27 DEBUG --- status code --- 05:51:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:27 DEBUG --- http response --- 05:51:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:51:27 DEBUG ----- output ----- 05:51:27 DEBUG { 05:51:27 DEBUG "status": "success", 05:51:27 DEBUG "data": { 05:51:27 DEBUG "resultType": "vector", 05:51:27 DEBUG "result": [ 05:51:27 DEBUG { 05:51:27 DEBUG "metric": { 05:51:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:51:27 DEBUG }, 05:51:27 DEBUG "value": [ 05:51:27 DEBUG 05:51:27 DEBUG 05:51:27 DEBUG 05:51:27 DEBUG [...] 05:51:27 DEBUG 05:51:27 DEBUG 05:51:27 DEBUG 05:51:27 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:51:27 DEBUG }, 05:51:27 DEBUG "value": [ 05:51:27 DEBUG 1712724687, 05:51:27 DEBUG "4.146913617953008" 05:51:27 DEBUG ] 05:51:27 DEBUG } 05:51:27 DEBUG ] 05:51:27 DEBUG } 05:51:27 DEBUG } 05:51:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712724687" 05:51:27 INFO [http_cmd]: http status code OK 05:51:27 DEBUG --- status code --- 05:51:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:27 DEBUG --- http response --- 05:51:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:51:27 DEBUG ----- output ----- 05:51:27 DEBUG { 05:51:27 DEBUG "status": "success", 05:51:27 DEBUG "data": { 05:51:27 DEBUG "resultType": "vector", 05:51:27 DEBUG "result": [ 05:51:27 DEBUG { 05:51:27 DEBUG "metric": { 05:51:27 DEBUG "pod": "ds-idrepo-0" 05:51:27 DEBUG }, 05:51:27 DEBUG "value": [ 05:51:27 DEBUG 05:51:27 DEBUG 05:51:27 DEBUG 05:51:27 DEBUG [...] 05:51:27 DEBUG 05:51:27 DEBUG 05:51:27 DEBUG 05:51:27 DEBUG "pod": "ds-cts-2" 05:51:27 DEBUG }, 05:51:27 DEBUG "value": [ 05:51:27 DEBUG 1712724687, 05:51:27 DEBUG "0" 05:51:27 DEBUG ] 05:51:27 DEBUG } 05:51:27 DEBUG ] 05:51:27 DEBUG } 05:51:27 DEBUG } 05:51:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724688" 05:51:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724688" 05:51:28 INFO [http_cmd]: http status code OK 05:51:28 DEBUG --- status code --- 05:51:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:28 DEBUG --- http response --- 05:51:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:51:28 DEBUG ----- output ----- 05:51:28 DEBUG { 05:51:28 DEBUG "status": "success", 05:51:28 DEBUG "data": { 05:51:28 DEBUG "resultType": "vector", 05:51:28 DEBUG "result": [ 05:51:28 DEBUG { 05:51:28 DEBUG "metric": { 05:51:28 DEBUG "pod": "ds-idrepo-2" 05:51:28 DEBUG }, 05:51:28 DEBUG "value": [ 05:51:28 DEBUG 05:51:28 DEBUG 05:51:28 DEBUG 05:51:28 DEBUG [...] 05:51:28 DEBUG 05:51:28 DEBUG 05:51:28 DEBUG 05:51:28 DEBUG "pod": "ds-cts-2" 05:51:28 DEBUG }, 05:51:28 DEBUG "value": [ 05:51:28 DEBUG 1712724688, 05:51:28 DEBUG "0.000033333333333333335" 05:51:28 DEBUG ] 05:51:28 DEBUG } 05:51:28 DEBUG ] 05:51:28 DEBUG } 05:51:28 DEBUG } 05:51:28 INFO [http_cmd]: http status code OK 05:51:28 DEBUG --- status code --- 05:51:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:28 DEBUG --- http response --- 05:51:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:51:28 DEBUG ----- output ----- 05:51:28 DEBUG { 05:51:28 DEBUG "status": "success", 05:51:28 DEBUG "data": { 05:51:28 DEBUG "resultType": "vector", 05:51:28 DEBUG "result": [ 05:51:28 DEBUG { 05:51:28 DEBUG "metric": { 05:51:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:51:28 DEBUG }, 05:51:28 DEBUG "value": [ 05:51:28 DEBUG 05:51:28 DEBUG 05:51:28 DEBUG 05:51:28 DEBUG [...] 05:51:28 DEBUG 05:51:28 DEBUG 05:51:28 DEBUG 05:51:28 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:51:28 DEBUG }, 05:51:28 DEBUG "value": [ 05:51:28 DEBUG 1712724688, 05:51:28 DEBUG "2.8226346108916442" 05:51:28 DEBUG ] 05:51:28 DEBUG } 05:51:28 DEBUG ] 05:51:28 DEBUG } 05:51:28 DEBUG } 05:51:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724688" 05:51:28 INFO [http_cmd]: http status code OK 05:51:28 DEBUG --- status code --- 05:51:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:28 DEBUG --- http response --- 05:51:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:51:28 DEBUG ----- output ----- 05:51:28 DEBUG { 05:51:28 DEBUG "status": "success", 05:51:28 DEBUG "data": { 05:51:28 DEBUG "resultType": "vector", 05:51:28 DEBUG "result": [ 05:51:28 DEBUG { 05:51:28 DEBUG "metric": { 05:51:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:51:28 DEBUG }, 05:51:28 DEBUG "value": [ 05:51:28 DEBUG 05:51:28 DEBUG 05:51:28 DEBUG 05:51:28 DEBUG [...] 05:51:28 DEBUG 05:51:28 DEBUG 05:51:28 DEBUG 05:51:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:51:28 DEBUG }, 05:51:28 DEBUG "value": [ 05:51:28 DEBUG 1712724688, 05:51:28 DEBUG "1427.6" 05:51:28 DEBUG ] 05:51:28 DEBUG } 05:51:28 DEBUG ] 05:51:28 DEBUG } 05:51:28 DEBUG } 05:51:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712724688" 05:51:28 INFO [http_cmd]: http status code OK 05:51:28 DEBUG --- status code --- 05:51:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:28 DEBUG --- http response --- 05:51:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:51:28 DEBUG ----- output ----- 05:51:28 DEBUG { 05:51:28 DEBUG "status": "success", 05:51:28 DEBUG "data": { 05:51:28 DEBUG "resultType": "vector", 05:51:28 DEBUG "result": [ 05:51:28 DEBUG { 05:51:28 DEBUG "metric": { 05:51:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:51:28 DEBUG }, 05:51:28 DEBUG "value": [ 05:51:28 DEBUG 05:51:28 DEBUG 05:51:28 DEBUG 05:51:28 DEBUG [...] 05:51:28 DEBUG 05:51:28 DEBUG 05:51:28 DEBUG 05:51:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:51:28 DEBUG }, 05:51:28 DEBUG "value": [ 05:51:28 DEBUG 1712724688, 05:51:28 DEBUG "0.044" 05:51:28 DEBUG ] 05:51:28 DEBUG } 05:51:28 DEBUG ] 05:51:28 DEBUG } 05:51:28 DEBUG } 05:51:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724688" 05:51:28 INFO [http_cmd]: http status code OK 05:51:28 DEBUG --- status code --- 05:51:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:28 DEBUG --- http response --- 05:51:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:51:28 DEBUG ----- output ----- 05:51:28 DEBUG { 05:51:28 DEBUG "status": "success", 05:51:28 DEBUG "data": { 05:51:28 DEBUG "resultType": "vector", 05:51:28 DEBUG "result": [ 05:51:28 DEBUG { 05:51:28 DEBUG "metric": { 05:51:28 DEBUG "pod": "ds-idrepo-0" 05:51:28 DEBUG }, 05:51:28 DEBUG "value": [ 05:51:28 DEBUG 05:51:28 DEBUG 05:51:28 DEBUG 05:51:28 DEBUG [...] 05:51:28 DEBUG 05:51:28 DEBUG 05:51:28 DEBUG 05:51:28 DEBUG "pod": "ds-idrepo-1" 05:51:28 DEBUG }, 05:51:28 DEBUG "value": [ 05:51:28 DEBUG 1712724688, 05:51:28 DEBUG "0" 05:51:28 DEBUG ] 05:51:28 DEBUG } 05:51:28 DEBUG ] 05:51:28 DEBUG } 05:51:28 DEBUG } 05:51:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724688" 05:51:28 INFO [http_cmd]: http status code OK 05:51:28 DEBUG --- status code --- 05:51:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:28 DEBUG --- http response --- 05:51:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:51:28 DEBUG ----- output ----- 05:51:28 DEBUG { 05:51:28 DEBUG "status": "success", 05:51:28 DEBUG "data": { 05:51:28 DEBUG "resultType": "vector", 05:51:28 DEBUG "result": [ 05:51:28 DEBUG { 05:51:28 DEBUG "metric": { 05:51:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:51:28 DEBUG }, 05:51:28 DEBUG "value": [ 05:51:28 DEBUG 05:51:28 DEBUG 05:51:28 DEBUG 05:51:28 DEBUG [...] 05:51:28 DEBUG 05:51:28 DEBUG 05:51:28 DEBUG 05:51:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:51:28 DEBUG }, 05:51:28 DEBUG "value": [ 05:51:28 DEBUG 1712724688, 05:51:28 DEBUG "1448.8" 05:51:28 DEBUG ] 05:51:28 DEBUG } 05:51:28 DEBUG ] 05:51:28 DEBUG } 05:51:28 DEBUG } 05:51:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724689" 05:51:29 INFO [http_cmd]: http status code OK 05:51:29 DEBUG --- status code --- 05:51:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:29 DEBUG --- http response --- 05:51:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:51:29 DEBUG ----- output ----- 05:51:29 DEBUG { 05:51:29 DEBUG "status": "success", 05:51:29 DEBUG "data": { 05:51:29 DEBUG "resultType": "vector", 05:51:29 DEBUG "result": [ 05:51:29 DEBUG { 05:51:29 DEBUG "metric": { 05:51:29 DEBUG "pod": "ds-idrepo-0" 05:51:29 DEBUG }, 05:51:29 DEBUG "value": [ 05:51:29 DEBUG 05:51:29 DEBUG 05:51:29 DEBUG 05:51:29 DEBUG [...] 05:51:29 DEBUG 05:51:29 DEBUG 05:51:29 DEBUG 05:51:29 DEBUG "pod": "ds-idrepo-1" 05:51:29 DEBUG }, 05:51:29 DEBUG "value": [ 05:51:29 DEBUG 1712724689, 05:51:29 DEBUG "0" 05:51:29 DEBUG ] 05:51:29 DEBUG } 05:51:29 DEBUG ] 05:51:29 DEBUG } 05:51:29 DEBUG } 05:51:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724689" 05:51:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712724689" 05:51:29 INFO [http_cmd]: http status code OK 05:51:29 DEBUG --- status code --- 05:51:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:29 DEBUG --- http response --- 05:51:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:51:29 DEBUG ----- output ----- 05:51:29 DEBUG { 05:51:29 DEBUG "status": "success", 05:51:29 DEBUG "data": { 05:51:29 DEBUG "resultType": "vector", 05:51:29 DEBUG "result": [ 05:51:29 INFO [http_cmd]: http status code OK 05:51:29 DEBUG { 05:51:29 DEBUG --- status code --- 05:51:29 DEBUG "metric": { 05:51:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:51:29 DEBUG --- http response --- 05:51:29 DEBUG }, 05:51:29 DEBUG "value": [ 05:51:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:51:29 DEBUG 05:51:29 DEBUG ----- output ----- 05:51:29 DEBUG 05:51:29 DEBUG { 05:51:29 DEBUG 05:51:29 DEBUG "status": "success", 05:51:29 DEBUG [...] 05:51:29 DEBUG "data": { 05:51:29 DEBUG 05:51:29 DEBUG "resultType": "vector", 05:51:29 DEBUG 05:51:29 DEBUG "result": [ 05:51:29 DEBUG 05:51:29 DEBUG { 05:51:29 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:51:29 DEBUG "metric": { 05:51:29 DEBUG }, 05:51:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:51:29 DEBUG "value": [ 05:51:29 DEBUG }, 05:51:29 DEBUG 1712724689, 05:51:29 DEBUG "value": [ 05:51:29 DEBUG "0" 05:51:29 DEBUG 05:51:29 DEBUG ] 05:51:29 DEBUG 05:51:29 DEBUG } 05:51:29 DEBUG 05:51:29 DEBUG ] 05:51:29 DEBUG [...] 05:51:29 DEBUG } 05:51:29 DEBUG 05:51:29 DEBUG } 05:51:29 DEBUG 05:51:29 DEBUG 05:51:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:51:29 DEBUG }, 05:51:29 DEBUG "value": [ 05:51:29 DEBUG 1712724689, 05:51:29 DEBUG "2485915648" 05:51:29 DEBUG ] 05:51:29 DEBUG } 05:51:29 DEBUG ] 05:51:29 DEBUG } 05:51:29 DEBUG } 05:51:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724689" 05:51:29 INFO [http_cmd]: http status code OK 05:51:29 DEBUG --- status code --- 05:51:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:29 DEBUG --- http response --- 05:51:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:51:29 DEBUG ----- output ----- 05:51:29 DEBUG { 05:51:29 DEBUG "status": "success", 05:51:29 DEBUG "data": { 05:51:29 DEBUG "resultType": "vector", 05:51:29 DEBUG "result": [ 05:51:29 DEBUG { 05:51:29 DEBUG "metric": { 05:51:29 DEBUG "instance": "10.142.0.25:9100" 05:51:29 DEBUG }, 05:51:29 DEBUG "value": [ 05:51:29 DEBUG 05:51:29 DEBUG 05:51:29 DEBUG 05:51:29 DEBUG [...] 05:51:29 DEBUG 05:51:29 DEBUG 05:51:29 DEBUG 05:51:29 DEBUG "instance": "10.142.0.77:9100" 05:51:29 DEBUG }, 05:51:29 DEBUG "value": [ 05:51:29 DEBUG 1712724689, 05:51:29 DEBUG "42558146.53333333" 05:51:29 DEBUG ] 05:51:29 DEBUG } 05:51:29 DEBUG ] 05:51:29 DEBUG } 05:51:29 DEBUG } 05:51:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724689" 05:51:29 INFO [http_cmd]: http status code OK 05:51:29 DEBUG --- status code --- 05:51:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:29 DEBUG --- http response --- 05:51:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:51:29 DEBUG ----- output ----- 05:51:29 DEBUG { 05:51:29 DEBUG "status": "success", 05:51:29 DEBUG "data": { 05:51:29 DEBUG "resultType": "vector", 05:51:29 DEBUG "result": [ 05:51:29 DEBUG { 05:51:29 DEBUG "metric": { 05:51:29 DEBUG "instance": "10.142.0.25:9100" 05:51:29 DEBUG }, 05:51:29 DEBUG "value": [ 05:51:29 DEBUG 05:51:29 DEBUG 05:51:29 DEBUG 05:51:29 DEBUG [...] 05:51:29 DEBUG 05:51:29 DEBUG 05:51:29 DEBUG 05:51:29 DEBUG "instance": "10.142.0.77:9100" 05:51:29 DEBUG }, 05:51:29 DEBUG "value": [ 05:51:29 DEBUG 1712724689, 05:51:29 DEBUG "0.04999999999999935" 05:51:29 DEBUG ] 05:51:29 DEBUG } 05:51:29 DEBUG ] 05:51:29 DEBUG } 05:51:29 DEBUG } 05:51:45 INFO 05:51:45 INFO [loop_until]: kubectl --namespace=xlou top pods 05:51:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:51:45 INFO [loop_until]: OK (rc = 0) 05:51:45 DEBUG --- stdout --- 05:51:45 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10526m 8939Mi am-654c4fcbcf-9scrj 9752m 8430Mi am-654c4fcbcf-v5qb5 8160m 8403Mi ds-cts-0 8231m 12592Mi ds-cts-1 8782m 13040Mi ds-cts-2 7941m 12795Mi ds-idrepo-0 10655m 5622Mi ds-idrepo-1 409m 5672Mi ds-idrepo-2 379m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 8m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4197m 1872Mi 05:51:45 DEBUG --- stderr --- 05:51:45 DEBUG 05:51:52 INFO 05:51:52 INFO [loop_until]: kubectl --namespace=xlou top node 05:51:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:51:52 INFO [loop_until]: OK (rc = 0) 05:51:52 DEBUG --- stdout --- 05:51:52 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 97m 0% 2304Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9241m 58% 9282Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 72m 0% 3256Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13241m 83% 6602Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10856m 68% 9312Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11802m 74% 9756Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 77m 0% 850Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10815m 68% 6289Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8473m 53% 13298Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 7902m 49% 13410Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 429m 2% 6377Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8584m 54% 13475Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 467m 2% 6339Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4326m 27% 5676Mi 9% 05:51:52 DEBUG --- stderr --- 05:51:52 DEBUG 05:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712724745" 05:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724745" 05:52:25 ERROR [http_cmd]: ERROR 05:52:25 ERROR --- status code --- 05:52:25 INFO [http_cmd]: http status code OK 05:52:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:52:25 DEBUG --- status code --- 05:52:25 ERROR --- http response --- 05:52:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:52:25 DEBUG --- http response --- 05:52:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:52:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724745" 05:52:25 INFO [http_cmd]: http status code OK 05:52:25 DEBUG --- status code --- 05:52:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:25 DEBUG --- http response --- 05:52:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:52:25 DEBUG ----- output ----- 05:52:25 DEBUG { 05:52:25 DEBUG "status": "success", 05:52:25 DEBUG "data": { 05:52:25 DEBUG "resultType": "vector", 05:52:25 DEBUG "result": [ 05:52:25 DEBUG { 05:52:25 DEBUG "metric": { 05:52:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:52:25 DEBUG }, 05:52:25 DEBUG "value": [ 05:52:25 DEBUG 05:52:25 DEBUG 05:52:25 DEBUG 05:52:25 DEBUG [...] 05:52:25 DEBUG 05:52:25 DEBUG 05:52:25 DEBUG 05:52:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:52:25 DEBUG }, 05:52:25 DEBUG "value": [ 05:52:25 DEBUG 1712724745, 05:52:25 DEBUG "0" 05:52:25 DEBUG ] 05:52:25 DEBUG } 05:52:25 DEBUG ] 05:52:25 DEBUG } 05:52:25 DEBUG } 05:52:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724746" 05:52:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724746" 05:52:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724746" 05:52:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724746" 05:52:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724746" 05:52:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724746" 05:52:26 INFO [http_cmd]: http status code OK 05:52:26 DEBUG --- status code --- 05:52:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724746" 05:52:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:26 DEBUG --- http response --- 05:52:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:52:26 DEBUG ----- output ----- 05:52:26 DEBUG { 05:52:26 DEBUG "status": "success", 05:52:26 DEBUG "data": { 05:52:26 DEBUG "resultType": "vector", 05:52:26 DEBUG "result": [ 05:52:26 DEBUG { 05:52:26 DEBUG "metric": { 05:52:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:52:26 DEBUG }, 05:52:26 DEBUG "value": [ 05:52:26 DEBUG 05:52:26 DEBUG 05:52:26 DEBUG 05:52:26 DEBUG [...] 05:52:26 DEBUG 05:52:26 DEBUG 05:52:26 DEBUG 05:52:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:52:26 DEBUG }, 05:52:26 DEBUG "value": [ 05:52:26 DEBUG 1712724746, 05:52:26 DEBUG "1438.3666666666666" 05:52:26 DEBUG ] 05:52:26 DEBUG } 05:52:26 DEBUG ] 05:52:26 DEBUG } 05:52:26 DEBUG } 05:52:26 INFO [http_cmd]: http status code OK 05:52:26 INFO [http_cmd]: http status code OK 05:52:26 DEBUG --- status code --- 05:52:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:26 DEBUG --- http response --- 05:52:26 DEBUG --- status code --- 05:52:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:52:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:26 DEBUG ----- output ----- 05:52:26 DEBUG --- http response --- 05:52:26 DEBUG { 05:52:26 DEBUG "status": "success", 05:52:26 INFO [http_cmd]: http status code OK 05:52:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:52:26 DEBUG "data": { 05:52:26 DEBUG --- status code --- 05:52:26 DEBUG ----- output ----- 05:52:26 DEBUG { 05:52:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:26 DEBUG "status": "success", 05:52:26 DEBUG "resultType": "vector", 05:52:26 DEBUG --- http response --- 05:52:26 INFO [http_cmd]: http status code OK 05:52:26 INFO [http_cmd]: http status code OK 05:52:26 INFO [http_cmd]: http status code OK 05:52:26 DEBUG "data": { 05:52:26 DEBUG "result": [ 05:52:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:52:26 DEBUG --- status code --- 05:52:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:26 DEBUG --- status code --- 05:52:26 DEBUG "resultType": "vector", 05:52:26 DEBUG { 05:52:26 DEBUG ----- output ----- 05:52:26 DEBUG --- http response --- 05:52:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:52:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:26 DEBUG "result": [ 05:52:26 DEBUG "metric": { 05:52:26 DEBUG { 05:52:26 DEBUG ----- output ----- 05:52:26 DEBUG { 05:52:26 DEBUG --- http response --- 05:52:26 DEBUG { 05:52:26 DEBUG "pod": "ds-idrepo-0" 05:52:26 DEBUG "status": "success", 05:52:26 DEBUG "status": "success", 05:52:26 DEBUG --- status code --- 05:52:26 DEBUG "metric": { 05:52:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:52:26 DEBUG }, 05:52:26 DEBUG "data": { 05:52:26 DEBUG "data": { 05:52:26 DEBUG "resultType": "vector", 05:52:26 DEBUG "pod": "ds-idrepo-0" 05:52:26 DEBUG ----- output ----- 05:52:26 DEBUG { 05:52:26 DEBUG "resultType": "vector", 05:52:26 DEBUG "result": [ 05:52:26 DEBUG { 05:52:26 DEBUG }, 05:52:26 DEBUG "status": "success", 05:52:26 DEBUG "value": [ 05:52:26 DEBUG "metric": { 05:52:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:26 DEBUG "result": [ 05:52:26 DEBUG "value": [ 05:52:26 DEBUG "data": { 05:52:26 DEBUG "resultType": "vector", 05:52:26 DEBUG "result": [ 05:52:26 DEBUG { 05:52:26 DEBUG { 05:52:26 DEBUG 05:52:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:52:26 DEBUG "metric": { 05:52:26 DEBUG "instance": "10.142.0.25:9100" 05:52:26 DEBUG --- http response --- 05:52:26 DEBUG "metric": { 05:52:26 DEBUG 05:52:26 DEBUG }, 05:52:26 DEBUG }, 05:52:26 DEBUG "value": [ 05:52:26 DEBUG 05:52:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:52:26 DEBUG 05:52:26 DEBUG "value": [ 05:52:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:52:26 DEBUG ----- output ----- 05:52:26 DEBUG { 05:52:26 DEBUG }, 05:52:26 DEBUG [...] 05:52:26 DEBUG 05:52:26 DEBUG 05:52:26 DEBUG 05:52:26 DEBUG [...] 05:52:26 DEBUG "value": [ 05:52:26 DEBUG 05:52:26 DEBUG 05:52:26 DEBUG 05:52:26 DEBUG 05:52:26 DEBUG [...] 05:52:26 DEBUG 05:52:26 DEBUG "status": "success", 05:52:26 DEBUG 05:52:26 DEBUG 05:52:26 DEBUG 05:52:26 DEBUG 05:52:26 DEBUG "instance": "10.142.0.77:9100" 05:52:26 DEBUG 05:52:26 DEBUG "pod": "ds-cts-2" 05:52:26 DEBUG }, 05:52:26 DEBUG "data": { 05:52:26 DEBUG "resultType": "vector", 05:52:26 DEBUG "result": [ 05:52:26 DEBUG 05:52:26 DEBUG }, 05:52:26 DEBUG "value": [ 05:52:26 DEBUG 1712724746, 05:52:26 DEBUG "42989021.63333333" 05:52:26 DEBUG ] 05:52:26 DEBUG [...] 05:52:26 DEBUG "value": [ 05:52:26 DEBUG 1712724746, 05:52:26 DEBUG "29.763690297636902" 05:52:26 DEBUG { 05:52:26 DEBUG "metric": { 05:52:26 DEBUG 05:52:26 DEBUG 05:52:26 DEBUG ] 05:52:26 DEBUG } 05:52:26 DEBUG ] 05:52:26 DEBUG } 05:52:26 DEBUG 05:52:26 DEBUG 05:52:26 DEBUG 05:52:26 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:52:26 DEBUG }, 05:52:26 DEBUG } 05:52:26 DEBUG 05:52:26 DEBUG "value": [ 05:52:26 DEBUG 1712724746, 05:52:26 DEBUG "0" 05:52:26 DEBUG ] 05:52:26 DEBUG } 05:52:26 DEBUG 05:52:26 DEBUG ] 05:52:26 DEBUG } 05:52:26 DEBUG } 05:52:26 DEBUG "instance": "10.142.0.25:9100" 05:52:26 DEBUG }, 05:52:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:52:26 DEBUG }, 05:52:26 DEBUG } 05:52:26 DEBUG ] 05:52:26 DEBUG } 05:52:26 DEBUG } 05:52:26 DEBUG 05:52:26 DEBUG "value": [ 05:52:26 DEBUG 05:52:26 DEBUG "value": [ 05:52:26 DEBUG 1712724746, 05:52:26 DEBUG "258430" 05:52:26 DEBUG ] 05:52:26 DEBUG 05:52:26 DEBUG } 05:52:26 DEBUG ] 05:52:26 DEBUG 05:52:26 DEBUG } 05:52:26 DEBUG } 05:52:26 DEBUG [...] 05:52:26 DEBUG 05:52:26 DEBUG 05:52:26 DEBUG [...] 05:52:26 DEBUG 05:52:26 DEBUG 05:52:26 DEBUG 05:52:26 DEBUG 05:52:26 DEBUG "instance": "10.142.0.77:9100" 05:52:26 DEBUG 05:52:26 DEBUG "pod": "ds-cts-2" 05:52:26 DEBUG }, 05:52:26 DEBUG }, 05:52:26 DEBUG "value": [ 05:52:26 DEBUG "value": [ 05:52:26 DEBUG 1712724746, 05:52:26 DEBUG 1712724746, 05:52:26 DEBUG "323739624448" 05:52:26 DEBUG ] 05:52:26 DEBUG } 05:52:26 DEBUG ] 05:52:26 DEBUG } 05:52:26 DEBUG } 05:52:26 DEBUG "0" 05:52:26 DEBUG ] 05:52:26 DEBUG } 05:52:26 DEBUG ] 05:52:26 DEBUG } 05:52:26 DEBUG } 05:52:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724746" 05:52:26 INFO [http_cmd]: http status code OK 05:52:26 DEBUG --- status code --- 05:52:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:26 DEBUG --- http response --- 05:52:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:52:26 DEBUG ----- output ----- 05:52:26 DEBUG { 05:52:26 DEBUG "status": "success", 05:52:26 DEBUG "data": { 05:52:26 DEBUG "resultType": "vector", 05:52:26 DEBUG "result": [ 05:52:26 DEBUG { 05:52:26 DEBUG "metric": { 05:52:26 DEBUG "pod": "ds-idrepo-0" 05:52:26 DEBUG }, 05:52:26 DEBUG "value": [ 05:52:26 DEBUG 05:52:26 DEBUG 05:52:26 DEBUG 05:52:26 DEBUG [...] 05:52:26 DEBUG 05:52:26 DEBUG 05:52:26 DEBUG 05:52:26 DEBUG "pod": "ds-idrepo-1" 05:52:26 DEBUG }, 05:52:26 DEBUG "value": [ 05:52:26 DEBUG 1712724746, 05:52:26 DEBUG "0" 05:52:26 DEBUG ] 05:52:26 DEBUG } 05:52:26 DEBUG ] 05:52:26 DEBUG } 05:52:26 DEBUG } 05:52:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724746" 05:52:26 INFO [http_cmd]: http status code OK 05:52:26 DEBUG --- status code --- 05:52:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:26 DEBUG --- http response --- 05:52:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:52:26 DEBUG ----- output ----- 05:52:26 DEBUG { 05:52:26 DEBUG "status": "success", 05:52:26 DEBUG "data": { 05:52:26 DEBUG "resultType": "vector", 05:52:26 DEBUG "result": [ 05:52:26 DEBUG { 05:52:26 DEBUG "metric": { 05:52:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:52:26 DEBUG }, 05:52:26 DEBUG "value": [ 05:52:26 DEBUG 05:52:26 DEBUG 05:52:26 DEBUG 05:52:26 DEBUG [...] 05:52:26 DEBUG 05:52:26 DEBUG 05:52:26 DEBUG 05:52:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:52:26 DEBUG }, 05:52:26 DEBUG "value": [ 05:52:26 DEBUG 1712724746, 05:52:26 DEBUG "1461.8" 05:52:26 DEBUG ] 05:52:26 DEBUG } 05:52:26 DEBUG ] 05:52:26 DEBUG } 05:52:26 DEBUG } 05:52:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724746" 05:52:26 INFO [http_cmd]: http status code OK 05:52:26 DEBUG --- status code --- 05:52:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:26 DEBUG --- http response --- 05:52:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:52:26 DEBUG ----- output ----- 05:52:26 DEBUG { 05:52:26 DEBUG "status": "success", 05:52:26 DEBUG "data": { 05:52:26 DEBUG "resultType": "vector", 05:52:26 DEBUG "result": [ 05:52:26 DEBUG { 05:52:26 DEBUG "metric": { 05:52:26 DEBUG "instance": "10.142.0.25:9100" 05:52:26 DEBUG }, 05:52:26 DEBUG "value": [ 05:52:26 DEBUG 05:52:26 DEBUG 05:52:26 DEBUG 05:52:26 DEBUG [...] 05:52:26 DEBUG 05:52:26 DEBUG 05:52:26 DEBUG 05:52:26 DEBUG "instance": "10.142.0.77:9100" 05:52:26 DEBUG }, 05:52:26 DEBUG "value": [ 05:52:26 DEBUG 1712724746, 05:52:26 DEBUG "0.15196666666666836" 05:52:26 DEBUG ] 05:52:26 DEBUG } 05:52:26 DEBUG ] 05:52:26 DEBUG } 05:52:26 DEBUG } 05:52:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724747" 05:52:27 INFO [http_cmd]: http status code OK 05:52:27 DEBUG --- status code --- 05:52:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:27 DEBUG --- http response --- 05:52:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:52:27 DEBUG ----- output ----- 05:52:27 DEBUG { 05:52:27 DEBUG "status": "success", 05:52:27 DEBUG "data": { 05:52:27 DEBUG "resultType": "vector", 05:52:27 DEBUG "result": [ 05:52:27 DEBUG { 05:52:27 DEBUG "metric": { 05:52:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:52:27 DEBUG }, 05:52:27 DEBUG "value": [ 05:52:27 DEBUG 05:52:27 DEBUG 05:52:27 DEBUG 05:52:27 DEBUG [...] 05:52:27 DEBUG 05:52:27 DEBUG 05:52:27 DEBUG 05:52:27 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:52:27 DEBUG }, 05:52:27 DEBUG "value": [ 05:52:27 DEBUG 1712724747, 05:52:27 DEBUG "12532477.898052886" 05:52:27 DEBUG ] 05:52:27 DEBUG } 05:52:27 DEBUG ] 05:52:27 DEBUG } 05:52:27 DEBUG } 05:52:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712724747" 05:52:27 INFO [http_cmd]: http status code OK 05:52:27 DEBUG --- status code --- 05:52:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:27 DEBUG --- http response --- 05:52:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:52:27 DEBUG ----- output ----- 05:52:27 DEBUG { 05:52:27 DEBUG "status": "success", 05:52:27 DEBUG "data": { 05:52:27 DEBUG "resultType": "vector", 05:52:27 DEBUG "result": [ 05:52:27 DEBUG { 05:52:27 DEBUG "metric": { 05:52:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:52:27 DEBUG }, 05:52:27 DEBUG "value": [ 05:52:27 DEBUG 05:52:27 DEBUG 05:52:27 DEBUG 05:52:27 DEBUG [...] 05:52:27 DEBUG 05:52:27 DEBUG 05:52:27 DEBUG 05:52:27 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:52:27 DEBUG }, 05:52:27 DEBUG "value": [ 05:52:27 DEBUG 1712724747, 05:52:27 DEBUG "4.295246625587598" 05:52:27 DEBUG ] 05:52:27 DEBUG } 05:52:27 DEBUG ] 05:52:27 DEBUG } 05:52:27 DEBUG } 05:52:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712724747" 05:52:27 INFO [http_cmd]: http status code OK 05:52:27 DEBUG --- status code --- 05:52:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:27 DEBUG --- http response --- 05:52:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:52:27 DEBUG ----- output ----- 05:52:27 DEBUG { 05:52:27 DEBUG "status": "success", 05:52:27 DEBUG "data": { 05:52:27 DEBUG "resultType": "vector", 05:52:27 DEBUG "result": [ 05:52:27 DEBUG { 05:52:27 DEBUG "metric": { 05:52:27 DEBUG "pod": "ds-idrepo-0" 05:52:27 DEBUG }, 05:52:27 DEBUG "value": [ 05:52:27 DEBUG 05:52:27 DEBUG 05:52:27 DEBUG 05:52:27 DEBUG [...] 05:52:27 DEBUG 05:52:27 DEBUG 05:52:27 DEBUG 05:52:27 DEBUG "pod": "ds-cts-2" 05:52:27 DEBUG }, 05:52:27 DEBUG "value": [ 05:52:27 DEBUG 1712724747, 05:52:27 DEBUG "0" 05:52:27 DEBUG ] 05:52:27 DEBUG } 05:52:27 DEBUG ] 05:52:27 DEBUG } 05:52:27 DEBUG } 05:52:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724748" 05:52:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724748" 05:52:28 INFO [http_cmd]: http status code OK 05:52:28 DEBUG --- status code --- 05:52:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:28 DEBUG --- http response --- 05:52:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:52:28 DEBUG ----- output ----- 05:52:28 DEBUG { 05:52:28 DEBUG "status": "success", 05:52:28 DEBUG "data": { 05:52:28 DEBUG "resultType": "vector", 05:52:28 DEBUG "result": [ 05:52:28 DEBUG { 05:52:28 DEBUG "metric": { 05:52:28 DEBUG "pod": "ds-idrepo-2" 05:52:28 DEBUG }, 05:52:28 DEBUG "value": [ 05:52:28 DEBUG 05:52:28 DEBUG 05:52:28 DEBUG 05:52:28 DEBUG [...] 05:52:28 DEBUG 05:52:28 DEBUG 05:52:28 DEBUG 05:52:28 DEBUG "pod": "ds-cts-2" 05:52:28 DEBUG }, 05:52:28 DEBUG "value": [ 05:52:28 DEBUG 1712724748, 05:52:28 DEBUG "0.000033330000333300005" 05:52:28 DEBUG ] 05:52:28 DEBUG } 05:52:28 DEBUG ] 05:52:28 DEBUG } 05:52:28 DEBUG } 05:52:28 INFO [http_cmd]: http status code OK 05:52:28 DEBUG --- status code --- 05:52:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:28 DEBUG --- http response --- 05:52:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:52:28 DEBUG ----- output ----- 05:52:28 DEBUG { 05:52:28 DEBUG "status": "success", 05:52:28 DEBUG "data": { 05:52:28 DEBUG "resultType": "vector", 05:52:28 DEBUG "result": [ 05:52:28 DEBUG { 05:52:28 DEBUG "metric": { 05:52:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:52:28 DEBUG }, 05:52:28 DEBUG "value": [ 05:52:28 DEBUG 05:52:28 DEBUG 05:52:28 DEBUG 05:52:28 DEBUG [...] 05:52:28 DEBUG 05:52:28 DEBUG 05:52:28 DEBUG 05:52:28 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:52:28 DEBUG }, 05:52:28 DEBUG "value": [ 05:52:28 DEBUG 1712724748, 05:52:28 DEBUG "4.2633934194511225" 05:52:28 DEBUG ] 05:52:28 DEBUG } 05:52:28 DEBUG ] 05:52:28 DEBUG } 05:52:28 DEBUG } 05:52:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724748" 05:52:28 INFO [http_cmd]: http status code OK 05:52:28 DEBUG --- status code --- 05:52:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:28 DEBUG --- http response --- 05:52:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:52:28 DEBUG ----- output ----- 05:52:28 DEBUG { 05:52:28 DEBUG "status": "success", 05:52:28 DEBUG "data": { 05:52:28 DEBUG "resultType": "vector", 05:52:28 DEBUG "result": [ 05:52:28 DEBUG { 05:52:28 DEBUG "metric": { 05:52:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:52:28 DEBUG }, 05:52:28 DEBUG "value": [ 05:52:28 DEBUG 05:52:28 DEBUG 05:52:28 DEBUG 05:52:28 DEBUG [...] 05:52:28 DEBUG 05:52:28 DEBUG 05:52:28 DEBUG 05:52:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:52:28 DEBUG }, 05:52:28 DEBUG "value": [ 05:52:28 DEBUG 1712724748, 05:52:28 DEBUG "1460.8666666666666" 05:52:28 DEBUG ] 05:52:28 DEBUG } 05:52:28 DEBUG ] 05:52:28 DEBUG } 05:52:28 DEBUG } 05:52:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712724748" 05:52:28 INFO [http_cmd]: http status code OK 05:52:28 DEBUG --- status code --- 05:52:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:28 DEBUG --- http response --- 05:52:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:52:28 DEBUG ----- output ----- 05:52:28 DEBUG { 05:52:28 DEBUG "status": "success", 05:52:28 DEBUG "data": { 05:52:28 DEBUG "resultType": "vector", 05:52:28 DEBUG "result": [ 05:52:28 DEBUG { 05:52:28 DEBUG "metric": { 05:52:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:52:28 DEBUG }, 05:52:28 DEBUG "value": [ 05:52:28 DEBUG 05:52:28 DEBUG 05:52:28 DEBUG 05:52:28 DEBUG [...] 05:52:28 DEBUG 05:52:28 DEBUG 05:52:28 DEBUG 05:52:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:52:28 DEBUG }, 05:52:28 DEBUG "value": [ 05:52:28 DEBUG 1712724748, 05:52:28 DEBUG "0.044" 05:52:28 DEBUG ] 05:52:28 DEBUG } 05:52:28 DEBUG ] 05:52:28 DEBUG } 05:52:28 DEBUG } 05:52:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724748" 05:52:28 INFO [http_cmd]: http status code OK 05:52:28 DEBUG --- status code --- 05:52:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:28 DEBUG --- http response --- 05:52:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:52:28 DEBUG ----- output ----- 05:52:28 DEBUG { 05:52:28 DEBUG "status": "success", 05:52:28 DEBUG "data": { 05:52:28 DEBUG "resultType": "vector", 05:52:28 DEBUG "result": [ 05:52:28 DEBUG { 05:52:28 DEBUG "metric": { 05:52:28 DEBUG "pod": "ds-idrepo-0" 05:52:28 DEBUG }, 05:52:28 DEBUG "value": [ 05:52:28 DEBUG 05:52:28 DEBUG 05:52:28 DEBUG 05:52:28 DEBUG [...] 05:52:28 DEBUG 05:52:28 DEBUG 05:52:28 DEBUG 05:52:28 DEBUG "pod": "ds-idrepo-1" 05:52:28 DEBUG }, 05:52:28 DEBUG "value": [ 05:52:28 DEBUG 1712724748, 05:52:28 DEBUG "0" 05:52:28 DEBUG ] 05:52:28 DEBUG } 05:52:28 DEBUG ] 05:52:28 DEBUG } 05:52:28 DEBUG } 05:52:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724748" 05:52:28 INFO [http_cmd]: http status code OK 05:52:28 DEBUG --- status code --- 05:52:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:28 DEBUG --- http response --- 05:52:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:52:28 DEBUG ----- output ----- 05:52:28 DEBUG { 05:52:28 DEBUG "status": "success", 05:52:28 DEBUG "data": { 05:52:28 DEBUG "resultType": "vector", 05:52:28 DEBUG "result": [ 05:52:28 DEBUG { 05:52:28 DEBUG "metric": { 05:52:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:52:28 DEBUG }, 05:52:28 DEBUG "value": [ 05:52:28 DEBUG 05:52:28 DEBUG 05:52:28 DEBUG 05:52:28 DEBUG [...] 05:52:28 DEBUG 05:52:28 DEBUG 05:52:28 DEBUG 05:52:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:52:28 DEBUG }, 05:52:28 DEBUG "value": [ 05:52:28 DEBUG 1712724748, 05:52:28 DEBUG "1426.8" 05:52:28 DEBUG ] 05:52:28 DEBUG } 05:52:28 DEBUG ] 05:52:28 DEBUG } 05:52:28 DEBUG } 05:52:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724749" 05:52:29 INFO [http_cmd]: http status code OK 05:52:29 DEBUG --- status code --- 05:52:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:29 DEBUG --- http response --- 05:52:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:52:29 DEBUG ----- output ----- 05:52:29 DEBUG { 05:52:29 DEBUG "status": "success", 05:52:29 DEBUG "data": { 05:52:29 DEBUG "resultType": "vector", 05:52:29 DEBUG "result": [ 05:52:29 DEBUG { 05:52:29 DEBUG "metric": { 05:52:29 DEBUG "pod": "ds-idrepo-0" 05:52:29 DEBUG }, 05:52:29 DEBUG "value": [ 05:52:29 DEBUG 05:52:29 DEBUG 05:52:29 DEBUG 05:52:29 DEBUG [...] 05:52:29 DEBUG 05:52:29 DEBUG 05:52:29 DEBUG 05:52:29 DEBUG "pod": "ds-idrepo-1" 05:52:29 DEBUG }, 05:52:29 DEBUG "value": [ 05:52:29 DEBUG 1712724749, 05:52:29 DEBUG "0" 05:52:29 DEBUG ] 05:52:29 DEBUG } 05:52:29 DEBUG ] 05:52:29 DEBUG } 05:52:29 DEBUG } 05:52:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724749" 05:52:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712724749" 05:52:29 INFO [http_cmd]: http status code OK 05:52:29 DEBUG --- status code --- 05:52:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:29 DEBUG --- http response --- 05:52:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:52:29 DEBUG ----- output ----- 05:52:29 DEBUG { 05:52:29 DEBUG "status": "success", 05:52:29 DEBUG "data": { 05:52:29 DEBUG "resultType": "vector", 05:52:29 DEBUG "result": [ 05:52:29 DEBUG { 05:52:29 DEBUG "metric": { 05:52:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:52:29 DEBUG }, 05:52:29 DEBUG "value": [ 05:52:29 DEBUG 05:52:29 DEBUG 05:52:29 DEBUG 05:52:29 DEBUG [...] 05:52:29 DEBUG 05:52:29 DEBUG 05:52:29 DEBUG 05:52:29 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 05:52:29 DEBUG }, 05:52:29 DEBUG "value": [ 05:52:29 DEBUG 1712724749, 05:52:29 DEBUG "0" 05:52:29 DEBUG ] 05:52:29 DEBUG } 05:52:29 DEBUG ] 05:52:29 DEBUG } 05:52:29 DEBUG } 05:52:29 INFO [http_cmd]: http status code OK 05:52:29 DEBUG --- status code --- 05:52:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:29 DEBUG --- http response --- 05:52:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:52:29 DEBUG ----- output ----- 05:52:29 DEBUG { 05:52:29 DEBUG "status": "success", 05:52:29 DEBUG "data": { 05:52:29 DEBUG "resultType": "vector", 05:52:29 DEBUG "result": [ 05:52:29 DEBUG { 05:52:29 DEBUG "metric": { 05:52:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:52:29 DEBUG }, 05:52:29 DEBUG "value": [ 05:52:29 DEBUG 05:52:29 DEBUG 05:52:29 DEBUG 05:52:29 DEBUG [...] 05:52:29 DEBUG 05:52:29 DEBUG 05:52:29 DEBUG 05:52:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:52:29 DEBUG }, 05:52:29 DEBUG "value": [ 05:52:29 DEBUG 1712724749, 05:52:29 DEBUG "2485981184" 05:52:29 DEBUG ] 05:52:29 DEBUG } 05:52:29 DEBUG ] 05:52:29 DEBUG } 05:52:29 DEBUG } 05:52:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724749" 05:52:29 INFO [http_cmd]: http status code OK 05:52:29 DEBUG --- status code --- 05:52:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:29 DEBUG --- http response --- 05:52:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:52:29 DEBUG ----- output ----- 05:52:29 DEBUG { 05:52:29 DEBUG "status": "success", 05:52:29 DEBUG "data": { 05:52:29 DEBUG "resultType": "vector", 05:52:29 DEBUG "result": [ 05:52:29 DEBUG { 05:52:29 DEBUG "metric": { 05:52:29 DEBUG "instance": "10.142.0.25:9100" 05:52:29 DEBUG }, 05:52:29 DEBUG "value": [ 05:52:29 DEBUG 05:52:29 DEBUG 05:52:29 DEBUG 05:52:29 DEBUG [...] 05:52:29 DEBUG 05:52:29 DEBUG 05:52:29 DEBUG 05:52:29 DEBUG "instance": "10.142.0.77:9100" 05:52:29 DEBUG }, 05:52:29 DEBUG "value": [ 05:52:29 DEBUG 1712724749, 05:52:29 DEBUG "42990165.5" 05:52:29 DEBUG ] 05:52:29 DEBUG } 05:52:29 DEBUG ] 05:52:29 DEBUG } 05:52:29 DEBUG } 05:52:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724749" 05:52:29 INFO [http_cmd]: http status code OK 05:52:29 DEBUG --- status code --- 05:52:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:29 DEBUG --- http response --- 05:52:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:52:29 DEBUG ----- output ----- 05:52:29 DEBUG { 05:52:29 DEBUG "status": "success", 05:52:29 DEBUG "data": { 05:52:29 DEBUG "resultType": "vector", 05:52:29 DEBUG "result": [ 05:52:29 DEBUG { 05:52:29 DEBUG "metric": { 05:52:29 DEBUG "instance": "10.142.0.25:9100" 05:52:29 DEBUG }, 05:52:29 DEBUG "value": [ 05:52:29 DEBUG 05:52:29 DEBUG 05:52:29 DEBUG 05:52:29 DEBUG [...] 05:52:29 DEBUG 05:52:29 DEBUG 05:52:29 DEBUG 05:52:29 DEBUG "instance": "10.142.0.77:9100" 05:52:29 DEBUG }, 05:52:29 DEBUG "value": [ 05:52:29 DEBUG 1712724749, 05:52:29 DEBUG "0.04933333333333305" 05:52:29 DEBUG ] 05:52:29 DEBUG } 05:52:29 DEBUG ] 05:52:29 DEBUG } 05:52:29 DEBUG } 05:52:45 INFO 05:52:45 INFO [loop_until]: kubectl --namespace=xlou top pods 05:52:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:52:45 INFO [loop_until]: OK (rc = 0) 05:52:45 DEBUG --- stdout --- 05:52:45 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10324m 8946Mi am-654c4fcbcf-9scrj 10262m 8434Mi am-654c4fcbcf-v5qb5 8394m 8406Mi ds-cts-0 8119m 13315Mi ds-cts-1 8532m 13528Mi ds-cts-2 8153m 12949Mi ds-idrepo-0 10693m 5622Mi ds-idrepo-1 358m 5671Mi ds-idrepo-2 419m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 8m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4160m 2227Mi 05:52:45 DEBUG --- stderr --- 05:52:45 DEBUG 05:52:52 INFO 05:52:52 INFO [loop_until]: kubectl --namespace=xlou top node 05:52:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:52:52 INFO [loop_until]: OK (rc = 0) 05:52:52 DEBUG --- stdout --- 05:52:52 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 101m 0% 2304Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9335m 58% 9282Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 75m 0% 3257Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 12685m 79% 6533Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 11004m 69% 9316Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11355m 71% 9771Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 72m 0% 854Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10851m 68% 6287Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8424m 53% 13942Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 8163m 51% 13541Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 480m 3% 6380Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8290m 52% 14104Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 410m 2% 6339Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4256m 26% 5888Mi 10% 05:52:52 DEBUG --- stderr --- 05:52:52 DEBUG 05:53:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712724805" 05:53:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724805" 05:53:25 ERROR [http_cmd]: ERROR 05:53:25 ERROR --- status code --- 05:53:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:53:25 ERROR --- http response --- 05:53:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:53:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:53:25 INFO [http_cmd]: http status code OK 05:53:25 DEBUG --- status code --- 05:53:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:25 DEBUG --- http response --- 05:53:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:53:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724805" 05:53:25 INFO [http_cmd]: http status code OK 05:53:25 DEBUG --- status code --- 05:53:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:25 DEBUG --- http response --- 05:53:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:53:25 DEBUG ----- output ----- 05:53:25 DEBUG { 05:53:25 DEBUG "status": "success", 05:53:25 DEBUG "data": { 05:53:25 DEBUG "resultType": "vector", 05:53:25 DEBUG "result": [ 05:53:25 DEBUG { 05:53:25 DEBUG "metric": { 05:53:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:53:25 DEBUG }, 05:53:25 DEBUG "value": [ 05:53:25 DEBUG 05:53:25 DEBUG 05:53:25 DEBUG 05:53:25 DEBUG [...] 05:53:25 DEBUG 05:53:25 DEBUG 05:53:25 DEBUG 05:53:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:53:25 DEBUG }, 05:53:25 DEBUG "value": [ 05:53:25 DEBUG 1712724805, 05:53:25 DEBUG "0" 05:53:25 DEBUG ] 05:53:25 DEBUG } 05:53:25 DEBUG ] 05:53:25 DEBUG } 05:53:25 DEBUG } 05:53:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724806" 05:53:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724806" 05:53:26 INFO [http_cmd]: http status code OK 05:53:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724806" 05:53:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724806" 05:53:26 DEBUG --- status code --- 05:53:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724806" 05:53:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724806" 05:53:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724806" 05:53:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:26 DEBUG --- http response --- 05:53:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:53:26 DEBUG ----- output ----- 05:53:26 DEBUG { 05:53:26 DEBUG "status": "success", 05:53:26 DEBUG "data": { 05:53:26 DEBUG "resultType": "vector", 05:53:26 DEBUG "result": [ 05:53:26 DEBUG { 05:53:26 DEBUG "metric": { 05:53:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:53:26 DEBUG }, 05:53:26 DEBUG "value": [ 05:53:26 DEBUG 05:53:26 DEBUG 05:53:26 DEBUG 05:53:26 DEBUG [...] 05:53:26 DEBUG 05:53:26 INFO [http_cmd]: http status code OK 05:53:26 INFO [http_cmd]: http status code OK 05:53:26 DEBUG 05:53:26 DEBUG --- status code --- 05:53:26 DEBUG --- status code --- 05:53:26 DEBUG 05:53:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:53:26 DEBUG --- http response --- 05:53:26 DEBUG --- http response --- 05:53:26 DEBUG }, 05:53:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:53:26 DEBUG ----- output ----- 05:53:26 DEBUG { 05:53:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:53:26 DEBUG "value": [ 05:53:26 DEBUG "status": "success", 05:53:26 DEBUG ----- output ----- 05:53:26 DEBUG { 05:53:26 DEBUG "data": { 05:53:26 DEBUG 1712724806, 05:53:26 DEBUG "status": "success", 05:53:26 DEBUG "resultType": "vector", 05:53:26 DEBUG "1427.3" 05:53:26 DEBUG ] 05:53:26 DEBUG "result": [ 05:53:26 DEBUG } 05:53:26 DEBUG "data": { 05:53:26 DEBUG { 05:53:26 DEBUG ] 05:53:26 DEBUG "resultType": "vector", 05:53:26 DEBUG "metric": { 05:53:26 DEBUG } 05:53:26 DEBUG "result": [ 05:53:26 DEBUG "pod": "ds-idrepo-0" 05:53:26 INFO [http_cmd]: http status code OK 05:53:26 DEBUG } 05:53:26 DEBUG { 05:53:26 DEBUG }, 05:53:26 DEBUG --- status code --- 05:53:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:26 DEBUG "value": [ 05:53:26 DEBUG "metric": { 05:53:26 DEBUG --- http response --- 05:53:26 DEBUG 05:53:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:53:26 DEBUG }, 05:53:26 DEBUG 05:53:26 DEBUG 05:53:26 DEBUG "value": [ 05:53:26 DEBUG [...] 05:53:26 DEBUG 05:53:26 DEBUG 05:53:26 DEBUG 05:53:26 DEBUG 05:53:26 DEBUG 05:53:26 DEBUG "pod": "ds-cts-2" 05:53:26 DEBUG }, 05:53:26 DEBUG "value": [ 05:53:26 DEBUG 1712724806, 05:53:26 DEBUG "29.7" 05:53:26 DEBUG ] 05:53:26 DEBUG } 05:53:26 DEBUG ] 05:53:26 DEBUG } 05:53:26 DEBUG } 05:53:26 DEBUG 05:53:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:53:26 INFO [http_cmd]: http status code OK 05:53:26 DEBUG [...] 05:53:26 INFO [http_cmd]: http status code OK 05:53:26 DEBUG ----- output ----- 05:53:26 DEBUG { 05:53:26 DEBUG --- status code --- 05:53:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:26 DEBUG --- status code --- 05:53:26 DEBUG "status": "success", 05:53:26 INFO [http_cmd]: http status code OK 05:53:26 DEBUG --- http response --- 05:53:26 DEBUG 05:53:26 DEBUG 05:53:26 DEBUG "data": { 05:53:26 DEBUG "resultType": "vector", 05:53:26 DEBUG "result": [ 05:53:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:26 DEBUG 05:53:26 DEBUG --- status code --- 05:53:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:53:26 DEBUG { 05:53:26 DEBUG --- http response --- 05:53:26 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:53:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:26 DEBUG ----- output ----- 05:53:26 DEBUG { 05:53:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:53:26 DEBUG }, 05:53:26 DEBUG --- http response --- 05:53:26 DEBUG "status": "success", 05:53:26 DEBUG "metric": { 05:53:26 DEBUG ----- output ----- 05:53:26 DEBUG "value": [ 05:53:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:53:26 DEBUG "data": { 05:53:26 DEBUG "instance": "10.142.0.25:9100" 05:53:26 DEBUG { 05:53:26 DEBUG 1712724806, 05:53:26 DEBUG ----- output ----- 05:53:26 DEBUG "resultType": "vector", 05:53:26 DEBUG }, 05:53:26 DEBUG "status": "success", 05:53:26 DEBUG "0" 05:53:26 DEBUG ] 05:53:26 DEBUG } 05:53:26 DEBUG "value": [ 05:53:26 DEBUG 05:53:26 DEBUG "result": [ 05:53:26 DEBUG ] 05:53:26 DEBUG { 05:53:26 DEBUG 05:53:26 DEBUG "data": { 05:53:26 DEBUG { 05:53:26 DEBUG } 05:53:26 DEBUG "status": "success", 05:53:26 DEBUG "data": { 05:53:26 DEBUG "resultType": "vector", 05:53:26 DEBUG "metric": { 05:53:26 DEBUG } 05:53:26 DEBUG 05:53:26 DEBUG "resultType": "vector", 05:53:26 DEBUG "result": [ 05:53:26 DEBUG "instance": "10.142.0.25:9100" 05:53:26 DEBUG [...] 05:53:26 DEBUG 05:53:26 DEBUG 05:53:26 DEBUG }, 05:53:26 DEBUG { 05:53:26 DEBUG "metric": { 05:53:26 DEBUG 05:53:26 DEBUG "value": [ 05:53:26 DEBUG "pod": "ds-idrepo-0" 05:53:26 DEBUG }, 05:53:26 DEBUG "instance": "10.142.0.77:9100" 05:53:26 DEBUG }, 05:53:26 DEBUG "value": [ 05:53:26 DEBUG 05:53:26 DEBUG "value": [ 05:53:26 DEBUG 05:53:26 DEBUG "result": [ 05:53:26 DEBUG 05:53:26 DEBUG 1712724806, 05:53:26 DEBUG 05:53:26 DEBUG { 05:53:26 DEBUG "metric": { 05:53:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:53:26 DEBUG 05:53:26 DEBUG 05:53:26 DEBUG [...] 05:53:26 DEBUG 05:53:26 DEBUG [...] 05:53:26 DEBUG 05:53:26 DEBUG 05:53:26 DEBUG "pod": "ds-cts-2" 05:53:26 DEBUG 05:53:26 DEBUG }, 05:53:26 DEBUG "value": [ 05:53:26 DEBUG 1712724806, 05:53:26 DEBUG 05:53:26 DEBUG "0" 05:53:26 DEBUG ] 05:53:26 DEBUG } 05:53:26 DEBUG 05:53:26 DEBUG ] 05:53:26 DEBUG } 05:53:26 DEBUG } 05:53:26 DEBUG "instance": "10.142.0.77:9100" 05:53:26 DEBUG }, 05:53:26 DEBUG }, 05:53:26 DEBUG "value": [ 05:53:26 DEBUG 05:53:26 DEBUG 05:53:26 DEBUG 05:53:26 DEBUG [...] 05:53:26 DEBUG "42311786.93333334" 05:53:26 DEBUG ] 05:53:26 DEBUG 05:53:26 DEBUG "value": [ 05:53:26 DEBUG 1712724806, 05:53:26 DEBUG "325650287616" 05:53:26 DEBUG ] 05:53:26 DEBUG } 05:53:26 DEBUG ] 05:53:26 DEBUG } 05:53:26 DEBUG } 05:53:26 DEBUG 05:53:26 DEBUG } 05:53:26 DEBUG ] 05:53:26 DEBUG } 05:53:26 DEBUG 05:53:26 DEBUG } 05:53:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:53:26 DEBUG }, 05:53:26 DEBUG "value": [ 05:53:26 DEBUG 1712724806, 05:53:26 DEBUG "258584.5" 05:53:26 DEBUG ] 05:53:26 DEBUG } 05:53:26 DEBUG ] 05:53:26 DEBUG } 05:53:26 DEBUG } 05:53:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724806" 05:53:26 INFO [http_cmd]: http status code OK 05:53:26 DEBUG --- status code --- 05:53:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:26 DEBUG --- http response --- 05:53:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:53:26 DEBUG ----- output ----- 05:53:26 DEBUG { 05:53:26 DEBUG "status": "success", 05:53:26 DEBUG "data": { 05:53:26 DEBUG "resultType": "vector", 05:53:26 DEBUG "result": [ 05:53:26 DEBUG { 05:53:26 DEBUG "metric": { 05:53:26 DEBUG "pod": "ds-idrepo-0" 05:53:26 DEBUG }, 05:53:26 DEBUG "value": [ 05:53:26 DEBUG 05:53:26 DEBUG 05:53:26 DEBUG 05:53:26 DEBUG [...] 05:53:26 DEBUG 05:53:26 DEBUG 05:53:26 DEBUG 05:53:26 DEBUG "pod": "ds-idrepo-1" 05:53:26 DEBUG }, 05:53:26 DEBUG "value": [ 05:53:26 DEBUG 1712724806, 05:53:26 DEBUG "0" 05:53:26 DEBUG ] 05:53:26 DEBUG } 05:53:26 DEBUG ] 05:53:26 DEBUG } 05:53:26 DEBUG } 05:53:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724806" 05:53:26 INFO [http_cmd]: http status code OK 05:53:26 DEBUG --- status code --- 05:53:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:26 DEBUG --- http response --- 05:53:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:53:26 DEBUG ----- output ----- 05:53:26 DEBUG { 05:53:26 DEBUG "status": "success", 05:53:26 DEBUG "data": { 05:53:26 DEBUG "resultType": "vector", 05:53:26 DEBUG "result": [ 05:53:26 DEBUG { 05:53:26 DEBUG "metric": { 05:53:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:53:26 DEBUG }, 05:53:26 DEBUG "value": [ 05:53:26 DEBUG 05:53:26 DEBUG 05:53:26 DEBUG 05:53:26 DEBUG [...] 05:53:26 DEBUG 05:53:26 DEBUG 05:53:26 DEBUG 05:53:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:53:26 DEBUG }, 05:53:26 DEBUG "value": [ 05:53:26 DEBUG 1712724806, 05:53:26 DEBUG "1423.6" 05:53:26 DEBUG ] 05:53:26 DEBUG } 05:53:26 DEBUG ] 05:53:26 DEBUG } 05:53:26 DEBUG } 05:53:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724806" 05:53:26 INFO [http_cmd]: http status code OK 05:53:26 DEBUG --- status code --- 05:53:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:26 DEBUG --- http response --- 05:53:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:53:26 DEBUG ----- output ----- 05:53:26 DEBUG { 05:53:26 DEBUG "status": "success", 05:53:26 DEBUG "data": { 05:53:26 DEBUG "resultType": "vector", 05:53:26 DEBUG "result": [ 05:53:26 DEBUG { 05:53:26 DEBUG "metric": { 05:53:26 DEBUG "instance": "10.142.0.25:9100" 05:53:26 DEBUG }, 05:53:26 DEBUG "value": [ 05:53:26 DEBUG 05:53:26 DEBUG 05:53:26 DEBUG 05:53:26 DEBUG [...] 05:53:26 DEBUG 05:53:26 DEBUG 05:53:26 DEBUG 05:53:26 DEBUG "instance": "10.142.0.77:9100" 05:53:26 DEBUG }, 05:53:26 DEBUG "value": [ 05:53:26 DEBUG 1712724806, 05:53:26 DEBUG "0.1628333333333315" 05:53:26 DEBUG ] 05:53:26 DEBUG } 05:53:26 DEBUG ] 05:53:26 DEBUG } 05:53:26 DEBUG } 05:53:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724807" 05:53:27 INFO [http_cmd]: http status code OK 05:53:27 DEBUG --- status code --- 05:53:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:27 DEBUG --- http response --- 05:53:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:53:27 DEBUG ----- output ----- 05:53:27 DEBUG { 05:53:27 DEBUG "status": "success", 05:53:27 DEBUG "data": { 05:53:27 DEBUG "resultType": "vector", 05:53:27 DEBUG "result": [ 05:53:27 DEBUG { 05:53:27 DEBUG "metric": { 05:53:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:53:27 DEBUG }, 05:53:27 DEBUG "value": [ 05:53:27 DEBUG 05:53:27 DEBUG 05:53:27 DEBUG 05:53:27 DEBUG [...] 05:53:27 DEBUG 05:53:27 DEBUG 05:53:27 DEBUG 05:53:27 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:53:27 DEBUG }, 05:53:27 DEBUG "value": [ 05:53:27 DEBUG 1712724807, 05:53:27 DEBUG "21735.5792034802" 05:53:27 DEBUG ] 05:53:27 DEBUG } 05:53:27 DEBUG ] 05:53:27 DEBUG } 05:53:27 DEBUG } 05:53:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712724807" 05:53:27 INFO [http_cmd]: http status code OK 05:53:27 DEBUG --- status code --- 05:53:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:27 DEBUG --- http response --- 05:53:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:53:27 DEBUG ----- output ----- 05:53:27 DEBUG { 05:53:27 DEBUG "status": "success", 05:53:27 DEBUG "data": { 05:53:27 DEBUG "resultType": "vector", 05:53:27 DEBUG "result": [ 05:53:27 DEBUG { 05:53:27 DEBUG "metric": { 05:53:27 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:53:27 DEBUG }, 05:53:27 DEBUG "value": [ 05:53:27 DEBUG 05:53:27 DEBUG 05:53:27 DEBUG 05:53:27 DEBUG [...] 05:53:27 DEBUG 05:53:27 DEBUG 05:53:27 DEBUG 05:53:27 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:53:27 DEBUG }, 05:53:27 DEBUG "value": [ 05:53:27 DEBUG 1712724807, 05:53:27 DEBUG "4.141247967747326" 05:53:27 DEBUG ] 05:53:27 DEBUG } 05:53:27 DEBUG ] 05:53:27 DEBUG } 05:53:27 DEBUG } 05:53:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712724808" 05:53:28 INFO [http_cmd]: http status code OK 05:53:28 DEBUG --- status code --- 05:53:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:28 DEBUG --- http response --- 05:53:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:53:28 DEBUG ----- output ----- 05:53:28 DEBUG { 05:53:28 DEBUG "status": "success", 05:53:28 DEBUG "data": { 05:53:28 DEBUG "resultType": "vector", 05:53:28 DEBUG "result": [ 05:53:28 DEBUG { 05:53:28 DEBUG "metric": { 05:53:28 DEBUG "pod": "ds-idrepo-0" 05:53:28 DEBUG }, 05:53:28 DEBUG "value": [ 05:53:28 DEBUG 05:53:28 DEBUG 05:53:28 DEBUG 05:53:28 DEBUG [...] 05:53:28 DEBUG 05:53:28 DEBUG 05:53:28 DEBUG 05:53:28 DEBUG "pod": "ds-cts-2" 05:53:28 DEBUG }, 05:53:28 DEBUG "value": [ 05:53:28 DEBUG 1712724808, 05:53:28 DEBUG "0" 05:53:28 DEBUG ] 05:53:28 DEBUG } 05:53:28 DEBUG ] 05:53:28 DEBUG } 05:53:28 DEBUG } 05:53:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724808" 05:53:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724808" 05:53:28 INFO [http_cmd]: http status code OK 05:53:28 DEBUG --- status code --- 05:53:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:28 DEBUG --- http response --- 05:53:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:53:28 DEBUG ----- output ----- 05:53:28 DEBUG { 05:53:28 DEBUG "status": "success", 05:53:28 DEBUG "data": { 05:53:28 DEBUG "resultType": "vector", 05:53:28 DEBUG "result": [ 05:53:28 DEBUG { 05:53:28 DEBUG "metric": { 05:53:28 DEBUG "pod": "ds-idrepo-2" 05:53:28 DEBUG }, 05:53:28 DEBUG "value": [ 05:53:28 DEBUG 05:53:28 DEBUG 05:53:28 DEBUG 05:53:28 DEBUG [...] 05:53:28 DEBUG 05:53:28 DEBUG 05:53:28 DEBUG 05:53:28 DEBUG "pod": "ds-cts-2" 05:53:28 DEBUG }, 05:53:28 DEBUG "value": [ 05:53:28 DEBUG 1712724808, 05:53:28 DEBUG "0.000021774444444444445" 05:53:28 DEBUG ] 05:53:28 DEBUG } 05:53:28 DEBUG ] 05:53:28 DEBUG } 05:53:28 DEBUG } 05:53:28 INFO [http_cmd]: http status code OK 05:53:28 DEBUG --- status code --- 05:53:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:28 DEBUG --- http response --- 05:53:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:53:28 DEBUG ----- output ----- 05:53:28 DEBUG { 05:53:28 DEBUG "status": "success", 05:53:28 DEBUG "data": { 05:53:28 DEBUG "resultType": "vector", 05:53:28 DEBUG "result": [ 05:53:28 DEBUG { 05:53:28 DEBUG "metric": { 05:53:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:53:28 DEBUG }, 05:53:28 DEBUG "value": [ 05:53:28 DEBUG 05:53:28 DEBUG 05:53:28 DEBUG 05:53:28 DEBUG [...] 05:53:28 DEBUG 05:53:28 DEBUG 05:53:28 DEBUG 05:53:28 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:53:28 DEBUG }, 05:53:28 DEBUG "value": [ 05:53:28 DEBUG 1712724808, 05:53:28 DEBUG "2.000785428708816" 05:53:28 DEBUG ] 05:53:28 DEBUG } 05:53:28 DEBUG ] 05:53:28 DEBUG } 05:53:28 DEBUG } 05:53:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724808" 05:53:28 INFO [http_cmd]: http status code OK 05:53:28 DEBUG --- status code --- 05:53:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:28 DEBUG --- http response --- 05:53:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:53:28 DEBUG ----- output ----- 05:53:28 DEBUG { 05:53:28 DEBUG "status": "success", 05:53:28 DEBUG "data": { 05:53:28 DEBUG "resultType": "vector", 05:53:28 DEBUG "result": [ 05:53:28 DEBUG { 05:53:28 DEBUG "metric": { 05:53:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:53:28 DEBUG }, 05:53:28 DEBUG "value": [ 05:53:28 DEBUG 05:53:28 DEBUG 05:53:28 DEBUG 05:53:28 DEBUG [...] 05:53:28 DEBUG 05:53:28 DEBUG 05:53:28 DEBUG 05:53:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:53:28 DEBUG }, 05:53:28 DEBUG "value": [ 05:53:28 DEBUG 1712724808, 05:53:28 DEBUG "1423.4" 05:53:28 DEBUG ] 05:53:28 DEBUG } 05:53:28 DEBUG ] 05:53:28 DEBUG } 05:53:28 DEBUG } 05:53:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712724808" 05:53:28 INFO [http_cmd]: http status code OK 05:53:28 DEBUG --- status code --- 05:53:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:28 DEBUG --- http response --- 05:53:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:53:28 DEBUG ----- output ----- 05:53:28 DEBUG { 05:53:28 DEBUG "status": "success", 05:53:28 DEBUG "data": { 05:53:28 DEBUG "resultType": "vector", 05:53:28 DEBUG "result": [ 05:53:28 DEBUG { 05:53:28 DEBUG "metric": { 05:53:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:53:28 DEBUG }, 05:53:28 DEBUG "value": [ 05:53:28 DEBUG 05:53:28 DEBUG 05:53:28 DEBUG 05:53:28 DEBUG [...] 05:53:28 DEBUG 05:53:28 DEBUG 05:53:28 DEBUG 05:53:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:53:28 DEBUG }, 05:53:28 DEBUG "value": [ 05:53:28 DEBUG 1712724808, 05:53:28 DEBUG "0.045" 05:53:28 DEBUG ] 05:53:28 DEBUG } 05:53:28 DEBUG ] 05:53:28 DEBUG } 05:53:28 DEBUG } 05:53:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724808" 05:53:28 INFO [http_cmd]: http status code OK 05:53:28 DEBUG --- status code --- 05:53:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:28 DEBUG --- http response --- 05:53:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:53:28 DEBUG ----- output ----- 05:53:28 DEBUG { 05:53:28 DEBUG "status": "success", 05:53:28 DEBUG "data": { 05:53:28 DEBUG "resultType": "vector", 05:53:28 DEBUG "result": [ 05:53:28 DEBUG { 05:53:28 DEBUG "metric": { 05:53:28 DEBUG "pod": "ds-idrepo-0" 05:53:28 DEBUG }, 05:53:28 DEBUG "value": [ 05:53:28 DEBUG 05:53:28 DEBUG 05:53:28 DEBUG 05:53:28 DEBUG [...] 05:53:28 DEBUG 05:53:28 DEBUG 05:53:28 DEBUG 05:53:28 DEBUG "pod": "ds-idrepo-1" 05:53:28 DEBUG }, 05:53:28 DEBUG "value": [ 05:53:28 DEBUG 1712724808, 05:53:28 DEBUG "0" 05:53:28 DEBUG ] 05:53:28 DEBUG } 05:53:28 DEBUG ] 05:53:28 DEBUG } 05:53:28 DEBUG } 05:53:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724809" 05:53:29 INFO [http_cmd]: http status code OK 05:53:29 DEBUG --- status code --- 05:53:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:29 DEBUG --- http response --- 05:53:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:53:29 DEBUG ----- output ----- 05:53:29 DEBUG { 05:53:29 DEBUG "status": "success", 05:53:29 DEBUG "data": { 05:53:29 DEBUG "resultType": "vector", 05:53:29 DEBUG "result": [ 05:53:29 DEBUG { 05:53:29 DEBUG "metric": { 05:53:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:53:29 DEBUG }, 05:53:29 DEBUG "value": [ 05:53:29 DEBUG 05:53:29 DEBUG 05:53:29 DEBUG 05:53:29 DEBUG [...] 05:53:29 DEBUG 05:53:29 DEBUG 05:53:29 DEBUG 05:53:29 DEBUG "pod": "am-654c4fcbcf-487cv" 05:53:29 DEBUG }, 05:53:29 DEBUG "value": [ 05:53:29 DEBUG 1712724809, 05:53:29 DEBUG "1442.4333333333334" 05:53:29 DEBUG ] 05:53:29 DEBUG } 05:53:29 DEBUG ] 05:53:29 DEBUG } 05:53:29 DEBUG } 05:53:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724809" 05:53:29 INFO [http_cmd]: http status code OK 05:53:29 DEBUG --- status code --- 05:53:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:29 DEBUG --- http response --- 05:53:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:53:29 DEBUG ----- output ----- 05:53:29 DEBUG { 05:53:29 DEBUG "status": "success", 05:53:29 DEBUG "data": { 05:53:29 DEBUG "resultType": "vector", 05:53:29 DEBUG "result": [ 05:53:29 DEBUG { 05:53:29 DEBUG "metric": { 05:53:29 DEBUG "pod": "ds-idrepo-0" 05:53:29 DEBUG }, 05:53:29 DEBUG "value": [ 05:53:29 DEBUG 05:53:29 DEBUG 05:53:29 DEBUG 05:53:29 DEBUG [...] 05:53:29 DEBUG 05:53:29 DEBUG 05:53:29 DEBUG 05:53:29 DEBUG "pod": "ds-idrepo-1" 05:53:29 DEBUG }, 05:53:29 DEBUG "value": [ 05:53:29 DEBUG 1712724809, 05:53:29 DEBUG "0" 05:53:29 DEBUG ] 05:53:29 DEBUG } 05:53:29 DEBUG ] 05:53:29 DEBUG } 05:53:29 DEBUG } 05:53:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724809" 05:53:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712724809" 05:53:29 INFO [http_cmd]: http status code OK 05:53:29 DEBUG --- status code --- 05:53:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:29 DEBUG --- http response --- 05:53:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:53:29 DEBUG ----- output ----- 05:53:29 DEBUG { 05:53:29 DEBUG "status": "success", 05:53:29 DEBUG "data": { 05:53:29 DEBUG "resultType": "vector", 05:53:29 DEBUG "result": [ 05:53:29 DEBUG { 05:53:29 DEBUG "metric": { 05:53:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:53:29 DEBUG }, 05:53:29 DEBUG "value": [ 05:53:29 DEBUG 05:53:29 DEBUG 05:53:29 DEBUG 05:53:29 DEBUG [...] 05:53:29 DEBUG 05:53:29 DEBUG 05:53:29 DEBUG 05:53:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:53:29 DEBUG }, 05:53:29 DEBUG "value": [ 05:53:29 DEBUG 1712724809, 05:53:29 DEBUG "0" 05:53:29 DEBUG ] 05:53:29 DEBUG } 05:53:29 DEBUG ] 05:53:29 DEBUG } 05:53:29 DEBUG } 05:53:29 INFO [http_cmd]: http status code OK 05:53:29 DEBUG --- status code --- 05:53:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:29 DEBUG --- http response --- 05:53:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:53:29 DEBUG ----- output ----- 05:53:29 DEBUG { 05:53:29 DEBUG "status": "success", 05:53:29 DEBUG "data": { 05:53:29 DEBUG "resultType": "vector", 05:53:29 DEBUG "result": [ 05:53:29 DEBUG { 05:53:29 DEBUG "metric": { 05:53:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:53:29 DEBUG }, 05:53:29 DEBUG "value": [ 05:53:29 DEBUG 05:53:29 DEBUG 05:53:29 DEBUG 05:53:29 DEBUG [...] 05:53:29 DEBUG 05:53:29 DEBUG 05:53:29 DEBUG 05:53:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:53:29 DEBUG }, 05:53:29 DEBUG "value": [ 05:53:29 DEBUG 1712724809, 05:53:29 DEBUG "2487164928" 05:53:29 DEBUG ] 05:53:29 DEBUG } 05:53:29 DEBUG ] 05:53:29 DEBUG } 05:53:29 DEBUG } 05:53:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724809" 05:53:29 INFO [http_cmd]: http status code OK 05:53:29 DEBUG --- status code --- 05:53:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:29 DEBUG --- http response --- 05:53:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:53:29 DEBUG ----- output ----- 05:53:29 DEBUG { 05:53:29 DEBUG "status": "success", 05:53:29 DEBUG "data": { 05:53:29 DEBUG "resultType": "vector", 05:53:29 DEBUG "result": [ 05:53:29 DEBUG { 05:53:29 DEBUG "metric": { 05:53:29 DEBUG "instance": "10.142.0.25:9100" 05:53:29 DEBUG }, 05:53:29 DEBUG "value": [ 05:53:29 DEBUG 05:53:29 DEBUG 05:53:29 DEBUG 05:53:29 DEBUG [...] 05:53:29 DEBUG 05:53:29 DEBUG 05:53:29 DEBUG 05:53:29 DEBUG "instance": "10.142.0.77:9100" 05:53:29 DEBUG }, 05:53:29 DEBUG "value": [ 05:53:29 DEBUG 1712724809, 05:53:29 DEBUG "42313092.3" 05:53:29 DEBUG ] 05:53:29 DEBUG } 05:53:29 DEBUG ] 05:53:29 DEBUG } 05:53:29 DEBUG } 05:53:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724809" 05:53:29 INFO [http_cmd]: http status code OK 05:53:29 DEBUG --- status code --- 05:53:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:29 DEBUG --- http response --- 05:53:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:53:29 DEBUG ----- output ----- 05:53:29 DEBUG { 05:53:29 DEBUG "status": "success", 05:53:29 DEBUG "data": { 05:53:29 DEBUG "resultType": "vector", 05:53:29 DEBUG "result": [ 05:53:29 DEBUG { 05:53:29 DEBUG "metric": { 05:53:29 DEBUG "instance": "10.142.0.25:9100" 05:53:29 DEBUG }, 05:53:29 DEBUG "value": [ 05:53:29 DEBUG 05:53:29 DEBUG 05:53:29 DEBUG 05:53:29 DEBUG [...] 05:53:29 DEBUG 05:53:29 DEBUG 05:53:29 DEBUG 05:53:29 DEBUG "instance": "10.142.0.77:9100" 05:53:29 DEBUG }, 05:53:29 DEBUG "value": [ 05:53:29 DEBUG 1712724809, 05:53:29 DEBUG "0.05133333333333324" 05:53:29 DEBUG ] 05:53:29 DEBUG } 05:53:29 DEBUG ] 05:53:29 DEBUG } 05:53:29 DEBUG } 05:53:45 INFO 05:53:45 INFO [loop_until]: kubectl --namespace=xlou top pods 05:53:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:53:45 INFO [loop_until]: OK (rc = 0) 05:53:45 DEBUG --- stdout --- 05:53:45 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10817m 8958Mi am-654c4fcbcf-9scrj 9658m 8432Mi am-654c4fcbcf-v5qb5 8464m 8407Mi ds-cts-0 8346m 13303Mi ds-cts-1 8650m 13334Mi ds-cts-2 8291m 13374Mi ds-idrepo-0 10855m 5622Mi ds-idrepo-1 350m 5672Mi ds-idrepo-2 394m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 8m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4198m 2228Mi 05:53:45 DEBUG --- stderr --- 05:53:45 DEBUG 05:53:52 INFO 05:53:53 INFO [loop_until]: kubectl --namespace=xlou top node 05:53:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:53:53 INFO [loop_until]: OK (rc = 0) 05:53:53 DEBUG --- stdout --- 05:53:53 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 92m 0% 2305Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9252m 58% 9282Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 77m 0% 3257Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13057m 82% 6565Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10851m 68% 9315Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11034m 69% 9782Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 74m 0% 853Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10789m 67% 6291Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8303m 52% 14027Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 8225m 51% 14009Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 430m 2% 6377Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8684m 54% 13876Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 414m 2% 6339Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4234m 26% 5891Mi 10% 05:53:53 DEBUG --- stderr --- 05:53:53 DEBUG 05:54:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724865" 05:54:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712724865" 05:54:25 ERROR [http_cmd]: ERROR 05:54:25 INFO [http_cmd]: http status code OK 05:54:25 ERROR --- status code --- 05:54:25 DEBUG --- status code --- 05:54:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:54:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:25 DEBUG --- http response --- 05:54:25 ERROR --- http response --- 05:54:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:54:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:54:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:54:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724865" 05:54:25 INFO [http_cmd]: http status code OK 05:54:25 DEBUG --- status code --- 05:54:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:25 DEBUG --- http response --- 05:54:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:54:25 DEBUG ----- output ----- 05:54:25 DEBUG { 05:54:25 DEBUG "status": "success", 05:54:25 DEBUG "data": { 05:54:25 DEBUG "resultType": "vector", 05:54:25 DEBUG "result": [ 05:54:25 DEBUG { 05:54:25 DEBUG "metric": { 05:54:25 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:54:25 DEBUG }, 05:54:25 DEBUG "value": [ 05:54:25 DEBUG 05:54:25 DEBUG 05:54:25 DEBUG 05:54:25 DEBUG [...] 05:54:25 DEBUG 05:54:25 DEBUG 05:54:25 DEBUG 05:54:25 DEBUG "pod": "am-654c4fcbcf-487cv" 05:54:25 DEBUG }, 05:54:25 DEBUG "value": [ 05:54:25 DEBUG 1712724865, 05:54:25 DEBUG "0" 05:54:25 DEBUG ] 05:54:25 DEBUG } 05:54:25 DEBUG ] 05:54:25 DEBUG } 05:54:25 DEBUG } 05:54:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724866" 05:54:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724866" 05:54:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724866" 05:54:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724866" 05:54:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724866" 05:54:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724866" 05:54:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724866" 05:54:26 INFO [http_cmd]: http status code OK 05:54:26 DEBUG --- status code --- 05:54:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:26 DEBUG --- http response --- 05:54:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:54:26 DEBUG ----- output ----- 05:54:26 DEBUG { 05:54:26 DEBUG "status": "success", 05:54:26 DEBUG "data": { 05:54:26 DEBUG "resultType": "vector", 05:54:26 DEBUG "result": [ 05:54:26 INFO [http_cmd]: http status code OK 05:54:26 DEBUG { 05:54:26 DEBUG "metric": { 05:54:26 DEBUG --- status code --- 05:54:26 INFO [http_cmd]: http status code OK 05:54:26 DEBUG --- status code --- 05:54:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:26 DEBUG --- http response --- 05:54:26 DEBUG --- http response --- 05:54:26 DEBUG "pod": "ds-idrepo-0" 05:54:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:54:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:54:26 DEBUG }, 05:54:26 DEBUG ----- output ----- 05:54:26 DEBUG ----- output ----- 05:54:26 DEBUG "value": [ 05:54:26 DEBUG { 05:54:26 DEBUG "status": "success", 05:54:26 DEBUG 05:54:26 DEBUG { 05:54:26 DEBUG "data": { 05:54:26 DEBUG 05:54:26 DEBUG "status": "success", 05:54:26 DEBUG "data": { 05:54:26 DEBUG "resultType": "vector", 05:54:26 DEBUG "result": [ 05:54:26 DEBUG { 05:54:26 DEBUG "resultType": "vector", 05:54:26 DEBUG "result": [ 05:54:26 DEBUG { 05:54:26 DEBUG "metric": { 05:54:26 DEBUG "pod": "ds-idrepo-0" 05:54:26 DEBUG }, 05:54:26 DEBUG "value": [ 05:54:26 DEBUG 05:54:26 DEBUG 05:54:26 DEBUG 05:54:26 DEBUG [...] 05:54:26 DEBUG 05:54:26 DEBUG [...] 05:54:26 DEBUG 05:54:26 INFO [http_cmd]: http status code OK 05:54:26 DEBUG "metric": { 05:54:26 INFO [http_cmd]: http status code OK 05:54:26 DEBUG 05:54:26 DEBUG 05:54:26 DEBUG --- status code --- 05:54:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:54:26 DEBUG --- status code --- 05:54:26 INFO [http_cmd]: http status code OK 05:54:26 DEBUG --- status code --- 05:54:26 DEBUG 05:54:26 DEBUG 05:54:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:26 DEBUG }, 05:54:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:26 INFO [http_cmd]: http status code OK 05:54:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:26 DEBUG 05:54:26 DEBUG "pod": "ds-cts-2" 05:54:26 DEBUG --- http response --- 05:54:26 DEBUG "value": [ 05:54:26 DEBUG --- http response --- 05:54:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:54:26 DEBUG --- http response --- 05:54:26 DEBUG "pod": "ds-cts-2" 05:54:26 DEBUG }, 05:54:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:54:26 DEBUG 05:54:26 DEBUG ----- output ----- 05:54:26 DEBUG { 05:54:26 DEBUG "status": "success", 05:54:26 DEBUG }, 05:54:26 DEBUG "value": [ 05:54:26 DEBUG ----- output ----- 05:54:26 DEBUG 05:54:26 DEBUG "data": { 05:54:26 DEBUG "resultType": "vector", 05:54:26 DEBUG "result": [ 05:54:26 DEBUG "value": [ 05:54:26 DEBUG 1712724866, 05:54:26 DEBUG { 05:54:26 DEBUG 05:54:26 DEBUG { 05:54:26 DEBUG "metric": { 05:54:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:54:26 DEBUG 1712724866, 05:54:26 DEBUG "0" 05:54:26 DEBUG "status": "success", 05:54:26 DEBUG "data": { 05:54:26 DEBUG "resultType": "vector", 05:54:26 DEBUG "result": [ 05:54:26 DEBUG { 05:54:26 DEBUG "29.233333333333334" 05:54:26 DEBUG ] 05:54:26 DEBUG "metric": { 05:54:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:54:26 DEBUG }, 05:54:26 DEBUG "value": [ 05:54:26 DEBUG 05:54:26 DEBUG ] 05:54:26 DEBUG } 05:54:26 DEBUG 05:54:26 DEBUG 05:54:26 DEBUG [...] 05:54:26 DEBUG 05:54:26 DEBUG 05:54:26 DEBUG } 05:54:26 DEBUG ] 05:54:26 DEBUG 05:54:26 DEBUG --- status code --- 05:54:26 DEBUG [...] 05:54:26 DEBUG 05:54:26 DEBUG 05:54:26 DEBUG ] 05:54:26 DEBUG } 05:54:26 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:54:26 DEBUG }, 05:54:26 DEBUG "value": [ 05:54:26 DEBUG 1712724866, 05:54:26 DEBUG 05:54:26 DEBUG } 05:54:26 DEBUG } 05:54:26 DEBUG "0" 05:54:26 DEBUG ] 05:54:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:54:26 DEBUG ----- output ----- 05:54:26 DEBUG { 05:54:26 DEBUG } 05:54:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:26 DEBUG } 05:54:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:54:26 DEBUG "status": "success", 05:54:26 DEBUG }, 05:54:26 DEBUG "value": [ 05:54:26 DEBUG 05:54:26 DEBUG }, 05:54:26 DEBUG "data": { 05:54:26 DEBUG --- http response --- 05:54:26 DEBUG 05:54:26 DEBUG ] 05:54:26 DEBUG "value": [ 05:54:26 DEBUG "resultType": "vector", 05:54:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:54:26 DEBUG 05:54:26 DEBUG } 05:54:26 DEBUG 1712724866, 05:54:26 DEBUG "result": [ 05:54:26 DEBUG ----- output ----- 05:54:26 DEBUG [...] 05:54:26 DEBUG } 05:54:26 DEBUG "1431.4333333333334" 05:54:26 DEBUG { 05:54:26 DEBUG { 05:54:26 DEBUG 05:54:26 DEBUG ] 05:54:26 DEBUG "metric": { 05:54:26 DEBUG "status": "success", 05:54:26 DEBUG 05:54:26 DEBUG } 05:54:26 DEBUG "instance": "10.142.0.25:9100" 05:54:26 DEBUG }, 05:54:26 DEBUG 05:54:26 DEBUG ] 05:54:26 DEBUG "value": [ 05:54:26 DEBUG "data": { 05:54:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:54:26 DEBUG } 05:54:26 DEBUG 05:54:26 DEBUG "resultType": "vector", 05:54:26 DEBUG }, 05:54:26 DEBUG "value": [ 05:54:26 DEBUG 1712724866, 05:54:26 DEBUG "258356.5" 05:54:26 DEBUG } 05:54:26 DEBUG "result": [ 05:54:26 DEBUG ] 05:54:26 DEBUG 05:54:26 DEBUG { 05:54:26 DEBUG } 05:54:26 DEBUG 05:54:26 DEBUG [...] 05:54:26 DEBUG ] 05:54:26 DEBUG } 05:54:26 DEBUG 05:54:26 DEBUG } 05:54:26 DEBUG "metric": { 05:54:26 DEBUG 05:54:26 DEBUG "instance": "10.142.0.25:9100" 05:54:26 DEBUG 05:54:26 DEBUG }, 05:54:26 DEBUG "instance": "10.142.0.77:9100" 05:54:26 DEBUG "value": [ 05:54:26 DEBUG }, 05:54:26 DEBUG 05:54:26 DEBUG "value": [ 05:54:26 DEBUG 05:54:26 DEBUG 1712724866, 05:54:26 DEBUG 05:54:26 DEBUG "42501112.06666666" 05:54:26 DEBUG [...] 05:54:26 DEBUG ] 05:54:26 DEBUG } 05:54:26 DEBUG ] 05:54:26 DEBUG } 05:54:26 DEBUG } 05:54:26 DEBUG 05:54:26 DEBUG 05:54:26 DEBUG 05:54:26 DEBUG "instance": "10.142.0.77:9100" 05:54:26 DEBUG }, 05:54:26 DEBUG "value": [ 05:54:26 DEBUG 1712724866, 05:54:26 DEBUG "328026440704" 05:54:26 DEBUG ] 05:54:26 DEBUG } 05:54:26 DEBUG ] 05:54:26 DEBUG } 05:54:26 DEBUG } 05:54:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724866" 05:54:26 INFO [http_cmd]: http status code OK 05:54:26 DEBUG --- status code --- 05:54:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:26 DEBUG --- http response --- 05:54:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:54:26 DEBUG ----- output ----- 05:54:26 DEBUG { 05:54:26 DEBUG "status": "success", 05:54:26 DEBUG "data": { 05:54:26 DEBUG "resultType": "vector", 05:54:26 DEBUG "result": [ 05:54:26 DEBUG { 05:54:26 DEBUG "metric": { 05:54:26 DEBUG "pod": "ds-idrepo-0" 05:54:26 DEBUG }, 05:54:26 DEBUG "value": [ 05:54:26 DEBUG 05:54:26 DEBUG 05:54:26 DEBUG 05:54:26 DEBUG [...] 05:54:26 DEBUG 05:54:26 DEBUG 05:54:26 DEBUG 05:54:26 DEBUG "pod": "ds-idrepo-1" 05:54:26 DEBUG }, 05:54:26 DEBUG "value": [ 05:54:26 DEBUG 1712724866, 05:54:26 DEBUG "0" 05:54:26 DEBUG ] 05:54:26 DEBUG } 05:54:26 DEBUG ] 05:54:26 DEBUG } 05:54:26 DEBUG } 05:54:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724866" 05:54:26 INFO [http_cmd]: http status code OK 05:54:26 DEBUG --- status code --- 05:54:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:26 DEBUG --- http response --- 05:54:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:54:26 DEBUG ----- output ----- 05:54:26 DEBUG { 05:54:26 DEBUG "status": "success", 05:54:26 DEBUG "data": { 05:54:26 DEBUG "resultType": "vector", 05:54:26 DEBUG "result": [ 05:54:26 DEBUG { 05:54:26 DEBUG "metric": { 05:54:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:54:26 DEBUG }, 05:54:26 DEBUG "value": [ 05:54:26 DEBUG 05:54:26 DEBUG 05:54:26 DEBUG 05:54:26 DEBUG [...] 05:54:26 DEBUG 05:54:26 DEBUG 05:54:26 DEBUG 05:54:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:54:26 DEBUG }, 05:54:26 DEBUG "value": [ 05:54:26 DEBUG 1712724866, 05:54:26 DEBUG "1435.2666666666667" 05:54:26 DEBUG ] 05:54:26 DEBUG } 05:54:26 DEBUG ] 05:54:26 DEBUG } 05:54:26 DEBUG } 05:54:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724867" 05:54:27 INFO [http_cmd]: http status code OK 05:54:27 DEBUG --- status code --- 05:54:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:27 DEBUG --- http response --- 05:54:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:54:27 DEBUG ----- output ----- 05:54:27 DEBUG { 05:54:27 DEBUG "status": "success", 05:54:27 DEBUG "data": { 05:54:27 DEBUG "resultType": "vector", 05:54:27 DEBUG "result": [ 05:54:27 DEBUG { 05:54:27 DEBUG "metric": { 05:54:27 DEBUG "instance": "10.142.0.25:9100" 05:54:27 DEBUG }, 05:54:27 DEBUG "value": [ 05:54:27 DEBUG 05:54:27 DEBUG 05:54:27 DEBUG 05:54:27 DEBUG [...] 05:54:27 DEBUG 05:54:27 DEBUG 05:54:27 DEBUG 05:54:27 DEBUG "instance": "10.142.0.77:9100" 05:54:27 DEBUG }, 05:54:27 DEBUG "value": [ 05:54:27 DEBUG 1712724867, 05:54:27 DEBUG "0.15426666666666786" 05:54:27 DEBUG ] 05:54:27 DEBUG } 05:54:27 DEBUG ] 05:54:27 DEBUG } 05:54:27 DEBUG } 05:54:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724867" 05:54:27 INFO [http_cmd]: http status code OK 05:54:27 DEBUG --- status code --- 05:54:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:27 DEBUG --- http response --- 05:54:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:54:27 DEBUG ----- output ----- 05:54:27 DEBUG { 05:54:27 DEBUG "status": "success", 05:54:27 DEBUG "data": { 05:54:27 DEBUG "resultType": "vector", 05:54:27 DEBUG "result": [ 05:54:27 DEBUG { 05:54:27 DEBUG "metric": { 05:54:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:54:27 DEBUG }, 05:54:27 DEBUG "value": [ 05:54:27 DEBUG 05:54:27 DEBUG 05:54:27 DEBUG 05:54:27 DEBUG [...] 05:54:27 DEBUG 05:54:27 DEBUG 05:54:27 DEBUG 05:54:27 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:54:27 DEBUG }, 05:54:27 DEBUG "value": [ 05:54:27 DEBUG 1712724867, 05:54:27 DEBUG "29212.35736153632" 05:54:27 DEBUG ] 05:54:27 DEBUG } 05:54:27 DEBUG ] 05:54:27 DEBUG } 05:54:27 DEBUG } 05:54:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712724868" 05:54:28 INFO [http_cmd]: http status code OK 05:54:28 DEBUG --- status code --- 05:54:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:28 DEBUG --- http response --- 05:54:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:54:28 DEBUG ----- output ----- 05:54:28 DEBUG { 05:54:28 DEBUG "status": "success", 05:54:28 DEBUG "data": { 05:54:28 DEBUG "resultType": "vector", 05:54:28 DEBUG "result": [ 05:54:28 DEBUG { 05:54:28 DEBUG "metric": { 05:54:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:54:28 DEBUG }, 05:54:28 DEBUG "value": [ 05:54:28 DEBUG 05:54:28 DEBUG 05:54:28 DEBUG 05:54:28 DEBUG [...] 05:54:28 DEBUG 05:54:28 DEBUG 05:54:28 DEBUG 05:54:28 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:54:28 DEBUG }, 05:54:28 DEBUG "value": [ 05:54:28 DEBUG 1712724868, 05:54:28 DEBUG "4.1284316270433" 05:54:28 DEBUG ] 05:54:28 DEBUG } 05:54:28 DEBUG ] 05:54:28 DEBUG } 05:54:28 DEBUG } 05:54:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712724868" 05:54:28 INFO [http_cmd]: http status code OK 05:54:28 DEBUG --- status code --- 05:54:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:28 DEBUG --- http response --- 05:54:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:54:28 DEBUG ----- output ----- 05:54:28 DEBUG { 05:54:28 DEBUG "status": "success", 05:54:28 DEBUG "data": { 05:54:28 DEBUG "resultType": "vector", 05:54:28 DEBUG "result": [ 05:54:28 DEBUG { 05:54:28 DEBUG "metric": { 05:54:28 DEBUG "pod": "ds-idrepo-0" 05:54:28 DEBUG }, 05:54:28 DEBUG "value": [ 05:54:28 DEBUG 05:54:28 DEBUG 05:54:28 DEBUG 05:54:28 DEBUG [...] 05:54:28 DEBUG 05:54:28 DEBUG 05:54:28 DEBUG 05:54:28 DEBUG "pod": "ds-cts-2" 05:54:28 DEBUG }, 05:54:28 DEBUG "value": [ 05:54:28 DEBUG 1712724868, 05:54:28 DEBUG "0" 05:54:28 DEBUG ] 05:54:28 DEBUG } 05:54:28 DEBUG ] 05:54:28 DEBUG } 05:54:28 DEBUG } 05:54:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724868" 05:54:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724868" 05:54:28 INFO [http_cmd]: http status code OK 05:54:28 DEBUG --- status code --- 05:54:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:28 DEBUG --- http response --- 05:54:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:54:28 DEBUG ----- output ----- 05:54:28 DEBUG { 05:54:28 DEBUG "status": "success", 05:54:28 DEBUG "data": { 05:54:28 DEBUG "resultType": "vector", 05:54:28 DEBUG "result": [ 05:54:28 DEBUG { 05:54:28 DEBUG "metric": { 05:54:28 DEBUG "pod": "ds-idrepo-2" 05:54:28 DEBUG }, 05:54:28 DEBUG "value": [ 05:54:28 DEBUG 05:54:28 DEBUG 05:54:28 DEBUG 05:54:28 DEBUG [...] 05:54:28 DEBUG 05:54:28 DEBUG 05:54:28 DEBUG 05:54:28 DEBUG "pod": "ds-cts-2" 05:54:28 DEBUG }, 05:54:28 DEBUG "value": [ 05:54:28 DEBUG 1712724868, 05:54:28 DEBUG "0.000055107777777777756" 05:54:28 DEBUG ] 05:54:28 DEBUG } 05:54:28 DEBUG ] 05:54:28 DEBUG } 05:54:28 DEBUG } 05:54:28 INFO [http_cmd]: http status code OK 05:54:28 DEBUG --- status code --- 05:54:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:28 DEBUG --- http response --- 05:54:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:54:28 DEBUG ----- output ----- 05:54:28 DEBUG { 05:54:28 DEBUG "status": "success", 05:54:28 DEBUG "data": { 05:54:28 DEBUG "resultType": "vector", 05:54:28 DEBUG "result": [ 05:54:28 DEBUG { 05:54:28 DEBUG "metric": { 05:54:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:54:28 DEBUG }, 05:54:28 DEBUG "value": [ 05:54:28 DEBUG 05:54:28 DEBUG 05:54:28 DEBUG 05:54:28 DEBUG [...] 05:54:28 DEBUG 05:54:28 DEBUG 05:54:28 DEBUG 05:54:28 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:54:28 DEBUG }, 05:54:28 DEBUG "value": [ 05:54:28 DEBUG 1712724868, 05:54:28 DEBUG "4.1090646288965225" 05:54:28 DEBUG ] 05:54:28 DEBUG } 05:54:28 DEBUG ] 05:54:28 DEBUG } 05:54:28 DEBUG } 05:54:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724868" 05:54:28 INFO [http_cmd]: http status code OK 05:54:28 DEBUG --- status code --- 05:54:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:28 DEBUG --- http response --- 05:54:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:54:28 DEBUG ----- output ----- 05:54:28 DEBUG { 05:54:28 DEBUG "status": "success", 05:54:28 DEBUG "data": { 05:54:28 DEBUG "resultType": "vector", 05:54:28 DEBUG "result": [ 05:54:28 DEBUG { 05:54:28 DEBUG "metric": { 05:54:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:54:28 DEBUG }, 05:54:28 DEBUG "value": [ 05:54:28 DEBUG 05:54:28 DEBUG 05:54:28 DEBUG 05:54:28 DEBUG [...] 05:54:28 DEBUG 05:54:28 DEBUG 05:54:28 DEBUG 05:54:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:54:28 DEBUG }, 05:54:28 DEBUG "value": [ 05:54:28 DEBUG 1712724868, 05:54:28 DEBUG "1434.7666666666667" 05:54:28 DEBUG ] 05:54:28 DEBUG } 05:54:28 DEBUG ] 05:54:28 DEBUG } 05:54:28 DEBUG } 05:54:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712724868" 05:54:28 INFO [http_cmd]: http status code OK 05:54:28 DEBUG --- status code --- 05:54:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:28 DEBUG --- http response --- 05:54:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:54:28 DEBUG ----- output ----- 05:54:28 DEBUG { 05:54:28 DEBUG "status": "success", 05:54:28 DEBUG "data": { 05:54:28 DEBUG "resultType": "vector", 05:54:28 DEBUG "result": [ 05:54:28 DEBUG { 05:54:28 DEBUG "metric": { 05:54:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:54:28 DEBUG }, 05:54:28 DEBUG "value": [ 05:54:28 DEBUG 05:54:28 DEBUG 05:54:28 DEBUG 05:54:28 DEBUG [...] 05:54:28 DEBUG 05:54:28 DEBUG 05:54:28 DEBUG 05:54:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:54:28 DEBUG }, 05:54:28 DEBUG "value": [ 05:54:28 DEBUG 1712724868, 05:54:28 DEBUG "0.046" 05:54:28 DEBUG ] 05:54:28 DEBUG } 05:54:28 DEBUG ] 05:54:28 DEBUG } 05:54:28 DEBUG } 05:54:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724868" 05:54:28 INFO [http_cmd]: http status code OK 05:54:28 DEBUG --- status code --- 05:54:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:28 DEBUG --- http response --- 05:54:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:54:28 DEBUG ----- output ----- 05:54:28 DEBUG { 05:54:28 DEBUG "status": "success", 05:54:28 DEBUG "data": { 05:54:28 DEBUG "resultType": "vector", 05:54:28 DEBUG "result": [ 05:54:28 DEBUG { 05:54:28 DEBUG "metric": { 05:54:28 DEBUG "pod": "ds-idrepo-0" 05:54:28 DEBUG }, 05:54:28 DEBUG "value": [ 05:54:28 DEBUG 05:54:28 DEBUG 05:54:28 DEBUG 05:54:28 DEBUG [...] 05:54:28 DEBUG 05:54:28 DEBUG 05:54:28 DEBUG 05:54:28 DEBUG "pod": "ds-idrepo-1" 05:54:28 DEBUG }, 05:54:28 DEBUG "value": [ 05:54:28 DEBUG 1712724868, 05:54:28 DEBUG "0" 05:54:28 DEBUG ] 05:54:28 DEBUG } 05:54:28 DEBUG ] 05:54:28 DEBUG } 05:54:28 DEBUG } 05:54:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724869" 05:54:29 INFO [http_cmd]: http status code OK 05:54:29 DEBUG --- status code --- 05:54:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:29 DEBUG --- http response --- 05:54:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:54:29 DEBUG ----- output ----- 05:54:29 DEBUG { 05:54:29 DEBUG "status": "success", 05:54:29 DEBUG "data": { 05:54:29 DEBUG "resultType": "vector", 05:54:29 DEBUG "result": [ 05:54:29 DEBUG { 05:54:29 DEBUG "metric": { 05:54:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:54:29 DEBUG }, 05:54:29 DEBUG "value": [ 05:54:29 DEBUG 05:54:29 DEBUG 05:54:29 DEBUG 05:54:29 DEBUG [...] 05:54:29 DEBUG 05:54:29 DEBUG 05:54:29 DEBUG 05:54:29 DEBUG "pod": "am-654c4fcbcf-487cv" 05:54:29 DEBUG }, 05:54:29 DEBUG "value": [ 05:54:29 DEBUG 1712724869, 05:54:29 DEBUG "1426.7" 05:54:29 DEBUG ] 05:54:29 DEBUG } 05:54:29 DEBUG ] 05:54:29 DEBUG } 05:54:29 DEBUG } 05:54:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724869" 05:54:29 INFO [http_cmd]: http status code OK 05:54:29 DEBUG --- status code --- 05:54:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:29 DEBUG --- http response --- 05:54:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:54:29 DEBUG ----- output ----- 05:54:29 DEBUG { 05:54:29 DEBUG "status": "success", 05:54:29 DEBUG "data": { 05:54:29 DEBUG "resultType": "vector", 05:54:29 DEBUG "result": [ 05:54:29 DEBUG { 05:54:29 DEBUG "metric": { 05:54:29 DEBUG "pod": "ds-idrepo-0" 05:54:29 DEBUG }, 05:54:29 DEBUG "value": [ 05:54:29 DEBUG 05:54:29 DEBUG 05:54:29 DEBUG 05:54:29 DEBUG [...] 05:54:29 DEBUG 05:54:29 DEBUG 05:54:29 DEBUG 05:54:29 DEBUG "pod": "ds-idrepo-1" 05:54:29 DEBUG }, 05:54:29 DEBUG "value": [ 05:54:29 DEBUG 1712724869, 05:54:29 DEBUG "0" 05:54:29 DEBUG ] 05:54:29 DEBUG } 05:54:29 DEBUG ] 05:54:29 DEBUG } 05:54:29 DEBUG } 05:54:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724869" 05:54:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712724869" 05:54:29 INFO [http_cmd]: http status code OK 05:54:29 DEBUG --- status code --- 05:54:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:29 DEBUG --- http response --- 05:54:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:54:29 DEBUG ----- output ----- 05:54:29 DEBUG { 05:54:29 DEBUG "status": "success", 05:54:29 DEBUG "data": { 05:54:29 DEBUG "resultType": "vector", 05:54:29 DEBUG "result": [ 05:54:29 DEBUG { 05:54:29 DEBUG "metric": { 05:54:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:54:29 DEBUG }, 05:54:29 DEBUG "value": [ 05:54:29 DEBUG 05:54:29 DEBUG 05:54:29 DEBUG 05:54:29 DEBUG [...] 05:54:29 DEBUG 05:54:29 DEBUG 05:54:29 DEBUG 05:54:29 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:54:29 DEBUG }, 05:54:29 DEBUG "value": [ 05:54:29 DEBUG 1712724869, 05:54:29 DEBUG "0" 05:54:29 DEBUG ] 05:54:29 DEBUG } 05:54:29 DEBUG ] 05:54:29 DEBUG } 05:54:29 DEBUG } 05:54:29 INFO [http_cmd]: http status code OK 05:54:29 DEBUG --- status code --- 05:54:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:29 DEBUG --- http response --- 05:54:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:54:29 DEBUG ----- output ----- 05:54:29 DEBUG { 05:54:29 DEBUG "status": "success", 05:54:29 DEBUG "data": { 05:54:29 DEBUG "resultType": "vector", 05:54:29 DEBUG "result": [ 05:54:29 DEBUG { 05:54:29 DEBUG "metric": { 05:54:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:54:29 DEBUG }, 05:54:29 DEBUG "value": [ 05:54:29 DEBUG 05:54:29 DEBUG 05:54:29 DEBUG 05:54:29 DEBUG [...] 05:54:29 DEBUG 05:54:29 DEBUG 05:54:29 DEBUG 05:54:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:54:29 DEBUG }, 05:54:29 DEBUG "value": [ 05:54:29 DEBUG 1712724869, 05:54:29 DEBUG "2489233408" 05:54:29 DEBUG ] 05:54:29 DEBUG } 05:54:29 DEBUG ] 05:54:29 DEBUG } 05:54:29 DEBUG } 05:54:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724869" 05:54:29 INFO [http_cmd]: http status code OK 05:54:29 DEBUG --- status code --- 05:54:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:29 DEBUG --- http response --- 05:54:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:54:29 DEBUG ----- output ----- 05:54:29 DEBUG { 05:54:29 DEBUG "status": "success", 05:54:29 DEBUG "data": { 05:54:29 DEBUG "resultType": "vector", 05:54:29 DEBUG "result": [ 05:54:29 DEBUG { 05:54:29 DEBUG "metric": { 05:54:29 DEBUG "instance": "10.142.0.25:9100" 05:54:29 DEBUG }, 05:54:29 DEBUG "value": [ 05:54:29 DEBUG 05:54:29 DEBUG 05:54:29 DEBUG 05:54:29 DEBUG [...] 05:54:29 DEBUG 05:54:29 DEBUG 05:54:29 DEBUG 05:54:29 DEBUG "instance": "10.142.0.77:9100" 05:54:29 DEBUG }, 05:54:29 DEBUG "value": [ 05:54:29 DEBUG 1712724869, 05:54:29 DEBUG "42502799.2" 05:54:29 DEBUG ] 05:54:29 DEBUG } 05:54:29 DEBUG ] 05:54:29 DEBUG } 05:54:29 DEBUG } 05:54:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724869" 05:54:29 INFO [http_cmd]: http status code OK 05:54:29 DEBUG --- status code --- 05:54:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:29 DEBUG --- http response --- 05:54:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:54:29 DEBUG ----- output ----- 05:54:29 DEBUG { 05:54:29 DEBUG "status": "success", 05:54:29 DEBUG "data": { 05:54:29 DEBUG "resultType": "vector", 05:54:29 DEBUG "result": [ 05:54:29 DEBUG { 05:54:29 DEBUG "metric": { 05:54:29 DEBUG "instance": "10.142.0.25:9100" 05:54:29 DEBUG }, 05:54:29 DEBUG "value": [ 05:54:29 DEBUG 05:54:29 DEBUG 05:54:29 DEBUG 05:54:29 DEBUG [...] 05:54:29 DEBUG 05:54:29 DEBUG 05:54:29 DEBUG 05:54:29 DEBUG "instance": "10.142.0.77:9100" 05:54:29 DEBUG }, 05:54:29 DEBUG "value": [ 05:54:29 DEBUG 1712724869, 05:54:29 DEBUG "0.047333333333333866" 05:54:29 DEBUG ] 05:54:29 DEBUG } 05:54:29 DEBUG ] 05:54:29 DEBUG } 05:54:29 DEBUG } 05:54:45 INFO 05:54:45 INFO [loop_until]: kubectl --namespace=xlou top pods 05:54:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:54:45 INFO [loop_until]: OK (rc = 0) 05:54:45 DEBUG --- stdout --- 05:54:45 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10268m 8957Mi am-654c4fcbcf-9scrj 9898m 8434Mi am-654c4fcbcf-v5qb5 8312m 8409Mi ds-cts-0 8435m 12708Mi ds-cts-1 8781m 12717Mi ds-cts-2 7780m 12958Mi ds-idrepo-0 10558m 5622Mi ds-idrepo-1 358m 5671Mi ds-idrepo-2 425m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 8m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4165m 2229Mi 05:54:45 DEBUG --- stderr --- 05:54:45 DEBUG 05:54:53 INFO 05:54:53 INFO [loop_until]: kubectl --namespace=xlou top node 05:54:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:54:53 INFO [loop_until]: OK (rc = 0) 05:54:53 DEBUG --- stdout --- 05:54:53 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 99m 0% 2317Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9209m 57% 9287Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 76m 0% 3262Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13120m 82% 6535Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10858m 68% 9315Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11358m 71% 9787Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 74m 0% 856Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10817m 68% 6294Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8351m 52% 13440Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 7884m 49% 13675Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 481m 3% 6380Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8697m 54% 13372Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 416m 2% 6340Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4264m 26% 5887Mi 10% 05:54:53 DEBUG --- stderr --- 05:54:53 DEBUG 05:55:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712724925" 05:55:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724925" 05:55:25 ERROR [http_cmd]: ERROR 05:55:25 ERROR --- status code --- 05:55:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:55:25 ERROR --- http response --- 05:55:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:55:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:55:25 INFO [http_cmd]: http status code OK 05:55:25 DEBUG --- status code --- 05:55:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:25 DEBUG --- http response --- 05:55:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:55:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724925" 05:55:26 INFO [http_cmd]: http status code OK 05:55:26 DEBUG --- status code --- 05:55:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:26 DEBUG --- http response --- 05:55:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:55:26 DEBUG ----- output ----- 05:55:26 DEBUG { 05:55:26 DEBUG "status": "success", 05:55:26 DEBUG "data": { 05:55:26 DEBUG "resultType": "vector", 05:55:26 DEBUG "result": [ 05:55:26 DEBUG { 05:55:26 DEBUG "metric": { 05:55:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:55:26 DEBUG }, 05:55:26 DEBUG "value": [ 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG [...] 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:55:26 DEBUG }, 05:55:26 DEBUG "value": [ 05:55:26 DEBUG 1712724925, 05:55:26 DEBUG "0" 05:55:26 DEBUG ] 05:55:26 DEBUG } 05:55:26 DEBUG ] 05:55:26 DEBUG } 05:55:26 DEBUG } 05:55:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724926" 05:55:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724926" 05:55:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724926" 05:55:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724926" 05:55:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724926" 05:55:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724926" 05:55:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724926" 05:55:26 INFO [http_cmd]: http status code OK 05:55:26 INFO [http_cmd]: http status code OK 05:55:26 INFO [http_cmd]: http status code OK 05:55:26 INFO [http_cmd]: http status code OK 05:55:26 DEBUG --- status code --- 05:55:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:26 DEBUG --- http response --- 05:55:26 DEBUG --- status code --- 05:55:26 DEBUG --- status code --- 05:55:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:26 DEBUG --- http response --- 05:55:26 DEBUG --- status code --- 05:55:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:26 DEBUG --- http response --- 05:55:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:55:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:55:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:26 DEBUG --- http response --- 05:55:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:55:26 DEBUG ----- output ----- 05:55:26 DEBUG ----- output ----- 05:55:26 INFO [http_cmd]: http status code OK 05:55:26 DEBUG --- status code --- 05:55:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:26 DEBUG --- http response --- 05:55:26 DEBUG { 05:55:26 DEBUG ----- output ----- 05:55:26 DEBUG { 05:55:26 DEBUG "status": "success", 05:55:26 DEBUG "status": "success", 05:55:26 DEBUG "data": { 05:55:26 DEBUG "resultType": "vector", 05:55:26 DEBUG "result": [ 05:55:26 DEBUG { 05:55:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:55:26 DEBUG ----- output ----- 05:55:26 DEBUG "data": { 05:55:26 DEBUG "resultType": "vector", 05:55:26 INFO [http_cmd]: http status code OK 05:55:26 INFO [http_cmd]: http status code OK 05:55:26 DEBUG "metric": { 05:55:26 DEBUG { 05:55:26 DEBUG "status": "success", 05:55:26 DEBUG "data": { 05:55:26 DEBUG "resultType": "vector", 05:55:26 DEBUG --- status code --- 05:55:26 DEBUG --- status code --- 05:55:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:26 DEBUG --- http response --- 05:55:26 DEBUG "result": [ 05:55:26 DEBUG { 05:55:26 DEBUG "metric": { 05:55:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:26 DEBUG "result": [ 05:55:26 DEBUG { 05:55:26 DEBUG "metric": { 05:55:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:55:26 DEBUG }, 05:55:26 DEBUG "value": [ 05:55:26 DEBUG --- http response --- 05:55:26 DEBUG 05:55:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:55:26 DEBUG ----- output ----- 05:55:26 DEBUG { 05:55:26 DEBUG "status": "success", 05:55:26 DEBUG "data": { 05:55:26 DEBUG "resultType": "vector", 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG [...] 05:55:26 DEBUG "result": [ 05:55:26 DEBUG { 05:55:26 DEBUG "metric": { 05:55:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:55:26 DEBUG { 05:55:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:55:26 DEBUG }, 05:55:26 DEBUG "value": [ 05:55:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:55:26 DEBUG "pod": "ds-idrepo-0" 05:55:26 DEBUG }, 05:55:26 DEBUG "status": "success", 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG ----- output ----- 05:55:26 DEBUG "value": [ 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG "data": { 05:55:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:55:26 DEBUG ----- output ----- 05:55:26 DEBUG { 05:55:26 DEBUG 05:55:26 DEBUG [...] 05:55:26 DEBUG 05:55:26 DEBUG "resultType": "vector", 05:55:26 DEBUG { 05:55:26 DEBUG "status": "success", 05:55:26 DEBUG "status": "success", 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG "pod": "ds-cts-2" 05:55:26 DEBUG "result": [ 05:55:26 DEBUG "data": { 05:55:26 DEBUG "resultType": "vector", 05:55:26 DEBUG "data": { 05:55:26 DEBUG }, 05:55:26 DEBUG "value": [ 05:55:26 DEBUG 1712724926, 05:55:26 DEBUG { 05:55:26 DEBUG "result": [ 05:55:26 DEBUG { 05:55:26 DEBUG "metric": { 05:55:26 DEBUG "instance": "10.142.0.25:9100" 05:55:26 DEBUG }, 05:55:26 DEBUG "value": [ 05:55:26 DEBUG "metric": { 05:55:26 DEBUG "resultType": "vector", 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG [...] 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG "instance": "10.142.0.25:9100" 05:55:26 DEBUG 05:55:26 DEBUG "instance": "10.142.0.77:9100" 05:55:26 DEBUG }, 05:55:26 DEBUG "value": [ 05:55:26 DEBUG 1712724926, 05:55:26 DEBUG "329909496832" 05:55:26 DEBUG }, 05:55:26 DEBUG "value": [ 05:55:26 DEBUG ] 05:55:26 DEBUG } 05:55:26 DEBUG ] 05:55:26 DEBUG } 05:55:26 DEBUG } 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG "result": [ 05:55:26 DEBUG { 05:55:26 DEBUG }, 05:55:26 DEBUG "value": [ 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG "metric": { 05:55:26 DEBUG "pod": "ds-idrepo-0" 05:55:26 DEBUG 05:55:26 DEBUG [...] 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG }, 05:55:26 DEBUG "value": [ 05:55:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:55:26 DEBUG }, 05:55:26 DEBUG "value": [ 05:55:26 DEBUG 05:55:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:55:26 DEBUG }, 05:55:26 DEBUG 1712724926, 05:55:26 DEBUG "1435.3333333333333" 05:55:26 DEBUG ] 05:55:26 DEBUG } 05:55:26 DEBUG "value": [ 05:55:26 DEBUG 1712724926, 05:55:26 DEBUG ] 05:55:26 DEBUG } 05:55:26 DEBUG } 05:55:26 DEBUG [...] 05:55:26 DEBUG "0" 05:55:26 DEBUG ] 05:55:26 DEBUG [...] 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:55:26 DEBUG }, 05:55:26 DEBUG 05:55:26 DEBUG [...] 05:55:26 DEBUG 05:55:26 DEBUG "value": [ 05:55:26 DEBUG 1712724926, 05:55:26 DEBUG 05:55:26 DEBUG } 05:55:26 DEBUG 05:55:26 DEBUG "258853.5" 05:55:26 DEBUG ] 05:55:26 DEBUG } 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG ] 05:55:26 DEBUG ] 05:55:26 DEBUG } 05:55:26 DEBUG } 05:55:26 DEBUG 05:55:26 DEBUG "29.233333333333334" 05:55:26 DEBUG "pod": "ds-cts-2" 05:55:26 DEBUG "instance": "10.142.0.77:9100" 05:55:26 DEBUG }, 05:55:26 DEBUG ] 05:55:26 DEBUG }, 05:55:26 DEBUG "value": [ 05:55:26 DEBUG } 05:55:26 DEBUG } 05:55:26 DEBUG ] 05:55:26 DEBUG "value": [ 05:55:26 DEBUG } 05:55:26 DEBUG 1712724926, 05:55:26 DEBUG } 05:55:26 DEBUG 1712724926, 05:55:26 DEBUG "0" 05:55:26 DEBUG } 05:55:26 DEBUG "42774831.4" 05:55:26 DEBUG ] 05:55:26 DEBUG ] 05:55:26 DEBUG } 05:55:26 DEBUG ] 05:55:26 DEBUG } 05:55:26 DEBUG } 05:55:26 DEBUG } 05:55:26 DEBUG ] 05:55:26 DEBUG } 05:55:26 DEBUG } 05:55:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724926" 05:55:26 INFO [http_cmd]: http status code OK 05:55:26 DEBUG --- status code --- 05:55:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:26 DEBUG --- http response --- 05:55:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:55:26 DEBUG ----- output ----- 05:55:26 DEBUG { 05:55:26 DEBUG "status": "success", 05:55:26 DEBUG "data": { 05:55:26 DEBUG "resultType": "vector", 05:55:26 DEBUG "result": [ 05:55:26 DEBUG { 05:55:26 DEBUG "metric": { 05:55:26 DEBUG "pod": "ds-idrepo-0" 05:55:26 DEBUG }, 05:55:26 DEBUG "value": [ 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG [...] 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG "pod": "ds-idrepo-1" 05:55:26 DEBUG }, 05:55:26 DEBUG "value": [ 05:55:26 DEBUG 1712724926, 05:55:26 DEBUG "0" 05:55:26 DEBUG ] 05:55:26 DEBUG } 05:55:26 DEBUG ] 05:55:26 DEBUG } 05:55:26 DEBUG } 05:55:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724926" 05:55:26 INFO [http_cmd]: http status code OK 05:55:26 DEBUG --- status code --- 05:55:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:26 DEBUG --- http response --- 05:55:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:55:26 DEBUG ----- output ----- 05:55:26 DEBUG { 05:55:26 DEBUG "status": "success", 05:55:26 DEBUG "data": { 05:55:26 DEBUG "resultType": "vector", 05:55:26 DEBUG "result": [ 05:55:26 DEBUG { 05:55:26 DEBUG "metric": { 05:55:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:55:26 DEBUG }, 05:55:26 DEBUG "value": [ 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG [...] 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG 05:55:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:55:26 DEBUG }, 05:55:26 DEBUG "value": [ 05:55:26 DEBUG 1712724926, 05:55:26 DEBUG "1440.9333333333334" 05:55:26 DEBUG ] 05:55:26 DEBUG } 05:55:26 DEBUG ] 05:55:26 DEBUG } 05:55:26 DEBUG } 05:55:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724927" 05:55:27 INFO [http_cmd]: http status code OK 05:55:27 DEBUG --- status code --- 05:55:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:27 DEBUG --- http response --- 05:55:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:55:27 DEBUG ----- output ----- 05:55:27 DEBUG { 05:55:27 DEBUG "status": "success", 05:55:27 DEBUG "data": { 05:55:27 DEBUG "resultType": "vector", 05:55:27 DEBUG "result": [ 05:55:27 DEBUG { 05:55:27 DEBUG "metric": { 05:55:27 DEBUG "instance": "10.142.0.25:9100" 05:55:27 DEBUG }, 05:55:27 DEBUG "value": [ 05:55:27 DEBUG 05:55:27 DEBUG 05:55:27 DEBUG 05:55:27 DEBUG [...] 05:55:27 DEBUG 05:55:27 DEBUG 05:55:27 DEBUG 05:55:27 DEBUG "instance": "10.142.0.77:9100" 05:55:27 DEBUG }, 05:55:27 DEBUG "value": [ 05:55:27 DEBUG 1712724927, 05:55:27 DEBUG "0.15243333333333467" 05:55:27 DEBUG ] 05:55:27 DEBUG } 05:55:27 DEBUG ] 05:55:27 DEBUG } 05:55:27 DEBUG } 05:55:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724928" 05:55:28 INFO [http_cmd]: http status code OK 05:55:28 DEBUG --- status code --- 05:55:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:28 DEBUG --- http response --- 05:55:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:55:28 DEBUG ----- output ----- 05:55:28 DEBUG { 05:55:28 DEBUG "status": "success", 05:55:28 DEBUG "data": { 05:55:28 DEBUG "resultType": "vector", 05:55:28 DEBUG "result": [ 05:55:28 DEBUG { 05:55:28 DEBUG "metric": { 05:55:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:55:28 DEBUG }, 05:55:28 DEBUG "value": [ 05:55:28 DEBUG 05:55:28 DEBUG 05:55:28 DEBUG 05:55:28 DEBUG [...] 05:55:28 DEBUG 05:55:28 DEBUG 05:55:28 DEBUG 05:55:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:55:28 DEBUG }, 05:55:28 DEBUG "value": [ 05:55:28 DEBUG 1712724928, 05:55:28 DEBUG "82177.4776288724" 05:55:28 DEBUG ] 05:55:28 DEBUG } 05:55:28 DEBUG ] 05:55:28 DEBUG } 05:55:28 DEBUG } 05:55:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712724928" 05:55:28 INFO [http_cmd]: http status code OK 05:55:28 DEBUG --- status code --- 05:55:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:28 DEBUG --- http response --- 05:55:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:55:28 DEBUG ----- output ----- 05:55:28 DEBUG { 05:55:28 DEBUG "status": "success", 05:55:28 DEBUG "data": { 05:55:28 DEBUG "resultType": "vector", 05:55:28 DEBUG "result": [ 05:55:28 DEBUG { 05:55:28 DEBUG "metric": { 05:55:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:55:28 DEBUG }, 05:55:28 DEBUG "value": [ 05:55:28 DEBUG 05:55:28 DEBUG 05:55:28 DEBUG 05:55:28 DEBUG [...] 05:55:28 DEBUG 05:55:28 DEBUG 05:55:28 DEBUG 05:55:28 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:55:28 DEBUG }, 05:55:28 DEBUG "value": [ 05:55:28 DEBUG 1712724928, 05:55:28 DEBUG "4.212750375745532" 05:55:28 DEBUG ] 05:55:28 DEBUG } 05:55:28 DEBUG ] 05:55:28 DEBUG } 05:55:28 DEBUG } 05:55:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712724928" 05:55:28 INFO [http_cmd]: http status code OK 05:55:28 DEBUG --- status code --- 05:55:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:28 DEBUG --- http response --- 05:55:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:55:28 DEBUG ----- output ----- 05:55:28 DEBUG { 05:55:28 DEBUG "status": "success", 05:55:28 DEBUG "data": { 05:55:28 DEBUG "resultType": "vector", 05:55:28 DEBUG "result": [ 05:55:28 DEBUG { 05:55:28 DEBUG "metric": { 05:55:28 DEBUG "pod": "ds-idrepo-0" 05:55:28 DEBUG }, 05:55:28 DEBUG "value": [ 05:55:28 DEBUG 05:55:28 DEBUG 05:55:28 DEBUG 05:55:28 DEBUG [...] 05:55:28 DEBUG 05:55:28 DEBUG 05:55:28 DEBUG 05:55:28 DEBUG "pod": "ds-cts-2" 05:55:28 DEBUG }, 05:55:28 DEBUG "value": [ 05:55:28 DEBUG 1712724928, 05:55:28 DEBUG "0" 05:55:28 DEBUG ] 05:55:28 DEBUG } 05:55:28 DEBUG ] 05:55:28 DEBUG } 05:55:28 DEBUG } 05:55:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724928" 05:55:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724928" 05:55:28 INFO [http_cmd]: http status code OK 05:55:28 DEBUG --- status code --- 05:55:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:28 DEBUG --- http response --- 05:55:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:55:28 DEBUG ----- output ----- 05:55:28 DEBUG { 05:55:28 INFO [http_cmd]: http status code OK 05:55:28 DEBUG "status": "success", 05:55:28 DEBUG --- status code --- 05:55:28 DEBUG "data": { 05:55:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:28 DEBUG "resultType": "vector", 05:55:28 DEBUG --- http response --- 05:55:28 DEBUG "result": [ 05:55:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:55:28 DEBUG { 05:55:28 DEBUG ----- output ----- 05:55:28 DEBUG "metric": { 05:55:28 DEBUG { 05:55:28 DEBUG "pod": "ds-idrepo-2" 05:55:28 DEBUG "status": "success", 05:55:28 DEBUG }, 05:55:28 DEBUG "data": { 05:55:28 DEBUG "value": [ 05:55:28 DEBUG "resultType": "vector", 05:55:28 DEBUG 05:55:28 DEBUG "result": [ 05:55:28 DEBUG 05:55:28 DEBUG { 05:55:28 DEBUG 05:55:28 DEBUG "metric": { 05:55:28 DEBUG [...] 05:55:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:55:28 DEBUG 05:55:28 DEBUG }, 05:55:28 DEBUG "value": [ 05:55:28 DEBUG 05:55:28 DEBUG 05:55:28 DEBUG "pod": "ds-cts-2" 05:55:28 DEBUG }, 05:55:28 DEBUG "value": [ 05:55:28 DEBUG 1712724928, 05:55:28 DEBUG 05:55:28 DEBUG "0.00006666666666666667" 05:55:28 DEBUG 05:55:28 DEBUG ] 05:55:28 DEBUG 05:55:28 DEBUG } 05:55:28 DEBUG [...] 05:55:28 DEBUG ] 05:55:28 DEBUG 05:55:28 DEBUG } 05:55:28 DEBUG 05:55:28 DEBUG } 05:55:28 DEBUG 05:55:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:55:28 DEBUG }, 05:55:28 DEBUG "value": [ 05:55:28 DEBUG 1712724928, 05:55:28 DEBUG "4.945546060373399" 05:55:28 DEBUG ] 05:55:28 DEBUG } 05:55:28 DEBUG ] 05:55:28 DEBUG } 05:55:28 DEBUG } 05:55:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724928" 05:55:28 INFO [http_cmd]: http status code OK 05:55:28 DEBUG --- status code --- 05:55:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:28 DEBUG --- http response --- 05:55:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:55:28 DEBUG ----- output ----- 05:55:28 DEBUG { 05:55:28 DEBUG "status": "success", 05:55:28 DEBUG "data": { 05:55:28 DEBUG "resultType": "vector", 05:55:28 DEBUG "result": [ 05:55:28 DEBUG { 05:55:28 DEBUG "metric": { 05:55:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:55:28 DEBUG }, 05:55:28 DEBUG "value": [ 05:55:28 DEBUG 05:55:28 DEBUG 05:55:28 DEBUG 05:55:28 DEBUG [...] 05:55:28 DEBUG 05:55:28 DEBUG 05:55:28 DEBUG 05:55:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:55:28 DEBUG }, 05:55:28 DEBUG "value": [ 05:55:28 DEBUG 1712724928, 05:55:28 DEBUG "1439.9" 05:55:28 DEBUG ] 05:55:28 DEBUG } 05:55:28 DEBUG ] 05:55:28 DEBUG } 05:55:28 DEBUG } 05:55:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712724928" 05:55:28 INFO [http_cmd]: http status code OK 05:55:28 DEBUG --- status code --- 05:55:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:28 DEBUG --- http response --- 05:55:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:55:28 DEBUG ----- output ----- 05:55:28 DEBUG { 05:55:28 DEBUG "status": "success", 05:55:28 DEBUG "data": { 05:55:28 DEBUG "resultType": "vector", 05:55:28 DEBUG "result": [ 05:55:28 DEBUG { 05:55:28 DEBUG "metric": { 05:55:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:55:28 DEBUG }, 05:55:28 DEBUG "value": [ 05:55:28 DEBUG 05:55:28 DEBUG 05:55:28 DEBUG 05:55:28 DEBUG [...] 05:55:28 DEBUG 05:55:28 DEBUG 05:55:28 DEBUG 05:55:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:55:28 DEBUG }, 05:55:28 DEBUG "value": [ 05:55:28 DEBUG 1712724928, 05:55:28 DEBUG "0.046" 05:55:28 DEBUG ] 05:55:28 DEBUG } 05:55:28 DEBUG ] 05:55:28 DEBUG } 05:55:28 DEBUG } 05:55:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724929" 05:55:29 INFO [http_cmd]: http status code OK 05:55:29 DEBUG --- status code --- 05:55:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:29 DEBUG --- http response --- 05:55:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:55:29 DEBUG ----- output ----- 05:55:29 DEBUG { 05:55:29 DEBUG "status": "success", 05:55:29 DEBUG "data": { 05:55:29 DEBUG "resultType": "vector", 05:55:29 DEBUG "result": [ 05:55:29 DEBUG { 05:55:29 DEBUG "metric": { 05:55:29 DEBUG "pod": "ds-idrepo-0" 05:55:29 DEBUG }, 05:55:29 DEBUG "value": [ 05:55:29 DEBUG 05:55:29 DEBUG 05:55:29 DEBUG 05:55:29 DEBUG [...] 05:55:29 DEBUG 05:55:29 DEBUG 05:55:29 DEBUG 05:55:29 DEBUG "pod": "ds-idrepo-1" 05:55:29 DEBUG }, 05:55:29 DEBUG "value": [ 05:55:29 DEBUG 1712724929, 05:55:29 DEBUG "0" 05:55:29 DEBUG ] 05:55:29 DEBUG } 05:55:29 DEBUG ] 05:55:29 DEBUG } 05:55:29 DEBUG } 05:55:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724929" 05:55:29 INFO [http_cmd]: http status code OK 05:55:29 DEBUG --- status code --- 05:55:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:29 DEBUG --- http response --- 05:55:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:55:29 DEBUG ----- output ----- 05:55:29 DEBUG { 05:55:29 DEBUG "status": "success", 05:55:29 DEBUG "data": { 05:55:29 DEBUG "resultType": "vector", 05:55:29 DEBUG "result": [ 05:55:29 DEBUG { 05:55:29 DEBUG "metric": { 05:55:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:55:29 DEBUG }, 05:55:29 DEBUG "value": [ 05:55:29 DEBUG 05:55:29 DEBUG 05:55:29 DEBUG 05:55:29 DEBUG [...] 05:55:29 DEBUG 05:55:29 DEBUG 05:55:29 DEBUG 05:55:29 DEBUG "pod": "am-654c4fcbcf-487cv" 05:55:29 DEBUG }, 05:55:29 DEBUG "value": [ 05:55:29 DEBUG 1712724929, 05:55:29 DEBUG "1471" 05:55:29 DEBUG ] 05:55:29 DEBUG } 05:55:29 DEBUG ] 05:55:29 DEBUG } 05:55:29 DEBUG } 05:55:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724929" 05:55:29 INFO [http_cmd]: http status code OK 05:55:29 DEBUG --- status code --- 05:55:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:29 DEBUG --- http response --- 05:55:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:55:29 DEBUG ----- output ----- 05:55:29 DEBUG { 05:55:29 DEBUG "status": "success", 05:55:29 DEBUG "data": { 05:55:29 DEBUG "resultType": "vector", 05:55:29 DEBUG "result": [ 05:55:29 DEBUG { 05:55:29 DEBUG "metric": { 05:55:29 DEBUG "pod": "ds-idrepo-0" 05:55:29 DEBUG }, 05:55:29 DEBUG "value": [ 05:55:29 DEBUG 05:55:29 DEBUG 05:55:29 DEBUG 05:55:29 DEBUG [...] 05:55:29 DEBUG 05:55:29 DEBUG 05:55:29 DEBUG 05:55:29 DEBUG "pod": "ds-idrepo-1" 05:55:29 DEBUG }, 05:55:29 DEBUG "value": [ 05:55:29 DEBUG 1712724929, 05:55:29 DEBUG "0" 05:55:29 DEBUG ] 05:55:29 DEBUG } 05:55:29 DEBUG ] 05:55:29 DEBUG } 05:55:29 DEBUG } 05:55:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724929" 05:55:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712724929" 05:55:29 INFO [http_cmd]: http status code OK 05:55:29 DEBUG --- status code --- 05:55:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:29 DEBUG --- http response --- 05:55:29 INFO [http_cmd]: http status code OK 05:55:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:55:29 DEBUG --- status code --- 05:55:29 DEBUG ----- output ----- 05:55:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:29 DEBUG { 05:55:29 DEBUG --- http response --- 05:55:29 DEBUG "status": "success", 05:55:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:55:29 DEBUG ----- output ----- 05:55:29 DEBUG "data": { 05:55:29 DEBUG "resultType": "vector", 05:55:29 DEBUG "result": [ 05:55:29 DEBUG { 05:55:29 DEBUG { 05:55:29 DEBUG "metric": { 05:55:29 DEBUG "status": "success", 05:55:29 DEBUG "data": { 05:55:29 DEBUG "resultType": "vector", 05:55:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:55:29 DEBUG "result": [ 05:55:29 DEBUG { 05:55:29 DEBUG "metric": { 05:55:29 DEBUG }, 05:55:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:55:29 DEBUG "value": [ 05:55:29 DEBUG }, 05:55:29 DEBUG 05:55:29 DEBUG 05:55:29 DEBUG 05:55:29 DEBUG "value": [ 05:55:29 DEBUG [...] 05:55:29 DEBUG 05:55:29 DEBUG 05:55:29 DEBUG 05:55:29 DEBUG 05:55:29 DEBUG 05:55:29 DEBUG [...] 05:55:29 DEBUG 05:55:29 DEBUG 05:55:29 DEBUG 05:55:29 DEBUG 05:55:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:55:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:55:29 DEBUG }, 05:55:29 DEBUG }, 05:55:29 DEBUG "value": [ 05:55:29 DEBUG "value": [ 05:55:29 DEBUG 1712724929, 05:55:29 DEBUG 1712724929, 05:55:29 DEBUG "2488127488" 05:55:29 DEBUG "0" 05:55:29 DEBUG ] 05:55:29 DEBUG ] 05:55:29 DEBUG } 05:55:29 DEBUG } 05:55:29 DEBUG ] 05:55:29 DEBUG } 05:55:29 DEBUG } 05:55:29 DEBUG ] 05:55:29 DEBUG } 05:55:29 DEBUG } 05:55:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724929" 05:55:29 INFO [http_cmd]: http status code OK 05:55:29 DEBUG --- status code --- 05:55:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:29 DEBUG --- http response --- 05:55:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:55:29 DEBUG ----- output ----- 05:55:29 DEBUG { 05:55:29 DEBUG "status": "success", 05:55:29 DEBUG "data": { 05:55:29 DEBUG "resultType": "vector", 05:55:29 DEBUG "result": [ 05:55:29 DEBUG { 05:55:29 DEBUG "metric": { 05:55:29 DEBUG "instance": "10.142.0.25:9100" 05:55:29 DEBUG }, 05:55:29 DEBUG "value": [ 05:55:29 DEBUG 05:55:29 DEBUG 05:55:29 DEBUG 05:55:29 DEBUG [...] 05:55:29 DEBUG 05:55:29 DEBUG 05:55:29 DEBUG 05:55:29 DEBUG "instance": "10.142.0.77:9100" 05:55:29 DEBUG }, 05:55:29 DEBUG "value": [ 05:55:29 DEBUG 1712724929, 05:55:29 DEBUG "42775645.2" 05:55:29 DEBUG ] 05:55:29 DEBUG } 05:55:29 DEBUG ] 05:55:29 DEBUG } 05:55:29 DEBUG } 05:55:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724929" 05:55:29 INFO [http_cmd]: http status code OK 05:55:29 DEBUG --- status code --- 05:55:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:29 DEBUG --- http response --- 05:55:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:55:29 DEBUG ----- output ----- 05:55:29 DEBUG { 05:55:29 DEBUG "status": "success", 05:55:29 DEBUG "data": { 05:55:29 DEBUG "resultType": "vector", 05:55:29 DEBUG "result": [ 05:55:29 DEBUG { 05:55:29 DEBUG "metric": { 05:55:29 DEBUG "instance": "10.142.0.25:9100" 05:55:29 DEBUG }, 05:55:29 DEBUG "value": [ 05:55:29 DEBUG 05:55:29 DEBUG 05:55:29 DEBUG 05:55:29 DEBUG [...] 05:55:29 DEBUG 05:55:29 DEBUG 05:55:29 DEBUG 05:55:29 DEBUG "instance": "10.142.0.77:9100" 05:55:29 DEBUG }, 05:55:29 DEBUG "value": [ 05:55:29 DEBUG 1712724929, 05:55:29 DEBUG "0.05199999999999984" 05:55:29 DEBUG ] 05:55:29 DEBUG } 05:55:29 DEBUG ] 05:55:29 DEBUG } 05:55:29 DEBUG } 05:55:45 INFO 05:55:45 INFO [loop_until]: kubectl --namespace=xlou top pods 05:55:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:55:46 INFO [loop_until]: OK (rc = 0) 05:55:46 DEBUG --- stdout --- 05:55:46 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10373m 8959Mi am-654c4fcbcf-9scrj 9840m 8436Mi am-654c4fcbcf-v5qb5 8532m 8407Mi ds-cts-0 8492m 13353Mi ds-cts-1 8740m 13368Mi ds-cts-2 7671m 12583Mi ds-idrepo-0 10738m 5622Mi ds-idrepo-1 359m 5672Mi ds-idrepo-2 385m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 2m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4100m 2232Mi 05:55:46 DEBUG --- stderr --- 05:55:46 DEBUG 05:55:53 INFO 05:55:53 INFO [loop_until]: kubectl --namespace=xlou top node 05:55:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:55:53 INFO [loop_until]: OK (rc = 0) 05:55:53 DEBUG --- stdout --- 05:55:53 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 94m 0% 2307Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9003m 56% 9284Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 79m 0% 3256Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 12724m 80% 6538Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10883m 68% 9314Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11334m 71% 9786Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 74m 0% 854Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10830m 68% 6290Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8579m 53% 14013Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 7792m 49% 13300Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 431m 2% 6375Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8539m 53% 14022Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 426m 2% 6343Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4256m 26% 5884Mi 10% 05:55:53 DEBUG --- stderr --- 05:55:53 DEBUG 05:56:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712724986" 05:56:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724986" 05:56:26 INFO [http_cmd]: http status code OK 05:56:26 ERROR [http_cmd]: ERROR 05:56:26 DEBUG --- status code --- 05:56:26 ERROR --- status code --- 05:56:26 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:56:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:26 ERROR --- http response --- 05:56:26 DEBUG --- http response --- 05:56:26 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:56:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724986" 05:56:26 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:56:26 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:56:26 INFO [http_cmd]: http status code OK 05:56:26 DEBUG --- status code --- 05:56:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:26 DEBUG --- http response --- 05:56:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:56:26 DEBUG ----- output ----- 05:56:26 DEBUG { 05:56:26 DEBUG "status": "success", 05:56:26 DEBUG "data": { 05:56:26 DEBUG "resultType": "vector", 05:56:26 DEBUG "result": [ 05:56:26 DEBUG { 05:56:26 DEBUG "metric": { 05:56:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:56:26 DEBUG }, 05:56:26 DEBUG "value": [ 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG [...] 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:56:26 DEBUG }, 05:56:26 DEBUG "value": [ 05:56:26 DEBUG 1712724986, 05:56:26 DEBUG "0" 05:56:26 DEBUG ] 05:56:26 DEBUG } 05:56:26 DEBUG ] 05:56:26 DEBUG } 05:56:26 DEBUG } 05:56:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724986" 05:56:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724986" 05:56:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724986" 05:56:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724986" 05:56:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724986" 05:56:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724986" 05:56:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724986" 05:56:26 INFO [http_cmd]: http status code OK 05:56:26 DEBUG --- status code --- 05:56:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:26 DEBUG --- http response --- 05:56:26 INFO [http_cmd]: http status code OK 05:56:26 INFO [http_cmd]: http status code OK 05:56:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:56:26 DEBUG ----- output ----- 05:56:26 DEBUG --- status code --- 05:56:26 DEBUG { 05:56:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:26 DEBUG "status": "success", 05:56:26 DEBUG --- http response --- 05:56:26 DEBUG "data": { 05:56:26 DEBUG "resultType": "vector", 05:56:26 DEBUG "result": [ 05:56:26 DEBUG { 05:56:26 DEBUG "metric": { 05:56:26 DEBUG "instance": "10.142.0.25:9100" 05:56:26 DEBUG }, 05:56:26 DEBUG "value": [ 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG [...] 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG --- status code --- 05:56:26 DEBUG "instance": "10.142.0.77:9100" 05:56:26 INFO [http_cmd]: http status code OK 05:56:26 DEBUG --- status code --- 05:56:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:26 DEBUG }, 05:56:26 DEBUG "value": [ 05:56:26 DEBUG 1712724986, 05:56:26 DEBUG "332055815168" 05:56:26 DEBUG ] 05:56:26 DEBUG } 05:56:26 DEBUG --- http response --- 05:56:26 DEBUG ] 05:56:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:56:26 DEBUG ----- output ----- 05:56:26 DEBUG } 05:56:26 DEBUG --- http response --- 05:56:26 DEBUG { 05:56:26 DEBUG "status": "success", 05:56:26 DEBUG "data": { 05:56:26 DEBUG "resultType": "vector", 05:56:26 DEBUG "result": [ 05:56:26 DEBUG { 05:56:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:56:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:56:26 DEBUG "metric": { 05:56:26 DEBUG ----- output ----- 05:56:26 DEBUG { 05:56:26 DEBUG "status": "success", 05:56:26 INFO [http_cmd]: http status code OK 05:56:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:56:26 DEBUG "data": { 05:56:26 DEBUG "resultType": "vector", 05:56:26 DEBUG "result": [ 05:56:26 DEBUG --- status code --- 05:56:26 DEBUG }, 05:56:26 DEBUG ----- output ----- 05:56:26 DEBUG { 05:56:26 INFO [http_cmd]: http status code OK 05:56:26 DEBUG --- status code --- 05:56:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:26 DEBUG --- http response --- 05:56:26 DEBUG "status": "success", 05:56:26 DEBUG "data": { 05:56:26 INFO [http_cmd]: http status code OK 05:56:26 DEBUG --- status code --- 05:56:26 DEBUG { 05:56:26 DEBUG "value": [ 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG "resultType": "vector", 05:56:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:26 DEBUG --- http response --- 05:56:26 DEBUG } 05:56:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:26 DEBUG --- http response --- 05:56:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:56:26 DEBUG "result": [ 05:56:26 DEBUG "metric": { 05:56:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:56:26 DEBUG }, 05:56:26 DEBUG "value": [ 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG [...] 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG { 05:56:26 DEBUG "metric": { 05:56:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:56:26 DEBUG ----- output ----- 05:56:26 DEBUG { 05:56:26 DEBUG ----- output ----- 05:56:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:56:26 DEBUG }, 05:56:26 DEBUG "status": "success", 05:56:26 DEBUG { 05:56:26 DEBUG "status": "success", 05:56:26 DEBUG "data": { 05:56:26 DEBUG "resultType": "vector", 05:56:26 DEBUG 05:56:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:56:26 DEBUG }, 05:56:26 DEBUG "value": [ 05:56:26 DEBUG 1712724986, 05:56:26 DEBUG "1448.3666666666666" 05:56:26 DEBUG ] 05:56:26 DEBUG } 05:56:26 DEBUG ] 05:56:26 DEBUG } 05:56:26 DEBUG } 05:56:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:56:26 DEBUG "value": [ 05:56:26 DEBUG 05:56:26 DEBUG [...] 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG ----- output ----- 05:56:26 DEBUG 05:56:26 DEBUG "data": { 05:56:26 DEBUG "resultType": "vector", 05:56:26 DEBUG "result": [ 05:56:26 DEBUG { 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG "metric": { 05:56:26 DEBUG "pod": "ds-idrepo-0" 05:56:26 DEBUG }, 05:56:26 DEBUG "value": [ 05:56:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:56:26 DEBUG }, 05:56:26 DEBUG 05:56:26 DEBUG { 05:56:26 DEBUG "result": [ 05:56:26 DEBUG { 05:56:26 DEBUG "metric": { 05:56:26 DEBUG "instance": "10.142.0.25:9100" 05:56:26 DEBUG }, 05:56:26 DEBUG "value": [ 05:56:26 DEBUG "status": "success", 05:56:26 DEBUG "data": { 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG [...] 05:56:26 DEBUG "resultType": "vector", 05:56:26 DEBUG "value": [ 05:56:26 DEBUG 1712724986, 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG "result": [ 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG [...] 05:56:26 DEBUG 05:56:26 DEBUG { 05:56:26 DEBUG [...] 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG "metric": { 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:56:26 DEBUG "pod": "ds-idrepo-0" 05:56:26 DEBUG "instance": "10.142.0.77:9100" 05:56:26 DEBUG }, 05:56:26 DEBUG 05:56:26 DEBUG }, 05:56:26 DEBUG }, 05:56:26 DEBUG "value": [ 05:56:26 DEBUG 1712724986, 05:56:26 DEBUG "pod": "ds-cts-2" 05:56:26 DEBUG "value": [ 05:56:26 DEBUG "value": [ 05:56:26 DEBUG "42572024.93333334" 05:56:26 DEBUG ] 05:56:26 DEBUG }, 05:56:26 DEBUG 1712724986, 05:56:26 DEBUG "0" 05:56:26 DEBUG ] 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG "258306.5" 05:56:26 DEBUG ] 05:56:26 DEBUG } 05:56:26 DEBUG ] 05:56:26 DEBUG } 05:56:26 DEBUG [...] 05:56:26 DEBUG 05:56:26 DEBUG } 05:56:26 DEBUG } 05:56:26 DEBUG ] 05:56:26 DEBUG } 05:56:26 DEBUG } 05:56:26 DEBUG "value": [ 05:56:26 DEBUG 1712724986, 05:56:26 DEBUG "0" 05:56:26 DEBUG 05:56:26 DEBUG } 05:56:26 DEBUG ] 05:56:26 DEBUG 05:56:26 DEBUG ] 05:56:26 DEBUG } 05:56:26 DEBUG "pod": "ds-cts-2" 05:56:26 DEBUG }, 05:56:26 DEBUG "value": [ 05:56:26 DEBUG ] 05:56:26 DEBUG 1712724986, 05:56:26 DEBUG "30" 05:56:26 DEBUG } 05:56:26 DEBUG ] 05:56:26 DEBUG } 05:56:26 DEBUG } 05:56:26 DEBUG ] 05:56:26 DEBUG } 05:56:26 DEBUG } 05:56:26 DEBUG } 05:56:26 DEBUG } 05:56:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724986" 05:56:26 INFO [http_cmd]: http status code OK 05:56:26 DEBUG --- status code --- 05:56:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:26 DEBUG --- http response --- 05:56:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:56:26 DEBUG ----- output ----- 05:56:26 DEBUG { 05:56:26 DEBUG "status": "success", 05:56:26 DEBUG "data": { 05:56:26 DEBUG "resultType": "vector", 05:56:26 DEBUG "result": [ 05:56:26 DEBUG { 05:56:26 DEBUG "metric": { 05:56:26 DEBUG "pod": "ds-idrepo-0" 05:56:26 DEBUG }, 05:56:26 DEBUG "value": [ 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG [...] 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG "pod": "ds-idrepo-1" 05:56:26 DEBUG }, 05:56:26 DEBUG "value": [ 05:56:26 DEBUG 1712724986, 05:56:26 DEBUG "0" 05:56:26 DEBUG ] 05:56:26 DEBUG } 05:56:26 DEBUG ] 05:56:26 DEBUG } 05:56:26 DEBUG } 05:56:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724986" 05:56:26 INFO [http_cmd]: http status code OK 05:56:26 DEBUG --- status code --- 05:56:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:26 DEBUG --- http response --- 05:56:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:56:26 DEBUG ----- output ----- 05:56:26 DEBUG { 05:56:26 DEBUG "status": "success", 05:56:26 DEBUG "data": { 05:56:26 DEBUG "resultType": "vector", 05:56:26 DEBUG "result": [ 05:56:26 DEBUG { 05:56:26 DEBUG "metric": { 05:56:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:56:26 DEBUG }, 05:56:26 DEBUG "value": [ 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG [...] 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG 05:56:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:56:26 DEBUG }, 05:56:26 DEBUG "value": [ 05:56:26 DEBUG 1712724986, 05:56:26 DEBUG "1459.0333333333333" 05:56:26 DEBUG ] 05:56:26 DEBUG } 05:56:26 DEBUG ] 05:56:26 DEBUG } 05:56:26 DEBUG } 05:56:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724987" 05:56:27 INFO [http_cmd]: http status code OK 05:56:27 DEBUG --- status code --- 05:56:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:27 DEBUG --- http response --- 05:56:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:56:27 DEBUG ----- output ----- 05:56:27 DEBUG { 05:56:27 DEBUG "status": "success", 05:56:27 DEBUG "data": { 05:56:27 DEBUG "resultType": "vector", 05:56:27 DEBUG "result": [ 05:56:27 DEBUG { 05:56:27 DEBUG "metric": { 05:56:27 DEBUG "instance": "10.142.0.25:9100" 05:56:27 DEBUG }, 05:56:27 DEBUG "value": [ 05:56:27 DEBUG 05:56:27 DEBUG 05:56:27 DEBUG 05:56:27 DEBUG [...] 05:56:27 DEBUG 05:56:27 DEBUG 05:56:27 DEBUG 05:56:27 DEBUG "instance": "10.142.0.77:9100" 05:56:27 DEBUG }, 05:56:27 DEBUG "value": [ 05:56:27 DEBUG 1712724987, 05:56:27 DEBUG "0.04300000000000021" 05:56:27 DEBUG ] 05:56:27 DEBUG } 05:56:27 DEBUG ] 05:56:27 DEBUG } 05:56:27 DEBUG } 05:56:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724988" 05:56:28 INFO [http_cmd]: http status code OK 05:56:28 DEBUG --- status code --- 05:56:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:28 DEBUG --- http response --- 05:56:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:56:28 DEBUG ----- output ----- 05:56:28 DEBUG { 05:56:28 DEBUG "status": "success", 05:56:28 DEBUG "data": { 05:56:28 DEBUG "resultType": "vector", 05:56:28 DEBUG "result": [ 05:56:28 DEBUG { 05:56:28 DEBUG "metric": { 05:56:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:56:28 DEBUG }, 05:56:28 DEBUG "value": [ 05:56:28 DEBUG 05:56:28 DEBUG 05:56:28 DEBUG 05:56:28 DEBUG [...] 05:56:28 DEBUG 05:56:28 DEBUG 05:56:28 DEBUG 05:56:28 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:56:28 DEBUG }, 05:56:28 DEBUG "value": [ 05:56:28 DEBUG 1712724988, 05:56:28 DEBUG "61619863.85465868" 05:56:28 DEBUG ] 05:56:28 DEBUG } 05:56:28 DEBUG ] 05:56:28 DEBUG } 05:56:28 DEBUG } 05:56:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712724988" 05:56:28 INFO [http_cmd]: http status code OK 05:56:28 DEBUG --- status code --- 05:56:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:28 DEBUG --- http response --- 05:56:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:56:28 DEBUG ----- output ----- 05:56:28 DEBUG { 05:56:28 DEBUG "status": "success", 05:56:28 DEBUG "data": { 05:56:28 DEBUG "resultType": "vector", 05:56:28 DEBUG "result": [ 05:56:28 DEBUG { 05:56:28 DEBUG "metric": { 05:56:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:56:28 DEBUG }, 05:56:28 DEBUG "value": [ 05:56:28 DEBUG 05:56:28 DEBUG 05:56:28 DEBUG 05:56:28 DEBUG [...] 05:56:28 DEBUG 05:56:28 DEBUG 05:56:28 DEBUG 05:56:28 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:56:28 DEBUG }, 05:56:28 DEBUG "value": [ 05:56:28 DEBUG 1712724988, 05:56:28 DEBUG "4.1906025234595035" 05:56:28 DEBUG ] 05:56:28 DEBUG } 05:56:28 DEBUG ] 05:56:28 DEBUG } 05:56:28 DEBUG } 05:56:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712724988" 05:56:28 INFO [http_cmd]: http status code OK 05:56:28 DEBUG --- status code --- 05:56:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:28 DEBUG --- http response --- 05:56:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:56:28 DEBUG ----- output ----- 05:56:28 DEBUG { 05:56:28 DEBUG "status": "success", 05:56:28 DEBUG "data": { 05:56:28 DEBUG "resultType": "vector", 05:56:28 DEBUG "result": [ 05:56:28 DEBUG { 05:56:28 DEBUG "metric": { 05:56:28 DEBUG "pod": "ds-idrepo-0" 05:56:28 DEBUG }, 05:56:28 DEBUG "value": [ 05:56:28 DEBUG 05:56:28 DEBUG 05:56:28 DEBUG 05:56:28 DEBUG [...] 05:56:28 DEBUG 05:56:28 DEBUG 05:56:28 DEBUG 05:56:28 DEBUG "pod": "ds-cts-2" 05:56:28 DEBUG }, 05:56:28 DEBUG "value": [ 05:56:28 DEBUG 1712724988, 05:56:28 DEBUG "0" 05:56:28 DEBUG ] 05:56:28 DEBUG } 05:56:28 DEBUG ] 05:56:28 DEBUG } 05:56:28 DEBUG } 05:56:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724988" 05:56:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724988" 05:56:28 INFO [http_cmd]: http status code OK 05:56:28 DEBUG --- status code --- 05:56:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:28 DEBUG --- http response --- 05:56:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:56:28 DEBUG ----- output ----- 05:56:28 DEBUG { 05:56:28 DEBUG "status": "success", 05:56:28 DEBUG "data": { 05:56:28 DEBUG "resultType": "vector", 05:56:28 DEBUG "result": [ 05:56:28 DEBUG { 05:56:28 DEBUG "metric": { 05:56:28 DEBUG "pod": "ds-idrepo-2" 05:56:28 DEBUG }, 05:56:28 DEBUG "value": [ 05:56:28 DEBUG 05:56:28 DEBUG 05:56:28 DEBUG 05:56:28 DEBUG [...] 05:56:28 DEBUG 05:56:28 DEBUG 05:56:28 DEBUG 05:56:28 DEBUG "pod": "ds-cts-2" 05:56:28 DEBUG }, 05:56:28 DEBUG "value": [ 05:56:28 DEBUG 1712724988, 05:56:28 DEBUG "0.000033333333333333335" 05:56:28 DEBUG ] 05:56:28 DEBUG } 05:56:28 DEBUG ] 05:56:28 DEBUG } 05:56:28 DEBUG } 05:56:28 INFO [http_cmd]: http status code OK 05:56:28 DEBUG --- status code --- 05:56:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:28 DEBUG --- http response --- 05:56:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:56:28 DEBUG ----- output ----- 05:56:28 DEBUG { 05:56:28 DEBUG "status": "success", 05:56:28 DEBUG "data": { 05:56:28 DEBUG "resultType": "vector", 05:56:28 DEBUG "result": [ 05:56:28 DEBUG { 05:56:28 DEBUG "metric": { 05:56:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:56:28 DEBUG }, 05:56:28 DEBUG "value": [ 05:56:28 DEBUG 05:56:28 DEBUG 05:56:28 DEBUG 05:56:28 DEBUG [...] 05:56:28 DEBUG 05:56:28 DEBUG 05:56:28 DEBUG 05:56:28 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:56:28 DEBUG }, 05:56:28 DEBUG "value": [ 05:56:28 DEBUG 1712724988, 05:56:28 DEBUG "2.0573106850349094" 05:56:28 DEBUG ] 05:56:28 DEBUG } 05:56:28 DEBUG ] 05:56:28 DEBUG } 05:56:28 DEBUG } 05:56:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724988" 05:56:28 INFO [http_cmd]: http status code OK 05:56:28 DEBUG --- status code --- 05:56:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:28 DEBUG --- http response --- 05:56:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:56:28 DEBUG ----- output ----- 05:56:28 DEBUG { 05:56:28 DEBUG "status": "success", 05:56:28 DEBUG "data": { 05:56:28 DEBUG "resultType": "vector", 05:56:28 DEBUG "result": [ 05:56:28 DEBUG { 05:56:28 DEBUG "metric": { 05:56:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:56:28 DEBUG }, 05:56:28 DEBUG "value": [ 05:56:28 DEBUG 05:56:28 DEBUG 05:56:28 DEBUG 05:56:28 DEBUG [...] 05:56:28 DEBUG 05:56:28 DEBUG 05:56:28 DEBUG 05:56:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:56:28 DEBUG }, 05:56:28 DEBUG "value": [ 05:56:28 DEBUG 1712724988, 05:56:28 DEBUG "1459.3333333333333" 05:56:28 DEBUG ] 05:56:28 DEBUG } 05:56:28 DEBUG ] 05:56:28 DEBUG } 05:56:28 DEBUG } 05:56:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712724988" 05:56:28 INFO [http_cmd]: http status code OK 05:56:28 DEBUG --- status code --- 05:56:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:28 DEBUG --- http response --- 05:56:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:56:28 DEBUG ----- output ----- 05:56:28 DEBUG { 05:56:28 DEBUG "status": "success", 05:56:28 DEBUG "data": { 05:56:28 DEBUG "resultType": "vector", 05:56:28 DEBUG "result": [ 05:56:28 DEBUG { 05:56:28 DEBUG "metric": { 05:56:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:56:28 DEBUG }, 05:56:28 DEBUG "value": [ 05:56:28 DEBUG 05:56:28 DEBUG 05:56:28 DEBUG 05:56:28 DEBUG [...] 05:56:28 DEBUG 05:56:28 DEBUG 05:56:28 DEBUG 05:56:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:56:28 DEBUG }, 05:56:28 DEBUG "value": [ 05:56:28 DEBUG 1712724988, 05:56:28 DEBUG "0.046" 05:56:28 DEBUG ] 05:56:28 DEBUG } 05:56:28 DEBUG ] 05:56:28 DEBUG } 05:56:28 DEBUG } 05:56:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724989" 05:56:29 INFO [http_cmd]: http status code OK 05:56:29 DEBUG --- status code --- 05:56:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:29 DEBUG --- http response --- 05:56:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:56:29 DEBUG ----- output ----- 05:56:29 DEBUG { 05:56:29 DEBUG "status": "success", 05:56:29 DEBUG "data": { 05:56:29 DEBUG "resultType": "vector", 05:56:29 DEBUG "result": [ 05:56:29 DEBUG { 05:56:29 DEBUG "metric": { 05:56:29 DEBUG "pod": "ds-idrepo-0" 05:56:29 DEBUG }, 05:56:29 DEBUG "value": [ 05:56:29 DEBUG 05:56:29 DEBUG 05:56:29 DEBUG 05:56:29 DEBUG [...] 05:56:29 DEBUG 05:56:29 DEBUG 05:56:29 DEBUG 05:56:29 DEBUG "pod": "ds-idrepo-1" 05:56:29 DEBUG }, 05:56:29 DEBUG "value": [ 05:56:29 DEBUG 1712724989, 05:56:29 DEBUG "0" 05:56:29 DEBUG ] 05:56:29 DEBUG } 05:56:29 DEBUG ] 05:56:29 DEBUG } 05:56:29 DEBUG } 05:56:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724989" 05:56:29 INFO [http_cmd]: http status code OK 05:56:29 DEBUG --- status code --- 05:56:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:29 DEBUG --- http response --- 05:56:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:56:29 DEBUG ----- output ----- 05:56:29 DEBUG { 05:56:29 DEBUG "status": "success", 05:56:29 DEBUG "data": { 05:56:29 DEBUG "resultType": "vector", 05:56:29 DEBUG "result": [ 05:56:29 DEBUG { 05:56:29 DEBUG "metric": { 05:56:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:56:29 DEBUG }, 05:56:29 DEBUG "value": [ 05:56:29 DEBUG 05:56:29 DEBUG 05:56:29 DEBUG 05:56:29 DEBUG [...] 05:56:29 DEBUG 05:56:29 DEBUG 05:56:29 DEBUG 05:56:29 DEBUG "pod": "am-654c4fcbcf-487cv" 05:56:29 DEBUG }, 05:56:29 DEBUG "value": [ 05:56:29 DEBUG 1712724989, 05:56:29 DEBUG "1422.7333333333333" 05:56:29 DEBUG ] 05:56:29 DEBUG } 05:56:29 DEBUG ] 05:56:29 DEBUG } 05:56:29 DEBUG } 05:56:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724989" 05:56:29 INFO [http_cmd]: http status code OK 05:56:29 DEBUG --- status code --- 05:56:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:29 DEBUG --- http response --- 05:56:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:56:29 DEBUG ----- output ----- 05:56:29 DEBUG { 05:56:29 DEBUG "status": "success", 05:56:29 DEBUG "data": { 05:56:29 DEBUG "resultType": "vector", 05:56:29 DEBUG "result": [ 05:56:29 DEBUG { 05:56:29 DEBUG "metric": { 05:56:29 DEBUG "pod": "ds-idrepo-0" 05:56:29 DEBUG }, 05:56:29 DEBUG "value": [ 05:56:29 DEBUG 05:56:29 DEBUG 05:56:29 DEBUG 05:56:29 DEBUG [...] 05:56:29 DEBUG 05:56:29 DEBUG 05:56:29 DEBUG 05:56:29 DEBUG "pod": "ds-idrepo-1" 05:56:29 DEBUG }, 05:56:29 DEBUG "value": [ 05:56:29 DEBUG 1712724989, 05:56:29 DEBUG "0" 05:56:29 DEBUG ] 05:56:29 DEBUG } 05:56:29 DEBUG ] 05:56:29 DEBUG } 05:56:29 DEBUG } 05:56:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712724989" 05:56:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712724989" 05:56:29 INFO [http_cmd]: http status code OK 05:56:29 DEBUG --- status code --- 05:56:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:29 DEBUG --- http response --- 05:56:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:56:29 DEBUG ----- output ----- 05:56:29 DEBUG { 05:56:29 DEBUG "status": "success", 05:56:29 DEBUG "data": { 05:56:29 DEBUG "resultType": "vector", 05:56:29 DEBUG "result": [ 05:56:29 DEBUG { 05:56:29 DEBUG "metric": { 05:56:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:56:29 DEBUG }, 05:56:29 DEBUG "value": [ 05:56:29 DEBUG 05:56:29 DEBUG 05:56:29 DEBUG 05:56:29 DEBUG [...] 05:56:29 DEBUG 05:56:29 DEBUG 05:56:29 DEBUG 05:56:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:56:29 DEBUG }, 05:56:29 DEBUG "value": [ 05:56:29 DEBUG 1712724989, 05:56:29 DEBUG "2487336960" 05:56:29 DEBUG ] 05:56:29 DEBUG } 05:56:29 DEBUG ] 05:56:29 DEBUG } 05:56:29 DEBUG } 05:56:29 INFO [http_cmd]: http status code OK 05:56:29 DEBUG --- status code --- 05:56:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:29 DEBUG --- http response --- 05:56:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:56:29 DEBUG ----- output ----- 05:56:29 DEBUG { 05:56:29 DEBUG "status": "success", 05:56:29 DEBUG "data": { 05:56:29 DEBUG "resultType": "vector", 05:56:29 DEBUG "result": [ 05:56:29 DEBUG { 05:56:29 DEBUG "metric": { 05:56:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:56:29 DEBUG }, 05:56:29 DEBUG "value": [ 05:56:29 DEBUG 05:56:29 DEBUG 05:56:29 DEBUG 05:56:29 DEBUG [...] 05:56:29 DEBUG 05:56:29 DEBUG 05:56:29 DEBUG 05:56:29 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:56:29 DEBUG }, 05:56:29 DEBUG "value": [ 05:56:29 DEBUG 1712724989, 05:56:29 DEBUG "0" 05:56:29 DEBUG ] 05:56:29 DEBUG } 05:56:29 DEBUG ] 05:56:29 DEBUG } 05:56:29 DEBUG } 05:56:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724989" 05:56:29 INFO [http_cmd]: http status code OK 05:56:29 DEBUG --- status code --- 05:56:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:29 DEBUG --- http response --- 05:56:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:56:29 DEBUG ----- output ----- 05:56:29 DEBUG { 05:56:29 DEBUG "status": "success", 05:56:29 DEBUG "data": { 05:56:29 DEBUG "resultType": "vector", 05:56:29 DEBUG "result": [ 05:56:29 DEBUG { 05:56:29 DEBUG "metric": { 05:56:29 DEBUG "instance": "10.142.0.25:9100" 05:56:29 DEBUG }, 05:56:29 DEBUG "value": [ 05:56:29 DEBUG 05:56:29 DEBUG 05:56:29 DEBUG 05:56:29 DEBUG [...] 05:56:29 DEBUG 05:56:29 DEBUG 05:56:29 DEBUG 05:56:29 DEBUG "instance": "10.142.0.77:9100" 05:56:29 DEBUG }, 05:56:29 DEBUG "value": [ 05:56:29 DEBUG 1712724989, 05:56:29 DEBUG "42573400.9" 05:56:29 DEBUG ] 05:56:29 DEBUG } 05:56:29 DEBUG ] 05:56:29 DEBUG } 05:56:29 DEBUG } 05:56:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712724989" 05:56:29 INFO [http_cmd]: http status code OK 05:56:29 DEBUG --- status code --- 05:56:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:29 DEBUG --- http response --- 05:56:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:56:29 DEBUG ----- output ----- 05:56:29 DEBUG { 05:56:29 DEBUG "status": "success", 05:56:29 DEBUG "data": { 05:56:29 DEBUG "resultType": "vector", 05:56:29 DEBUG "result": [ 05:56:29 DEBUG { 05:56:29 DEBUG "metric": { 05:56:29 DEBUG "instance": "10.142.0.25:9100" 05:56:29 DEBUG }, 05:56:29 DEBUG "value": [ 05:56:29 DEBUG 05:56:29 DEBUG 05:56:29 DEBUG 05:56:29 DEBUG [...] 05:56:29 DEBUG 05:56:29 DEBUG 05:56:29 DEBUG 05:56:29 DEBUG "instance": "10.142.0.77:9100" 05:56:29 DEBUG }, 05:56:29 DEBUG "value": [ 05:56:29 DEBUG 1712724989, 05:56:29 DEBUG "0.01800000000000021" 05:56:29 DEBUG ] 05:56:29 DEBUG } 05:56:29 DEBUG ] 05:56:29 DEBUG } 05:56:29 DEBUG } 05:56:46 INFO 05:56:46 INFO [loop_until]: kubectl --namespace=xlou top pods 05:56:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:56:46 INFO [loop_until]: OK (rc = 0) 05:56:46 DEBUG --- stdout --- 05:56:46 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10504m 8967Mi am-654c4fcbcf-9scrj 10015m 8442Mi am-654c4fcbcf-v5qb5 8455m 8425Mi ds-cts-0 8865m 13368Mi ds-cts-1 8240m 12994Mi ds-cts-2 7631m 13379Mi ds-idrepo-0 10893m 5622Mi ds-idrepo-1 419m 5671Mi ds-idrepo-2 398m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 6m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4166m 2233Mi 05:56:46 DEBUG --- stderr --- 05:56:46 DEBUG 05:56:53 INFO 05:56:53 INFO [loop_until]: kubectl --namespace=xlou top node 05:56:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:56:53 INFO [loop_until]: OK (rc = 0) 05:56:53 DEBUG --- stdout --- 05:56:53 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 94m 0% 2311Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9417m 59% 9302Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 81m 0% 3259Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13272m 83% 6546Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10594m 66% 9321Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11473m 72% 9787Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 74m 0% 853Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10880m 68% 6292Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8629m 54% 14027Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 7760m 48% 14020Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 431m 2% 6380Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8587m 54% 13726Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 469m 2% 6341Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4291m 27% 5888Mi 10% 05:56:53 DEBUG --- stderr --- 05:56:53 DEBUG 05:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712725046" 05:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725046" 05:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725046" 05:57:26 INFO [http_cmd]: http status code OK 05:57:26 ERROR [http_cmd]: ERROR 05:57:26 DEBUG --- status code --- 05:57:26 ERROR --- status code --- 05:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:26 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:57:26 DEBUG --- http response --- 05:57:26 ERROR --- http response --- 05:57:26 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:57:26 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:57:26 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:57:26 INFO [http_cmd]: http status code OK 05:57:26 DEBUG --- status code --- 05:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:26 DEBUG --- http response --- 05:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:57:26 DEBUG ----- output ----- 05:57:26 DEBUG { 05:57:26 DEBUG "status": "success", 05:57:26 DEBUG "data": { 05:57:26 DEBUG "resultType": "vector", 05:57:26 DEBUG "result": [ 05:57:26 DEBUG { 05:57:26 DEBUG "metric": { 05:57:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:57:26 DEBUG }, 05:57:26 DEBUG "value": [ 05:57:26 DEBUG 05:57:26 DEBUG 05:57:26 DEBUG 05:57:26 DEBUG [...] 05:57:26 DEBUG 05:57:26 DEBUG 05:57:26 DEBUG 05:57:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:57:26 DEBUG }, 05:57:26 DEBUG "value": [ 05:57:26 DEBUG 1712725046, 05:57:26 DEBUG "0" 05:57:26 DEBUG ] 05:57:26 DEBUG } 05:57:26 DEBUG ] 05:57:26 DEBUG } 05:57:26 DEBUG } 05:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725046" 05:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725046" 05:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725046" 05:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725046" 05:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725046" 05:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725046" 05:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725046" 05:57:26 INFO [http_cmd]: http status code OK 05:57:26 DEBUG --- status code --- 05:57:26 INFO [http_cmd]: http status code OK 05:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:26 DEBUG --- status code --- 05:57:26 DEBUG --- http response --- 05:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:57:26 DEBUG --- http response --- 05:57:26 DEBUG ----- output ----- 05:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:57:26 DEBUG { 05:57:26 DEBUG ----- output ----- 05:57:26 DEBUG "status": "success", 05:57:26 DEBUG { 05:57:26 DEBUG "data": { 05:57:26 DEBUG "resultType": "vector", 05:57:26 DEBUG "result": [ 05:57:26 DEBUG { 05:57:26 DEBUG "status": "success", 05:57:26 DEBUG "metric": { 05:57:26 DEBUG "data": { 05:57:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:57:26 DEBUG "resultType": "vector", 05:57:26 DEBUG }, 05:57:26 DEBUG "result": [ 05:57:26 DEBUG { 05:57:26 DEBUG "metric": { 05:57:26 DEBUG "instance": "10.142.0.25:9100" 05:57:26 INFO [http_cmd]: http status code OK 05:57:26 DEBUG "value": [ 05:57:26 DEBUG }, 05:57:26 INFO [http_cmd]: http status code OK 05:57:26 INFO [http_cmd]: http status code OK 05:57:26 DEBUG --- status code --- 05:57:26 INFO [http_cmd]: http status code OK 05:57:26 DEBUG 05:57:26 DEBUG "value": [ 05:57:26 DEBUG --- status code --- 05:57:26 DEBUG --- status code --- 05:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:26 DEBUG --- status code --- 05:57:26 DEBUG 05:57:26 DEBUG 05:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:26 DEBUG --- http response --- 05:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:26 DEBUG 05:57:26 DEBUG 05:57:26 DEBUG --- http response --- 05:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 05:57:26 DEBUG --- http response --- 05:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:57:26 DEBUG ----- output ----- 05:57:26 DEBUG ----- output ----- 05:57:26 DEBUG ----- output ----- 05:57:26 DEBUG { 05:57:26 DEBUG [...] 05:57:26 DEBUG 05:57:26 DEBUG { 05:57:26 DEBUG { 05:57:26 INFO [http_cmd]: http status code OK 05:57:26 DEBUG --- status code --- 05:57:26 DEBUG 05:57:26 DEBUG 05:57:26 DEBUG "status": "success", 05:57:26 DEBUG [...] 05:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:26 DEBUG 05:57:26 DEBUG "data": { 05:57:26 DEBUG --- http response --- 05:57:26 DEBUG "status": "success", 05:57:26 DEBUG "status": "success", 05:57:26 DEBUG "data": { 05:57:26 DEBUG --- http response --- 05:57:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:57:26 DEBUG "resultType": "vector", 05:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:57:26 DEBUG "data": { 05:57:26 DEBUG "resultType": "vector", 05:57:26 DEBUG 05:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:57:26 DEBUG }, 05:57:26 DEBUG "result": [ 05:57:26 DEBUG ----- output ----- 05:57:26 DEBUG "result": [ 05:57:26 DEBUG { 05:57:26 DEBUG "metric": { 05:57:26 DEBUG ----- output ----- 05:57:26 DEBUG "value": [ 05:57:26 DEBUG { 05:57:26 DEBUG { 05:57:26 DEBUG "pod": "ds-idrepo-0" 05:57:26 DEBUG "resultType": "vector", 05:57:26 DEBUG 05:57:26 DEBUG { 05:57:26 DEBUG 1712725046, 05:57:26 DEBUG "metric": { 05:57:26 DEBUG "status": "success", 05:57:26 DEBUG }, 05:57:26 DEBUG "result": [ 05:57:26 DEBUG { 05:57:26 DEBUG "metric": { 05:57:26 DEBUG "1438.1333333333334" 05:57:26 DEBUG "pod": "ds-idrepo-0" 05:57:26 DEBUG "data": { 05:57:26 DEBUG "value": [ 05:57:26 DEBUG 05:57:26 DEBUG 05:57:26 DEBUG 05:57:26 DEBUG ] 05:57:26 DEBUG }, 05:57:26 DEBUG "resultType": "vector", 05:57:26 DEBUG "result": [ 05:57:26 DEBUG { 05:57:26 DEBUG "metric": { 05:57:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:57:26 DEBUG }, 05:57:26 DEBUG "value": [ 05:57:26 DEBUG 05:57:26 DEBUG } 05:57:26 DEBUG 05:57:26 DEBUG "value": [ 05:57:26 DEBUG 05:57:26 DEBUG 05:57:26 DEBUG 05:57:26 DEBUG 05:57:26 DEBUG [...] 05:57:26 DEBUG [...] 05:57:26 DEBUG 05:57:26 DEBUG 05:57:26 DEBUG "status": "success", 05:57:26 DEBUG "data": { 05:57:26 DEBUG [...] 05:57:26 DEBUG 05:57:26 DEBUG 05:57:26 DEBUG ] 05:57:26 DEBUG 05:57:26 DEBUG "resultType": "vector", 05:57:26 DEBUG "result": [ 05:57:26 DEBUG 05:57:26 DEBUG "instance": "10.142.0.77:9100" 05:57:26 DEBUG 05:57:26 DEBUG } 05:57:26 DEBUG 05:57:26 DEBUG { 05:57:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:57:26 DEBUG 05:57:26 DEBUG }, 05:57:26 DEBUG "value": [ 05:57:26 DEBUG } 05:57:26 DEBUG "pod": "ds-cts-2" 05:57:26 DEBUG "metric": { 05:57:26 DEBUG }, 05:57:26 DEBUG "value": [ 05:57:26 DEBUG 1712725046, 05:57:26 DEBUG "pod": "ds-cts-2" 05:57:26 DEBUG }, 05:57:26 DEBUG "instance": "10.142.0.25:9100" 05:57:26 DEBUG }, 05:57:26 DEBUG "value": [ 05:57:26 DEBUG "334134674432" 05:57:26 DEBUG ] 05:57:26 DEBUG } 05:57:26 DEBUG "value": [ 05:57:26 DEBUG 05:57:26 DEBUG 05:57:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:57:26 DEBUG ] 05:57:26 DEBUG }, 05:57:26 DEBUG 1712725046, 05:57:26 DEBUG 05:57:26 DEBUG }, 05:57:26 DEBUG "value": [ 05:57:26 DEBUG } 05:57:26 DEBUG "value": [ 05:57:26 DEBUG "0" 05:57:26 DEBUG ] 05:57:26 DEBUG 1712725046, 05:57:26 DEBUG "258987.5" 05:57:26 DEBUG } 05:57:26 DEBUG 1712725046, 05:57:26 DEBUG } 05:57:26 DEBUG ] 05:57:26 DEBUG ] 05:57:26 DEBUG } 05:57:26 DEBUG ] 05:57:26 DEBUG } 05:57:26 DEBUG 05:57:26 DEBUG } 05:57:26 DEBUG 05:57:26 DEBUG "0" 05:57:26 DEBUG } 05:57:26 DEBUG [...] 05:57:26 DEBUG } 05:57:26 DEBUG 05:57:26 DEBUG ] 05:57:26 DEBUG } 05:57:26 DEBUG 05:57:26 DEBUG 05:57:26 DEBUG ] 05:57:26 DEBUG [...] 05:57:26 DEBUG 05:57:26 DEBUG } 05:57:26 DEBUG } 05:57:26 DEBUG 05:57:26 DEBUG 05:57:26 DEBUG 05:57:26 DEBUG "instance": "10.142.0.77:9100" 05:57:26 DEBUG }, 05:57:26 DEBUG "value": [ 05:57:26 DEBUG 1712725046, 05:57:26 DEBUG "42543750.36666667" 05:57:26 DEBUG ] 05:57:26 DEBUG } 05:57:26 DEBUG ] 05:57:26 DEBUG } 05:57:26 DEBUG } 05:57:26 DEBUG 05:57:26 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 05:57:26 DEBUG }, 05:57:26 DEBUG "value": [ 05:57:26 DEBUG 1712725046, 05:57:26 DEBUG "0" 05:57:26 DEBUG ] 05:57:26 DEBUG } 05:57:26 DEBUG ] 05:57:26 DEBUG } 05:57:26 DEBUG } 05:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725046" 05:57:26 INFO [http_cmd]: http status code OK 05:57:26 DEBUG --- status code --- 05:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:26 DEBUG --- http response --- 05:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:57:26 DEBUG ----- output ----- 05:57:26 DEBUG { 05:57:26 DEBUG "status": "success", 05:57:26 DEBUG "data": { 05:57:26 DEBUG "resultType": "vector", 05:57:26 DEBUG "result": [ 05:57:26 DEBUG { 05:57:26 DEBUG "metric": { 05:57:26 DEBUG "pod": "ds-idrepo-0" 05:57:26 DEBUG }, 05:57:26 DEBUG "value": [ 05:57:26 DEBUG 05:57:26 DEBUG 05:57:26 DEBUG 05:57:26 DEBUG [...] 05:57:26 DEBUG 05:57:26 DEBUG 05:57:26 DEBUG 05:57:26 DEBUG "pod": "ds-idrepo-1" 05:57:26 DEBUG }, 05:57:26 DEBUG "value": [ 05:57:26 DEBUG 1712725046, 05:57:26 DEBUG "0" 05:57:26 DEBUG ] 05:57:26 DEBUG } 05:57:26 DEBUG ] 05:57:26 DEBUG } 05:57:26 DEBUG } 05:57:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725047" 05:57:27 INFO [http_cmd]: http status code OK 05:57:27 DEBUG --- status code --- 05:57:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:27 DEBUG --- http response --- 05:57:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:57:27 DEBUG ----- output ----- 05:57:27 DEBUG { 05:57:27 DEBUG "status": "success", 05:57:27 DEBUG "data": { 05:57:27 DEBUG "resultType": "vector", 05:57:27 DEBUG "result": [ 05:57:27 DEBUG { 05:57:27 DEBUG "metric": { 05:57:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:57:27 DEBUG }, 05:57:27 DEBUG "value": [ 05:57:27 DEBUG 05:57:27 DEBUG 05:57:27 DEBUG 05:57:27 DEBUG [...] 05:57:27 DEBUG 05:57:27 DEBUG 05:57:27 DEBUG 05:57:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:57:27 DEBUG }, 05:57:27 DEBUG "value": [ 05:57:27 DEBUG 1712725047, 05:57:27 DEBUG "1438.2333333333333" 05:57:27 DEBUG ] 05:57:27 DEBUG } 05:57:27 DEBUG ] 05:57:27 DEBUG } 05:57:27 DEBUG } 05:57:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725047" 05:57:27 INFO [http_cmd]: http status code OK 05:57:27 DEBUG --- status code --- 05:57:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:27 DEBUG --- http response --- 05:57:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:57:27 DEBUG ----- output ----- 05:57:27 DEBUG { 05:57:27 DEBUG "status": "success", 05:57:27 DEBUG "data": { 05:57:27 DEBUG "resultType": "vector", 05:57:27 DEBUG "result": [ 05:57:27 DEBUG { 05:57:27 DEBUG "metric": { 05:57:27 DEBUG "instance": "10.142.0.25:9100" 05:57:27 DEBUG }, 05:57:27 DEBUG "value": [ 05:57:27 DEBUG 05:57:27 DEBUG 05:57:27 DEBUG 05:57:27 DEBUG [...] 05:57:27 DEBUG 05:57:27 DEBUG 05:57:27 DEBUG 05:57:27 DEBUG "instance": "10.142.0.77:9100" 05:57:27 DEBUG }, 05:57:27 DEBUG "value": [ 05:57:27 DEBUG 1712725047, 05:57:27 DEBUG "0.14453333333333399" 05:57:27 DEBUG ] 05:57:27 DEBUG } 05:57:27 DEBUG ] 05:57:27 DEBUG } 05:57:27 DEBUG } 05:57:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725048" 05:57:28 INFO [http_cmd]: http status code OK 05:57:28 DEBUG --- status code --- 05:57:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:28 DEBUG --- http response --- 05:57:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:57:28 DEBUG ----- output ----- 05:57:28 DEBUG { 05:57:28 DEBUG "status": "success", 05:57:28 DEBUG "data": { 05:57:28 DEBUG "resultType": "vector", 05:57:28 DEBUG "result": [ 05:57:28 DEBUG { 05:57:28 DEBUG "metric": { 05:57:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:57:28 DEBUG }, 05:57:28 DEBUG "value": [ 05:57:28 DEBUG 05:57:28 DEBUG 05:57:28 DEBUG 05:57:28 DEBUG [...] 05:57:28 DEBUG 05:57:28 DEBUG 05:57:28 DEBUG 05:57:28 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:57:28 DEBUG }, 05:57:28 DEBUG "value": [ 05:57:28 DEBUG 1712725048, 05:57:28 DEBUG "0" 05:57:28 DEBUG ] 05:57:28 DEBUG } 05:57:28 DEBUG ] 05:57:28 DEBUG } 05:57:28 DEBUG } 05:57:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712725048" 05:57:28 INFO [http_cmd]: http status code OK 05:57:28 DEBUG --- status code --- 05:57:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:28 DEBUG --- http response --- 05:57:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:57:28 DEBUG ----- output ----- 05:57:28 DEBUG { 05:57:28 DEBUG "status": "success", 05:57:28 DEBUG "data": { 05:57:28 DEBUG "resultType": "vector", 05:57:28 DEBUG "result": [ 05:57:28 DEBUG { 05:57:28 DEBUG "metric": { 05:57:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:57:28 DEBUG }, 05:57:28 DEBUG "value": [ 05:57:28 DEBUG 05:57:28 DEBUG 05:57:28 DEBUG 05:57:28 DEBUG [...] 05:57:28 DEBUG 05:57:28 DEBUG 05:57:28 DEBUG 05:57:28 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:57:28 DEBUG }, 05:57:28 DEBUG "value": [ 05:57:28 DEBUG 1712725048, 05:57:28 DEBUG "4.189633564787491" 05:57:28 DEBUG ] 05:57:28 DEBUG } 05:57:28 DEBUG ] 05:57:28 DEBUG } 05:57:28 DEBUG } 05:57:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712725048" 05:57:28 INFO [http_cmd]: http status code OK 05:57:28 DEBUG --- status code --- 05:57:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:28 DEBUG --- http response --- 05:57:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:57:28 DEBUG ----- output ----- 05:57:28 DEBUG { 05:57:28 DEBUG "status": "success", 05:57:28 DEBUG "data": { 05:57:28 DEBUG "resultType": "vector", 05:57:28 DEBUG "result": [ 05:57:28 DEBUG { 05:57:28 DEBUG "metric": { 05:57:28 DEBUG "pod": "ds-idrepo-0" 05:57:28 DEBUG }, 05:57:28 DEBUG "value": [ 05:57:28 DEBUG 05:57:28 DEBUG 05:57:28 DEBUG 05:57:28 DEBUG [...] 05:57:28 DEBUG 05:57:28 DEBUG 05:57:28 DEBUG 05:57:28 DEBUG "pod": "ds-cts-2" 05:57:28 DEBUG }, 05:57:28 DEBUG "value": [ 05:57:28 DEBUG 1712725048, 05:57:28 DEBUG "0" 05:57:28 DEBUG ] 05:57:28 DEBUG } 05:57:28 DEBUG ] 05:57:28 DEBUG } 05:57:28 DEBUG } 05:57:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725048" 05:57:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725048" 05:57:28 INFO [http_cmd]: http status code OK 05:57:28 DEBUG --- status code --- 05:57:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:28 DEBUG --- http response --- 05:57:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:57:28 INFO [http_cmd]: http status code OK 05:57:28 DEBUG ----- output ----- 05:57:28 DEBUG --- status code --- 05:57:28 DEBUG { 05:57:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:28 DEBUG "status": "success", 05:57:28 DEBUG --- http response --- 05:57:28 DEBUG "data": { 05:57:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:57:28 DEBUG "resultType": "vector", 05:57:28 DEBUG ----- output ----- 05:57:28 DEBUG { 05:57:28 DEBUG "result": [ 05:57:28 DEBUG "status": "success", 05:57:28 DEBUG { 05:57:28 DEBUG "data": { 05:57:28 DEBUG "metric": { 05:57:28 DEBUG "resultType": "vector", 05:57:28 DEBUG "pod": "ds-idrepo-2" 05:57:28 DEBUG "result": [ 05:57:28 DEBUG }, 05:57:28 DEBUG { 05:57:28 DEBUG "value": [ 05:57:28 DEBUG "metric": { 05:57:28 DEBUG 05:57:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:57:28 DEBUG 05:57:28 DEBUG }, 05:57:28 DEBUG 05:57:28 DEBUG "value": [ 05:57:28 DEBUG [...] 05:57:28 DEBUG 05:57:28 DEBUG 05:57:28 DEBUG 05:57:28 DEBUG 05:57:28 DEBUG 05:57:28 DEBUG 05:57:28 DEBUG [...] 05:57:28 DEBUG "pod": "ds-cts-2" 05:57:28 DEBUG 05:57:28 DEBUG }, 05:57:28 DEBUG 05:57:28 DEBUG "value": [ 05:57:28 DEBUG 05:57:28 DEBUG 1712725048, 05:57:28 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:57:28 DEBUG "0.00019865666666666666" 05:57:28 DEBUG }, 05:57:28 DEBUG ] 05:57:28 DEBUG "value": [ 05:57:28 DEBUG } 05:57:28 DEBUG 1712725048, 05:57:28 DEBUG ] 05:57:28 DEBUG "0" 05:57:28 DEBUG } 05:57:28 DEBUG ] 05:57:28 DEBUG } 05:57:28 DEBUG } 05:57:28 DEBUG ] 05:57:28 DEBUG } 05:57:28 DEBUG } 05:57:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725048" 05:57:28 INFO [http_cmd]: http status code OK 05:57:28 DEBUG --- status code --- 05:57:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:28 DEBUG --- http response --- 05:57:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:57:28 DEBUG ----- output ----- 05:57:28 DEBUG { 05:57:28 DEBUG "status": "success", 05:57:28 DEBUG "data": { 05:57:28 DEBUG "resultType": "vector", 05:57:28 DEBUG "result": [ 05:57:28 DEBUG { 05:57:28 DEBUG "metric": { 05:57:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:57:28 DEBUG }, 05:57:28 DEBUG "value": [ 05:57:28 DEBUG 05:57:28 DEBUG 05:57:28 DEBUG 05:57:28 DEBUG [...] 05:57:28 DEBUG 05:57:28 DEBUG 05:57:28 DEBUG 05:57:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:57:28 DEBUG }, 05:57:28 DEBUG "value": [ 05:57:28 DEBUG 1712725048, 05:57:28 DEBUG "1438.3" 05:57:28 DEBUG ] 05:57:28 DEBUG } 05:57:28 DEBUG ] 05:57:28 DEBUG } 05:57:28 DEBUG } 05:57:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712725048" 05:57:28 INFO [http_cmd]: http status code OK 05:57:28 DEBUG --- status code --- 05:57:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:28 DEBUG --- http response --- 05:57:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:57:28 DEBUG ----- output ----- 05:57:28 DEBUG { 05:57:28 DEBUG "status": "success", 05:57:28 DEBUG "data": { 05:57:28 DEBUG "resultType": "vector", 05:57:28 DEBUG "result": [ 05:57:28 DEBUG { 05:57:28 DEBUG "metric": { 05:57:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:57:28 DEBUG }, 05:57:28 DEBUG "value": [ 05:57:28 DEBUG 05:57:28 DEBUG 05:57:28 DEBUG 05:57:28 DEBUG [...] 05:57:28 DEBUG 05:57:28 DEBUG 05:57:28 DEBUG 05:57:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:57:28 DEBUG }, 05:57:28 DEBUG "value": [ 05:57:28 DEBUG 1712725048, 05:57:28 DEBUG "0.046" 05:57:28 DEBUG ] 05:57:28 DEBUG } 05:57:28 DEBUG ] 05:57:28 DEBUG } 05:57:28 DEBUG } 05:57:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725049" 05:57:29 INFO [http_cmd]: http status code OK 05:57:29 DEBUG --- status code --- 05:57:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:29 DEBUG --- http response --- 05:57:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:57:29 DEBUG ----- output ----- 05:57:29 DEBUG { 05:57:29 DEBUG "status": "success", 05:57:29 DEBUG "data": { 05:57:29 DEBUG "resultType": "vector", 05:57:29 DEBUG "result": [ 05:57:29 DEBUG { 05:57:29 DEBUG "metric": { 05:57:29 DEBUG "pod": "ds-idrepo-0" 05:57:29 DEBUG }, 05:57:29 DEBUG "value": [ 05:57:29 DEBUG 05:57:29 DEBUG 05:57:29 DEBUG 05:57:29 DEBUG [...] 05:57:29 DEBUG 05:57:29 DEBUG 05:57:29 DEBUG 05:57:29 DEBUG "pod": "ds-idrepo-1" 05:57:29 DEBUG }, 05:57:29 DEBUG "value": [ 05:57:29 DEBUG 1712725049, 05:57:29 DEBUG "0" 05:57:29 DEBUG ] 05:57:29 DEBUG } 05:57:29 DEBUG ] 05:57:29 DEBUG } 05:57:29 DEBUG } 05:57:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725049" 05:57:29 INFO [http_cmd]: http status code OK 05:57:29 DEBUG --- status code --- 05:57:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:29 DEBUG --- http response --- 05:57:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:57:29 DEBUG ----- output ----- 05:57:29 DEBUG { 05:57:29 DEBUG "status": "success", 05:57:29 DEBUG "data": { 05:57:29 DEBUG "resultType": "vector", 05:57:29 DEBUG "result": [ 05:57:29 DEBUG { 05:57:29 DEBUG "metric": { 05:57:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:57:29 DEBUG }, 05:57:29 DEBUG "value": [ 05:57:29 DEBUG 05:57:29 DEBUG 05:57:29 DEBUG 05:57:29 DEBUG [...] 05:57:29 DEBUG 05:57:29 DEBUG 05:57:29 DEBUG 05:57:29 DEBUG "pod": "am-654c4fcbcf-487cv" 05:57:29 DEBUG }, 05:57:29 DEBUG "value": [ 05:57:29 DEBUG 1712725049, 05:57:29 DEBUG "1430.5" 05:57:29 DEBUG ] 05:57:29 DEBUG } 05:57:29 DEBUG ] 05:57:29 DEBUG } 05:57:29 DEBUG } 05:57:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725049" 05:57:29 INFO [http_cmd]: http status code OK 05:57:29 DEBUG --- status code --- 05:57:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:29 DEBUG --- http response --- 05:57:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:57:29 DEBUG ----- output ----- 05:57:29 DEBUG { 05:57:29 DEBUG "status": "success", 05:57:29 DEBUG "data": { 05:57:29 DEBUG "resultType": "vector", 05:57:29 DEBUG "result": [ 05:57:29 DEBUG { 05:57:29 DEBUG "metric": { 05:57:29 DEBUG "pod": "ds-idrepo-0" 05:57:29 DEBUG }, 05:57:29 DEBUG "value": [ 05:57:29 DEBUG 05:57:29 DEBUG 05:57:29 DEBUG 05:57:29 DEBUG [...] 05:57:29 DEBUG 05:57:29 DEBUG 05:57:29 DEBUG 05:57:29 DEBUG "pod": "ds-idrepo-1" 05:57:29 DEBUG }, 05:57:29 DEBUG "value": [ 05:57:29 DEBUG 1712725049, 05:57:29 DEBUG "0" 05:57:29 DEBUG ] 05:57:29 DEBUG } 05:57:29 DEBUG ] 05:57:29 DEBUG } 05:57:29 DEBUG } 05:57:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712725049" 05:57:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725049" 05:57:29 INFO [http_cmd]: http status code OK 05:57:29 DEBUG --- status code --- 05:57:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:29 DEBUG --- http response --- 05:57:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:57:29 DEBUG ----- output ----- 05:57:29 DEBUG { 05:57:29 DEBUG "status": "success", 05:57:29 DEBUG "data": { 05:57:29 DEBUG "resultType": "vector", 05:57:29 DEBUG "result": [ 05:57:29 DEBUG { 05:57:29 DEBUG "metric": { 05:57:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:57:29 DEBUG }, 05:57:29 DEBUG "value": [ 05:57:29 DEBUG 05:57:29 DEBUG 05:57:29 DEBUG 05:57:29 DEBUG [...] 05:57:29 DEBUG 05:57:29 DEBUG 05:57:29 DEBUG 05:57:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:57:29 DEBUG }, 05:57:29 DEBUG "value": [ 05:57:29 DEBUG 1712725049, 05:57:29 DEBUG "2486743040" 05:57:29 DEBUG ] 05:57:29 DEBUG } 05:57:29 DEBUG ] 05:57:29 DEBUG } 05:57:29 DEBUG } 05:57:29 INFO [http_cmd]: http status code OK 05:57:29 DEBUG --- status code --- 05:57:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:29 DEBUG --- http response --- 05:57:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:57:29 DEBUG ----- output ----- 05:57:29 DEBUG { 05:57:29 DEBUG "status": "success", 05:57:29 DEBUG "data": { 05:57:29 DEBUG "resultType": "vector", 05:57:29 DEBUG "result": [ 05:57:29 DEBUG { 05:57:29 DEBUG "metric": { 05:57:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:57:29 DEBUG }, 05:57:29 DEBUG "value": [ 05:57:29 DEBUG 05:57:29 DEBUG 05:57:29 DEBUG 05:57:29 DEBUG [...] 05:57:29 DEBUG 05:57:29 DEBUG 05:57:29 DEBUG 05:57:29 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 05:57:29 DEBUG }, 05:57:29 DEBUG "value": [ 05:57:29 DEBUG 1712725049, 05:57:29 DEBUG "0" 05:57:29 DEBUG ] 05:57:29 DEBUG } 05:57:29 DEBUG ] 05:57:29 DEBUG } 05:57:29 DEBUG } 05:57:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725049" 05:57:29 INFO [http_cmd]: http status code OK 05:57:29 DEBUG --- status code --- 05:57:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:29 DEBUG --- http response --- 05:57:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:57:29 DEBUG ----- output ----- 05:57:29 DEBUG { 05:57:29 DEBUG "status": "success", 05:57:29 DEBUG "data": { 05:57:29 DEBUG "resultType": "vector", 05:57:29 DEBUG "result": [ 05:57:29 DEBUG { 05:57:29 DEBUG "metric": { 05:57:29 DEBUG "instance": "10.142.0.25:9100" 05:57:29 DEBUG }, 05:57:29 DEBUG "value": [ 05:57:29 DEBUG 05:57:29 DEBUG 05:57:29 DEBUG 05:57:29 DEBUG [...] 05:57:29 DEBUG 05:57:29 DEBUG 05:57:29 DEBUG 05:57:29 DEBUG "instance": "10.142.0.77:9100" 05:57:29 DEBUG }, 05:57:29 DEBUG "value": [ 05:57:29 DEBUG 1712725049, 05:57:29 DEBUG "42545087.66666667" 05:57:29 DEBUG ] 05:57:29 DEBUG } 05:57:29 DEBUG ] 05:57:29 DEBUG } 05:57:29 DEBUG } 05:57:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725049" 05:57:29 INFO [http_cmd]: http status code OK 05:57:29 DEBUG --- status code --- 05:57:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:29 DEBUG --- http response --- 05:57:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:57:29 DEBUG ----- output ----- 05:57:29 DEBUG { 05:57:29 DEBUG "status": "success", 05:57:29 DEBUG "data": { 05:57:29 DEBUG "resultType": "vector", 05:57:29 DEBUG "result": [ 05:57:29 DEBUG { 05:57:29 DEBUG "metric": { 05:57:29 DEBUG "instance": "10.142.0.25:9100" 05:57:29 DEBUG }, 05:57:29 DEBUG "value": [ 05:57:29 DEBUG 05:57:29 DEBUG 05:57:29 DEBUG 05:57:29 DEBUG [...] 05:57:29 DEBUG 05:57:29 DEBUG 05:57:29 DEBUG 05:57:29 DEBUG "instance": "10.142.0.77:9100" 05:57:29 DEBUG }, 05:57:29 DEBUG "value": [ 05:57:29 DEBUG 1712725049, 05:57:29 DEBUG "0.04799999999999963" 05:57:29 DEBUG ] 05:57:29 DEBUG } 05:57:29 DEBUG ] 05:57:29 DEBUG } 05:57:29 DEBUG } 05:57:46 INFO 05:57:46 INFO [loop_until]: kubectl --namespace=xlou top pods 05:57:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:57:46 INFO [loop_until]: OK (rc = 0) 05:57:46 DEBUG --- stdout --- 05:57:46 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10192m 8961Mi am-654c4fcbcf-9scrj 9821m 8439Mi am-654c4fcbcf-v5qb5 8238m 8423Mi ds-cts-0 8192m 12901Mi ds-cts-1 8379m 12735Mi ds-cts-2 7936m 12991Mi ds-idrepo-0 10897m 5622Mi ds-idrepo-1 368m 5671Mi ds-idrepo-2 369m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 9m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4145m 2234Mi 05:57:46 DEBUG --- stderr --- 05:57:46 DEBUG 05:57:53 INFO 05:57:53 INFO [loop_until]: kubectl --namespace=xlou top node 05:57:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:57:53 INFO [loop_until]: OK (rc = 0) 05:57:53 DEBUG --- stdout --- 05:57:53 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 91m 0% 2305Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9196m 57% 9305Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 76m 0% 3257Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13125m 82% 6539Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10944m 68% 9320Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11341m 71% 9788Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 77m 0% 853Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10802m 67% 6294Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8363m 52% 13562Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 7970m 50% 13587Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 415m 2% 6379Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8660m 54% 13340Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 429m 2% 6344Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4267m 26% 5887Mi 10% 05:57:53 DEBUG --- stderr --- 05:57:53 DEBUG 05:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712725106" 05:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725106" 05:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725106" 05:58:26 ERROR [http_cmd]: ERROR 05:58:26 ERROR --- status code --- 05:58:26 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:58:26 INFO [http_cmd]: http status code OK 05:58:26 ERROR --- http response --- 05:58:26 DEBUG --- status code --- 05:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:26 DEBUG --- http response --- 05:58:26 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:58:26 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:58:26 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:58:26 INFO [http_cmd]: http status code OK 05:58:26 DEBUG --- status code --- 05:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:26 DEBUG --- http response --- 05:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:58:26 DEBUG ----- output ----- 05:58:26 DEBUG { 05:58:26 DEBUG "status": "success", 05:58:26 DEBUG "data": { 05:58:26 DEBUG "resultType": "vector", 05:58:26 DEBUG "result": [ 05:58:26 DEBUG { 05:58:26 DEBUG "metric": { 05:58:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:58:26 DEBUG }, 05:58:26 DEBUG "value": [ 05:58:26 DEBUG 05:58:26 DEBUG 05:58:26 DEBUG 05:58:26 DEBUG [...] 05:58:26 DEBUG 05:58:26 DEBUG 05:58:26 DEBUG 05:58:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:58:26 DEBUG }, 05:58:26 DEBUG "value": [ 05:58:26 DEBUG 1712725106, 05:58:26 DEBUG "0" 05:58:26 DEBUG ] 05:58:26 DEBUG } 05:58:26 DEBUG ] 05:58:26 DEBUG } 05:58:26 DEBUG } 05:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725106" 05:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725106" 05:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725106" 05:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725106" 05:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725106" 05:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725106" 05:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725106" 05:58:26 INFO [http_cmd]: http status code OK 05:58:26 DEBUG --- status code --- 05:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:26 INFO [http_cmd]: http status code OK 05:58:26 DEBUG --- status code --- 05:58:26 INFO [http_cmd]: http status code OK 05:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:26 DEBUG --- status code --- 05:58:26 DEBUG --- http response --- 05:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:58:26 DEBUG --- http response --- 05:58:26 INFO [http_cmd]: http status code OK 05:58:26 DEBUG --- http response --- 05:58:26 INFO [http_cmd]: http status code OK 05:58:26 DEBUG ----- output ----- 05:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:58:26 DEBUG --- status code --- 05:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:58:26 DEBUG --- status code --- 05:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:26 DEBUG { 05:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:26 DEBUG ----- output ----- 05:58:26 DEBUG --- http response --- 05:58:26 DEBUG ----- output ----- 05:58:26 DEBUG "status": "success", 05:58:26 DEBUG --- http response --- 05:58:26 DEBUG { 05:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:58:26 DEBUG { 05:58:26 DEBUG "data": { 05:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:58:26 DEBUG "status": "success", 05:58:26 DEBUG "data": { 05:58:26 DEBUG "status": "success", 05:58:26 DEBUG "data": { 05:58:26 DEBUG ----- output ----- 05:58:26 DEBUG ----- output ----- 05:58:26 DEBUG "resultType": "vector", 05:58:26 DEBUG "resultType": "vector", 05:58:26 DEBUG "result": [ 05:58:26 DEBUG { 05:58:26 DEBUG { 05:58:26 DEBUG "status": "success", 05:58:26 DEBUG "data": { 05:58:26 DEBUG "metric": { 05:58:26 DEBUG "instance": "10.142.0.25:9100" 05:58:26 DEBUG }, 05:58:26 DEBUG "result": [ 05:58:26 DEBUG { 05:58:26 DEBUG "metric": { 05:58:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:58:26 DEBUG }, 05:58:26 DEBUG "value": [ 05:58:26 DEBUG 05:58:26 DEBUG 05:58:26 DEBUG 05:58:26 DEBUG { 05:58:26 DEBUG "status": "success", 05:58:26 DEBUG "data": { 05:58:26 DEBUG [...] 05:58:26 DEBUG 05:58:26 DEBUG "resultType": "vector", 05:58:26 DEBUG "result": [ 05:58:26 DEBUG { 05:58:26 DEBUG 05:58:26 DEBUG 05:58:26 DEBUG "metric": { 05:58:26 DEBUG "instance": "10.142.0.25:9100" 05:58:26 DEBUG }, 05:58:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:58:26 DEBUG "resultType": "vector", 05:58:26 INFO [http_cmd]: http status code OK 05:58:26 DEBUG "value": [ 05:58:26 DEBUG 05:58:26 DEBUG 05:58:26 DEBUG }, 05:58:26 DEBUG "result": [ 05:58:26 INFO [http_cmd]: http status code OK 05:58:26 DEBUG --- status code --- 05:58:26 DEBUG 05:58:26 DEBUG [...] 05:58:26 DEBUG 05:58:26 DEBUG "value": [ 05:58:26 DEBUG 1712725106, 05:58:26 DEBUG "259034" 05:58:26 DEBUG --- status code --- 05:58:26 DEBUG 05:58:26 DEBUG 05:58:26 DEBUG "instance": "10.142.0.77:9100" 05:58:26 DEBUG ] 05:58:26 DEBUG } 05:58:26 DEBUG ] 05:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:26 DEBUG }, 05:58:26 DEBUG "resultType": "vector", 05:58:26 DEBUG "result": [ 05:58:26 DEBUG { 05:58:26 DEBUG "metric": { 05:58:26 DEBUG "pod": "ds-idrepo-0" 05:58:26 DEBUG --- http response --- 05:58:26 DEBUG "value": [ 05:58:26 DEBUG 1712725106, 05:58:26 DEBUG "42417841.3" 05:58:26 DEBUG ] 05:58:26 DEBUG } 05:58:26 DEBUG ] 05:58:26 DEBUG } 05:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 05:58:26 DEBUG } 05:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:26 DEBUG --- http response --- 05:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:58:26 DEBUG ----- output ----- 05:58:26 DEBUG { 05:58:26 DEBUG ----- output ----- 05:58:26 DEBUG }, 05:58:26 DEBUG "value": [ 05:58:26 DEBUG { 05:58:26 DEBUG "status": "success", 05:58:26 DEBUG "data": { 05:58:26 DEBUG { 05:58:26 DEBUG "status": "success", 05:58:26 DEBUG 05:58:26 DEBUG 05:58:26 DEBUG 05:58:26 DEBUG [...] 05:58:26 DEBUG 05:58:26 DEBUG "data": { 05:58:26 DEBUG 05:58:26 DEBUG 05:58:26 DEBUG "pod": "ds-cts-2" 05:58:26 DEBUG }, 05:58:26 DEBUG "value": [ 05:58:26 DEBUG "resultType": "vector", 05:58:26 DEBUG 1712725106, 05:58:26 DEBUG "0" 05:58:26 DEBUG ] 05:58:26 DEBUG } 05:58:26 DEBUG ] 05:58:26 DEBUG } 05:58:26 DEBUG "resultType": "vector", 05:58:26 DEBUG "result": [ 05:58:26 DEBUG { 05:58:26 DEBUG "metric": { 05:58:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:58:26 DEBUG "result": [ 05:58:26 DEBUG { 05:58:26 DEBUG } 05:58:26 DEBUG "metric": { 05:58:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:58:26 DEBUG }, 05:58:26 DEBUG "value": [ 05:58:26 DEBUG 05:58:26 DEBUG 05:58:26 DEBUG "value": [ 05:58:26 DEBUG "metric": { 05:58:26 DEBUG "pod": "ds-idrepo-0" 05:58:26 DEBUG }, 05:58:26 DEBUG 05:58:26 DEBUG [...] 05:58:26 DEBUG 05:58:26 DEBUG 05:58:26 DEBUG "value": [ 05:58:26 DEBUG 05:58:26 DEBUG 05:58:26 DEBUG 05:58:26 DEBUG 05:58:26 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 05:58:26 DEBUG }, 05:58:26 DEBUG [...] 05:58:26 DEBUG } 05:58:26 DEBUG } 05:58:26 DEBUG }, 05:58:26 DEBUG "value": [ 05:58:26 DEBUG 05:58:26 DEBUG 05:58:26 DEBUG 05:58:26 DEBUG 05:58:26 DEBUG 05:58:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:58:26 DEBUG 05:58:26 DEBUG [...] 05:58:26 DEBUG 05:58:26 DEBUG }, 05:58:26 DEBUG "value": [ 05:58:26 DEBUG 1712725106, 05:58:26 DEBUG 05:58:26 DEBUG 05:58:26 DEBUG 1712725106, 05:58:26 DEBUG "0" 05:58:26 DEBUG "instance": "10.142.0.77:9100" 05:58:26 DEBUG }, 05:58:26 DEBUG "1427.4" 05:58:26 DEBUG ] 05:58:26 DEBUG "value": [ 05:58:26 DEBUG 1712725106, 05:58:26 DEBUG "335976428544" 05:58:26 DEBUG ] 05:58:26 DEBUG } 05:58:26 DEBUG ] 05:58:26 DEBUG } 05:58:26 DEBUG ] 05:58:26 DEBUG } 05:58:26 DEBUG ] 05:58:26 DEBUG } 05:58:26 DEBUG } 05:58:26 DEBUG } 05:58:26 DEBUG } 05:58:26 DEBUG "value": [ 05:58:26 DEBUG ] 05:58:26 DEBUG 05:58:26 DEBUG } 05:58:26 DEBUG 05:58:26 DEBUG } 05:58:26 DEBUG 05:58:26 DEBUG [...] 05:58:26 DEBUG 05:58:26 DEBUG 05:58:26 DEBUG 05:58:26 DEBUG "pod": "ds-cts-2" 05:58:26 DEBUG }, 05:58:26 DEBUG "value": [ 05:58:26 DEBUG 1712725106, 05:58:26 DEBUG "0" 05:58:26 DEBUG ] 05:58:26 DEBUG } 05:58:26 DEBUG ] 05:58:26 DEBUG } 05:58:26 DEBUG } 05:58:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725106" 05:58:27 INFO [http_cmd]: http status code OK 05:58:27 DEBUG --- status code --- 05:58:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:27 DEBUG --- http response --- 05:58:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:58:27 DEBUG ----- output ----- 05:58:27 DEBUG { 05:58:27 DEBUG "status": "success", 05:58:27 DEBUG "data": { 05:58:27 DEBUG "resultType": "vector", 05:58:27 DEBUG "result": [ 05:58:27 DEBUG { 05:58:27 DEBUG "metric": { 05:58:27 DEBUG "pod": "ds-idrepo-0" 05:58:27 DEBUG }, 05:58:27 DEBUG "value": [ 05:58:27 DEBUG 05:58:27 DEBUG 05:58:27 DEBUG 05:58:27 DEBUG [...] 05:58:27 DEBUG 05:58:27 DEBUG 05:58:27 DEBUG 05:58:27 DEBUG "pod": "ds-idrepo-1" 05:58:27 DEBUG }, 05:58:27 DEBUG "value": [ 05:58:27 DEBUG 1712725106, 05:58:27 DEBUG "0" 05:58:27 DEBUG ] 05:58:27 DEBUG } 05:58:27 DEBUG ] 05:58:27 DEBUG } 05:58:27 DEBUG } 05:58:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725107" 05:58:27 INFO [http_cmd]: http status code OK 05:58:27 DEBUG --- status code --- 05:58:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:27 DEBUG --- http response --- 05:58:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:58:27 DEBUG ----- output ----- 05:58:27 DEBUG { 05:58:27 DEBUG "status": "success", 05:58:27 DEBUG "data": { 05:58:27 DEBUG "resultType": "vector", 05:58:27 DEBUG "result": [ 05:58:27 DEBUG { 05:58:27 DEBUG "metric": { 05:58:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:58:27 DEBUG }, 05:58:27 DEBUG "value": [ 05:58:27 DEBUG 05:58:27 DEBUG 05:58:27 DEBUG 05:58:27 DEBUG [...] 05:58:27 DEBUG 05:58:27 DEBUG 05:58:27 DEBUG 05:58:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:58:27 DEBUG }, 05:58:27 DEBUG "value": [ 05:58:27 DEBUG 1712725107, 05:58:27 DEBUG "1432.1" 05:58:27 DEBUG ] 05:58:27 DEBUG } 05:58:27 DEBUG ] 05:58:27 DEBUG } 05:58:27 DEBUG } 05:58:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725107" 05:58:27 INFO [http_cmd]: http status code OK 05:58:27 DEBUG --- status code --- 05:58:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:27 DEBUG --- http response --- 05:58:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:58:27 DEBUG ----- output ----- 05:58:27 DEBUG { 05:58:27 DEBUG "status": "success", 05:58:27 DEBUG "data": { 05:58:27 DEBUG "resultType": "vector", 05:58:27 DEBUG "result": [ 05:58:27 DEBUG { 05:58:27 DEBUG "metric": { 05:58:27 DEBUG "instance": "10.142.0.25:9100" 05:58:27 DEBUG }, 05:58:27 DEBUG "value": [ 05:58:27 DEBUG 05:58:27 DEBUG 05:58:27 DEBUG 05:58:27 DEBUG [...] 05:58:27 DEBUG 05:58:27 DEBUG 05:58:27 DEBUG 05:58:27 DEBUG "instance": "10.142.0.77:9100" 05:58:27 DEBUG }, 05:58:27 DEBUG "value": [ 05:58:27 DEBUG 1712725107, 05:58:27 DEBUG "0.15446666666666786" 05:58:27 DEBUG ] 05:58:27 DEBUG } 05:58:27 DEBUG ] 05:58:27 DEBUG } 05:58:27 DEBUG } 05:58:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725108" 05:58:28 INFO [http_cmd]: http status code OK 05:58:28 DEBUG --- status code --- 05:58:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:28 DEBUG --- http response --- 05:58:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:58:28 DEBUG ----- output ----- 05:58:28 DEBUG { 05:58:28 DEBUG "status": "success", 05:58:28 DEBUG "data": { 05:58:28 DEBUG "resultType": "vector", 05:58:28 DEBUG "result": [ 05:58:28 DEBUG { 05:58:28 DEBUG "metric": { 05:58:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:58:28 DEBUG }, 05:58:28 DEBUG "value": [ 05:58:28 DEBUG 05:58:28 DEBUG 05:58:28 DEBUG 05:58:28 DEBUG [...] 05:58:28 DEBUG 05:58:28 DEBUG 05:58:28 DEBUG 05:58:28 DEBUG "pod": "ds-idrepo-2" 05:58:28 DEBUG }, 05:58:28 DEBUG "value": [ 05:58:28 DEBUG 1712725108, 05:58:28 DEBUG "0" 05:58:28 DEBUG ] 05:58:28 DEBUG } 05:58:28 DEBUG ] 05:58:28 DEBUG } 05:58:28 DEBUG } 05:58:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712725108" 05:58:28 INFO [http_cmd]: http status code OK 05:58:28 DEBUG --- status code --- 05:58:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:28 DEBUG --- http response --- 05:58:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:58:28 DEBUG ----- output ----- 05:58:28 DEBUG { 05:58:28 DEBUG "status": "success", 05:58:28 DEBUG "data": { 05:58:28 DEBUG "resultType": "vector", 05:58:28 DEBUG "result": [ 05:58:28 DEBUG { 05:58:28 DEBUG "metric": { 05:58:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:58:28 DEBUG }, 05:58:28 DEBUG "value": [ 05:58:28 DEBUG 05:58:28 DEBUG 05:58:28 DEBUG 05:58:28 DEBUG [...] 05:58:28 DEBUG 05:58:28 DEBUG 05:58:28 DEBUG 05:58:28 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:58:28 DEBUG }, 05:58:28 DEBUG "value": [ 05:58:28 DEBUG 1712725108, 05:58:28 DEBUG "4.152451882419364" 05:58:28 DEBUG ] 05:58:28 DEBUG } 05:58:28 DEBUG ] 05:58:28 DEBUG } 05:58:28 DEBUG } 05:58:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712725108" 05:58:28 INFO [http_cmd]: http status code OK 05:58:28 DEBUG --- status code --- 05:58:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:28 DEBUG --- http response --- 05:58:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:58:28 DEBUG ----- output ----- 05:58:28 DEBUG { 05:58:28 DEBUG "status": "success", 05:58:28 DEBUG "data": { 05:58:28 DEBUG "resultType": "vector", 05:58:28 DEBUG "result": [ 05:58:28 DEBUG { 05:58:28 DEBUG "metric": { 05:58:28 DEBUG "pod": "ds-idrepo-0" 05:58:28 DEBUG }, 05:58:28 DEBUG "value": [ 05:58:28 DEBUG 05:58:28 DEBUG 05:58:28 DEBUG 05:58:28 DEBUG [...] 05:58:28 DEBUG 05:58:28 DEBUG 05:58:28 DEBUG 05:58:28 DEBUG "pod": "ds-cts-2" 05:58:28 DEBUG }, 05:58:28 DEBUG "value": [ 05:58:28 DEBUG 1712725108, 05:58:28 DEBUG "0" 05:58:28 DEBUG ] 05:58:28 DEBUG } 05:58:28 DEBUG ] 05:58:28 DEBUG } 05:58:28 DEBUG } 05:58:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725108" 05:58:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725108" 05:58:28 INFO [http_cmd]: http status code OK 05:58:28 INFO [http_cmd]: http status code OK 05:58:28 DEBUG --- status code --- 05:58:28 DEBUG --- status code --- 05:58:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:28 DEBUG --- http response --- 05:58:28 DEBUG --- http response --- 05:58:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:58:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:58:28 DEBUG ----- output ----- 05:58:28 DEBUG ----- output ----- 05:58:28 DEBUG { 05:58:28 DEBUG { 05:58:28 DEBUG "status": "success", 05:58:28 DEBUG "status": "success", 05:58:28 DEBUG "data": { 05:58:28 DEBUG "data": { 05:58:28 DEBUG "resultType": "vector", 05:58:28 DEBUG "resultType": "vector", 05:58:28 DEBUG "result": [ 05:58:28 DEBUG "result": [ 05:58:28 DEBUG { 05:58:28 DEBUG { 05:58:28 DEBUG "metric": { 05:58:28 DEBUG "metric": { 05:58:28 DEBUG "pod": "ds-idrepo-2" 05:58:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:58:28 DEBUG }, 05:58:28 DEBUG }, 05:58:28 DEBUG "value": [ 05:58:28 DEBUG "value": [ 05:58:28 DEBUG 05:58:28 DEBUG 05:58:28 DEBUG 05:58:28 DEBUG 05:58:28 DEBUG 05:58:28 DEBUG 05:58:28 DEBUG [...] 05:58:28 DEBUG [...] 05:58:28 DEBUG 05:58:28 DEBUG 05:58:28 DEBUG 05:58:28 DEBUG 05:58:28 DEBUG 05:58:28 DEBUG 05:58:28 DEBUG "pod": "ds-cts-2" 05:58:28 DEBUG "pod": "ds-idrepo-2" 05:58:28 DEBUG }, 05:58:28 DEBUG }, 05:58:28 DEBUG "value": [ 05:58:28 DEBUG "value": [ 05:58:28 DEBUG 1712725108, 05:58:28 DEBUG "0" 05:58:28 DEBUG 1712725108, 05:58:28 DEBUG ] 05:58:28 DEBUG "0.00006666666666666667" 05:58:28 DEBUG } 05:58:28 DEBUG ] 05:58:28 DEBUG ] 05:58:28 DEBUG } 05:58:28 DEBUG } 05:58:28 DEBUG ] 05:58:28 DEBUG } 05:58:28 DEBUG } 05:58:28 DEBUG } 05:58:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725108" 05:58:28 INFO [http_cmd]: http status code OK 05:58:28 DEBUG --- status code --- 05:58:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:28 DEBUG --- http response --- 05:58:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:58:28 DEBUG ----- output ----- 05:58:28 DEBUG { 05:58:28 DEBUG "status": "success", 05:58:28 DEBUG "data": { 05:58:28 DEBUG "resultType": "vector", 05:58:28 DEBUG "result": [ 05:58:28 DEBUG { 05:58:28 DEBUG "metric": { 05:58:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:58:28 DEBUG }, 05:58:28 DEBUG "value": [ 05:58:28 DEBUG 05:58:28 DEBUG 05:58:28 DEBUG 05:58:28 DEBUG [...] 05:58:28 DEBUG 05:58:28 DEBUG 05:58:28 DEBUG 05:58:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:58:28 DEBUG }, 05:58:28 DEBUG "value": [ 05:58:28 DEBUG 1712725108, 05:58:28 DEBUG "1431.4666666666667" 05:58:28 DEBUG ] 05:58:28 DEBUG } 05:58:28 DEBUG ] 05:58:28 DEBUG } 05:58:28 DEBUG } 05:58:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712725108" 05:58:28 INFO [http_cmd]: http status code OK 05:58:28 DEBUG --- status code --- 05:58:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:28 DEBUG --- http response --- 05:58:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:58:28 DEBUG ----- output ----- 05:58:28 DEBUG { 05:58:28 DEBUG "status": "success", 05:58:28 DEBUG "data": { 05:58:28 DEBUG "resultType": "vector", 05:58:28 DEBUG "result": [ 05:58:28 DEBUG { 05:58:28 DEBUG "metric": { 05:58:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:58:28 DEBUG }, 05:58:28 DEBUG "value": [ 05:58:28 DEBUG 05:58:28 DEBUG 05:58:28 DEBUG 05:58:28 DEBUG [...] 05:58:28 DEBUG 05:58:28 DEBUG 05:58:28 DEBUG 05:58:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:58:28 DEBUG }, 05:58:28 DEBUG "value": [ 05:58:28 DEBUG 1712725108, 05:58:28 DEBUG "0.046" 05:58:28 DEBUG ] 05:58:28 DEBUG } 05:58:28 DEBUG ] 05:58:28 DEBUG } 05:58:28 DEBUG } 05:58:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725109" 05:58:29 INFO [http_cmd]: http status code OK 05:58:29 DEBUG --- status code --- 05:58:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:29 DEBUG --- http response --- 05:58:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:58:29 DEBUG ----- output ----- 05:58:29 DEBUG { 05:58:29 DEBUG "status": "success", 05:58:29 DEBUG "data": { 05:58:29 DEBUG "resultType": "vector", 05:58:29 DEBUG "result": [ 05:58:29 DEBUG { 05:58:29 DEBUG "metric": { 05:58:29 DEBUG "pod": "ds-idrepo-0" 05:58:29 DEBUG }, 05:58:29 DEBUG "value": [ 05:58:29 DEBUG 05:58:29 DEBUG 05:58:29 DEBUG 05:58:29 DEBUG [...] 05:58:29 DEBUG 05:58:29 DEBUG 05:58:29 DEBUG 05:58:29 DEBUG "pod": "ds-idrepo-1" 05:58:29 DEBUG }, 05:58:29 DEBUG "value": [ 05:58:29 DEBUG 1712725109, 05:58:29 DEBUG "0" 05:58:29 DEBUG ] 05:58:29 DEBUG } 05:58:29 DEBUG ] 05:58:29 DEBUG } 05:58:29 DEBUG } 05:58:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725109" 05:58:29 INFO [http_cmd]: http status code OK 05:58:29 DEBUG --- status code --- 05:58:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:29 DEBUG --- http response --- 05:58:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:58:29 DEBUG ----- output ----- 05:58:29 DEBUG { 05:58:29 DEBUG "status": "success", 05:58:29 DEBUG "data": { 05:58:29 DEBUG "resultType": "vector", 05:58:29 DEBUG "result": [ 05:58:29 DEBUG { 05:58:29 DEBUG "metric": { 05:58:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:58:29 DEBUG }, 05:58:29 DEBUG "value": [ 05:58:29 DEBUG 05:58:29 DEBUG 05:58:29 DEBUG 05:58:29 DEBUG [...] 05:58:29 DEBUG 05:58:29 DEBUG 05:58:29 DEBUG 05:58:29 DEBUG "pod": "am-654c4fcbcf-487cv" 05:58:29 DEBUG }, 05:58:29 DEBUG "value": [ 05:58:29 DEBUG 1712725109, 05:58:29 DEBUG "1439.3666666666666" 05:58:29 DEBUG ] 05:58:29 DEBUG } 05:58:29 DEBUG ] 05:58:29 DEBUG } 05:58:29 DEBUG } 05:58:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725109" 05:58:29 INFO [http_cmd]: http status code OK 05:58:29 DEBUG --- status code --- 05:58:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:29 DEBUG --- http response --- 05:58:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:58:29 DEBUG ----- output ----- 05:58:29 DEBUG { 05:58:29 DEBUG "status": "success", 05:58:29 DEBUG "data": { 05:58:29 DEBUG "resultType": "vector", 05:58:29 DEBUG "result": [ 05:58:29 DEBUG { 05:58:29 DEBUG "metric": { 05:58:29 DEBUG "pod": "ds-idrepo-0" 05:58:29 DEBUG }, 05:58:29 DEBUG "value": [ 05:58:29 DEBUG 05:58:29 DEBUG 05:58:29 DEBUG 05:58:29 DEBUG [...] 05:58:29 DEBUG 05:58:29 DEBUG 05:58:29 DEBUG 05:58:29 DEBUG "pod": "ds-idrepo-1" 05:58:29 DEBUG }, 05:58:29 DEBUG "value": [ 05:58:29 DEBUG 1712725109, 05:58:29 DEBUG "0" 05:58:29 DEBUG ] 05:58:29 DEBUG } 05:58:29 DEBUG ] 05:58:29 DEBUG } 05:58:29 DEBUG } 05:58:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712725109" 05:58:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725109" 05:58:29 INFO [http_cmd]: http status code OK 05:58:29 DEBUG --- status code --- 05:58:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:29 DEBUG --- http response --- 05:58:29 INFO [http_cmd]: http status code OK 05:58:29 DEBUG --- status code --- 05:58:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:58:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:29 DEBUG ----- output ----- 05:58:29 DEBUG { 05:58:29 DEBUG "status": "success", 05:58:29 DEBUG "data": { 05:58:29 DEBUG --- http response --- 05:58:29 DEBUG "resultType": "vector", 05:58:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:58:29 DEBUG "result": [ 05:58:29 DEBUG ----- output ----- 05:58:29 DEBUG { 05:58:29 DEBUG { 05:58:29 DEBUG "metric": { 05:58:29 DEBUG "status": "success", 05:58:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:58:29 DEBUG "data": { 05:58:29 DEBUG "resultType": "vector", 05:58:29 DEBUG }, 05:58:29 DEBUG "result": [ 05:58:29 DEBUG "value": [ 05:58:29 DEBUG { 05:58:29 DEBUG 05:58:29 DEBUG "metric": { 05:58:29 DEBUG 05:58:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:58:29 DEBUG 05:58:29 DEBUG }, 05:58:29 DEBUG [...] 05:58:29 DEBUG "value": [ 05:58:29 DEBUG 05:58:29 DEBUG 05:58:29 DEBUG 05:58:29 DEBUG 05:58:29 DEBUG 05:58:29 DEBUG 05:58:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:58:29 DEBUG [...] 05:58:29 DEBUG }, 05:58:29 DEBUG 05:58:29 DEBUG "value": [ 05:58:29 DEBUG 05:58:29 DEBUG 1712725109, 05:58:29 DEBUG "2486521856" 05:58:29 DEBUG 05:58:29 DEBUG ] 05:58:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:58:29 DEBUG } 05:58:29 DEBUG }, 05:58:29 DEBUG ] 05:58:29 DEBUG "value": [ 05:58:29 DEBUG } 05:58:29 DEBUG 1712725109, 05:58:29 DEBUG } 05:58:29 DEBUG "0" 05:58:29 DEBUG ] 05:58:29 DEBUG } 05:58:29 DEBUG ] 05:58:29 DEBUG } 05:58:29 DEBUG } 05:58:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725109" 05:58:29 INFO [http_cmd]: http status code OK 05:58:29 DEBUG --- status code --- 05:58:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:29 DEBUG --- http response --- 05:58:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:58:29 DEBUG ----- output ----- 05:58:29 DEBUG { 05:58:29 DEBUG "status": "success", 05:58:29 DEBUG "data": { 05:58:29 DEBUG "resultType": "vector", 05:58:29 DEBUG "result": [ 05:58:29 DEBUG { 05:58:29 DEBUG "metric": { 05:58:29 DEBUG "instance": "10.142.0.25:9100" 05:58:29 DEBUG }, 05:58:29 DEBUG "value": [ 05:58:29 DEBUG 05:58:29 DEBUG 05:58:29 DEBUG 05:58:29 DEBUG [...] 05:58:29 DEBUG 05:58:29 DEBUG 05:58:29 DEBUG 05:58:29 DEBUG "instance": "10.142.0.77:9100" 05:58:29 DEBUG }, 05:58:29 DEBUG "value": [ 05:58:29 DEBUG 1712725109, 05:58:29 DEBUG "42419714.3" 05:58:29 DEBUG ] 05:58:29 DEBUG } 05:58:29 DEBUG ] 05:58:29 DEBUG } 05:58:29 DEBUG } 05:58:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725109" 05:58:29 INFO [http_cmd]: http status code OK 05:58:29 DEBUG --- status code --- 05:58:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:29 DEBUG --- http response --- 05:58:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:58:29 DEBUG ----- output ----- 05:58:29 DEBUG { 05:58:29 DEBUG "status": "success", 05:58:29 DEBUG "data": { 05:58:29 DEBUG "resultType": "vector", 05:58:29 DEBUG "result": [ 05:58:29 DEBUG { 05:58:29 DEBUG "metric": { 05:58:29 DEBUG "instance": "10.142.0.25:9100" 05:58:29 DEBUG }, 05:58:29 DEBUG "value": [ 05:58:29 DEBUG 05:58:29 DEBUG 05:58:29 DEBUG 05:58:29 DEBUG [...] 05:58:29 DEBUG 05:58:29 DEBUG 05:58:29 DEBUG 05:58:29 DEBUG "instance": "10.142.0.77:9100" 05:58:29 DEBUG }, 05:58:29 DEBUG "value": [ 05:58:29 DEBUG 1712725109, 05:58:29 DEBUG "0.05066666666666666" 05:58:29 DEBUG ] 05:58:29 DEBUG } 05:58:29 DEBUG ] 05:58:29 DEBUG } 05:58:29 DEBUG } 05:58:46 INFO 05:58:46 INFO [loop_until]: kubectl --namespace=xlou top pods 05:58:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:58:46 INFO [loop_until]: OK (rc = 0) 05:58:46 DEBUG --- stdout --- 05:58:46 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10485m 8965Mi am-654c4fcbcf-9scrj 9691m 8438Mi am-654c4fcbcf-v5qb5 8466m 8425Mi ds-cts-0 8004m 13360Mi ds-cts-1 8966m 13381Mi ds-cts-2 8175m 12605Mi ds-idrepo-0 10844m 5622Mi ds-idrepo-1 402m 5671Mi ds-idrepo-2 393m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1191Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 9m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4282m 2230Mi 05:58:46 DEBUG --- stderr --- 05:58:46 DEBUG 05:58:53 INFO 05:58:53 INFO [loop_until]: kubectl --namespace=xlou top node 05:58:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:58:53 INFO [loop_until]: OK (rc = 0) 05:58:53 DEBUG --- stdout --- 05:58:53 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 95m 0% 2306Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9274m 58% 9300Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 81m 0% 3259Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 12799m 80% 6541Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10918m 68% 9319Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11436m 71% 9782Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 79m 0% 853Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10876m 68% 6294Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8616m 54% 14003Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 8239m 51% 13240Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 442m 2% 6378Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8746m 55% 14046Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 464m 2% 6341Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4297m 27% 5885Mi 10% 05:58:53 DEBUG --- stderr --- 05:58:54 DEBUG 05:59:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725166" 05:59:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712725166" 05:59:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725166" 05:59:26 ERROR [http_cmd]: ERROR 05:59:26 ERROR --- status code --- 05:59:26 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:59:26 ERROR --- http response --- 05:59:26 INFO [http_cmd]: http status code OK 05:59:26 DEBUG --- status code --- 05:59:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:26 DEBUG --- http response --- 05:59:26 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:59:26 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:59:26 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:59:26 INFO [http_cmd]: http status code OK 05:59:26 DEBUG --- status code --- 05:59:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:26 DEBUG --- http response --- 05:59:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:59:26 DEBUG ----- output ----- 05:59:26 DEBUG { 05:59:26 DEBUG "status": "success", 05:59:26 DEBUG "data": { 05:59:26 DEBUG "resultType": "vector", 05:59:26 DEBUG "result": [ 05:59:26 DEBUG { 05:59:26 DEBUG "metric": { 05:59:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:59:26 DEBUG }, 05:59:26 DEBUG "value": [ 05:59:26 DEBUG 05:59:26 DEBUG 05:59:26 DEBUG 05:59:26 DEBUG [...] 05:59:26 DEBUG 05:59:26 DEBUG 05:59:26 DEBUG 05:59:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:59:26 DEBUG }, 05:59:26 DEBUG "value": [ 05:59:26 DEBUG 1712725166, 05:59:26 DEBUG "0" 05:59:26 DEBUG ] 05:59:26 DEBUG } 05:59:26 DEBUG ] 05:59:26 DEBUG } 05:59:26 DEBUG } 05:59:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725166" 05:59:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725166" 05:59:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725166" 05:59:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725166" 05:59:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725166" 05:59:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725166" 05:59:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725166" 05:59:26 INFO [http_cmd]: http status code OK 05:59:26 DEBUG --- status code --- 05:59:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:26 DEBUG --- http response --- 05:59:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:59:26 DEBUG ----- output ----- 05:59:26 DEBUG { 05:59:26 DEBUG "status": "success", 05:59:26 DEBUG "data": { 05:59:26 DEBUG "resultType": "vector", 05:59:26 DEBUG "result": [ 05:59:26 DEBUG { 05:59:26 DEBUG "metric": { 05:59:26 DEBUG "instance": "10.142.0.25:9100" 05:59:26 DEBUG }, 05:59:26 DEBUG "value": [ 05:59:26 DEBUG 05:59:26 DEBUG 05:59:26 DEBUG 05:59:26 DEBUG [...] 05:59:26 DEBUG 05:59:26 DEBUG 05:59:26 DEBUG 05:59:26 DEBUG "instance": "10.142.0.77:9100" 05:59:26 DEBUG }, 05:59:26 DEBUG "value": [ 05:59:26 DEBUG 1712725166, 05:59:26 DEBUG "42711705.099999994" 05:59:26 DEBUG ] 05:59:26 DEBUG } 05:59:26 DEBUG ] 05:59:26 DEBUG } 05:59:26 DEBUG } 05:59:26 INFO [http_cmd]: http status code OK 05:59:26 DEBUG --- status code --- 05:59:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:26 INFO [http_cmd]: http status code OK 05:59:26 DEBUG --- http response --- 05:59:26 INFO [http_cmd]: http status code OK 05:59:26 DEBUG --- status code --- 05:59:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:26 DEBUG --- http response --- 05:59:26 DEBUG --- status code --- 05:59:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:59:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:59:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:26 DEBUG ----- output ----- 05:59:26 DEBUG ----- output ----- 05:59:26 DEBUG --- http response --- 05:59:26 DEBUG { 05:59:26 DEBUG { 05:59:26 INFO [http_cmd]: http status code OK 05:59:26 DEBUG --- status code --- 05:59:26 INFO [http_cmd]: http status code OK 05:59:26 DEBUG --- status code --- 05:59:26 DEBUG "status": "success", 05:59:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:26 DEBUG --- http response --- 05:59:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:59:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:26 DEBUG "data": { 05:59:26 DEBUG ----- output ----- 05:59:26 DEBUG { 05:59:26 DEBUG "status": "success", 05:59:26 DEBUG --- http response --- 05:59:26 DEBUG "resultType": "vector", 05:59:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:59:26 DEBUG ----- output ----- 05:59:26 DEBUG { 05:59:26 DEBUG "status": "success", 05:59:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:59:26 DEBUG "result": [ 05:59:26 DEBUG { 05:59:26 DEBUG "metric": { 05:59:26 DEBUG "data": { 05:59:26 DEBUG "resultType": "vector", 05:59:26 DEBUG ----- output ----- 05:59:26 DEBUG "data": { 05:59:26 DEBUG "resultType": "vector", 05:59:26 DEBUG "result": [ 05:59:26 DEBUG "result": [ 05:59:26 DEBUG { 05:59:26 DEBUG { 05:59:26 DEBUG { 05:59:26 DEBUG "metric": { 05:59:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:59:26 DEBUG "metric": { 05:59:26 DEBUG "status": "success", 05:59:26 DEBUG "status": "success", 05:59:26 DEBUG "data": { 05:59:26 DEBUG "instance": "10.142.0.25:9100" 05:59:26 DEBUG }, 05:59:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:59:26 DEBUG "data": { 05:59:26 INFO [http_cmd]: http status code OK 05:59:26 DEBUG "resultType": "vector", 05:59:26 DEBUG }, 05:59:26 DEBUG "value": [ 05:59:26 DEBUG 05:59:26 DEBUG "resultType": "vector", 05:59:26 DEBUG --- status code --- 05:59:26 DEBUG "result": [ 05:59:26 DEBUG "value": [ 05:59:26 DEBUG 05:59:26 DEBUG }, 05:59:26 DEBUG "result": [ 05:59:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:26 DEBUG { 05:59:26 DEBUG "metric": { 05:59:26 DEBUG 05:59:26 DEBUG "value": [ 05:59:26 DEBUG 05:59:26 DEBUG 05:59:26 DEBUG 05:59:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:59:26 DEBUG [...] 05:59:26 DEBUG { 05:59:26 DEBUG --- http response --- 05:59:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:59:26 DEBUG ----- output ----- 05:59:26 DEBUG }, 05:59:26 DEBUG "value": [ 05:59:26 DEBUG 05:59:26 DEBUG 05:59:26 DEBUG 05:59:26 DEBUG 05:59:26 DEBUG 05:59:26 DEBUG { 05:59:26 DEBUG "status": "success", 05:59:26 DEBUG "data": { 05:59:26 DEBUG 05:59:26 DEBUG 05:59:26 DEBUG [...] 05:59:26 DEBUG "resultType": "vector", 05:59:26 DEBUG "result": [ 05:59:26 DEBUG { 05:59:26 DEBUG 05:59:26 DEBUG 05:59:26 DEBUG 05:59:26 DEBUG 05:59:26 DEBUG "metric": { 05:59:26 DEBUG "pod": "ds-idrepo-0" 05:59:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:59:26 DEBUG }, 05:59:26 DEBUG "instance": "10.142.0.77:9100" 05:59:26 DEBUG }, 05:59:26 DEBUG }, 05:59:26 DEBUG "value": [ 05:59:26 DEBUG "value": [ 05:59:26 DEBUG 1712725166, 05:59:26 DEBUG "value": [ 05:59:26 DEBUG 1712725166, 05:59:26 DEBUG 05:59:26 DEBUG 05:59:26 DEBUG "259107" 05:59:26 DEBUG ] 05:59:26 DEBUG "338358705152" 05:59:26 DEBUG ] 05:59:26 DEBUG } 05:59:26 DEBUG ] 05:59:26 DEBUG [...] 05:59:26 DEBUG 05:59:26 DEBUG 05:59:26 DEBUG 05:59:26 DEBUG "pod": "am-654c4fcbcf-487cv" 05:59:26 DEBUG }, 05:59:26 DEBUG "value": [ 05:59:26 DEBUG "metric": { 05:59:26 DEBUG 1712725166, 05:59:26 DEBUG "1438.0666666666666" 05:59:26 DEBUG ] 05:59:26 DEBUG } 05:59:26 DEBUG ] 05:59:26 DEBUG } 05:59:26 DEBUG } 05:59:26 DEBUG "pod": "ds-idrepo-0" 05:59:26 DEBUG 05:59:26 DEBUG [...] 05:59:26 DEBUG 05:59:26 DEBUG 05:59:26 DEBUG 05:59:26 DEBUG "pod": "ds-cts-2" 05:59:26 DEBUG }, 05:59:26 DEBUG "value": [ 05:59:26 DEBUG 1712725166, 05:59:26 DEBUG "29.933333333333334" 05:59:26 DEBUG ] 05:59:26 DEBUG } 05:59:26 DEBUG ] 05:59:26 DEBUG } 05:59:26 DEBUG } 05:59:26 DEBUG }, 05:59:26 DEBUG 05:59:26 DEBUG [...] 05:59:26 DEBUG 05:59:26 DEBUG "value": [ 05:59:26 DEBUG 05:59:26 DEBUG 05:59:26 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:59:26 DEBUG }, 05:59:26 DEBUG "value": [ 05:59:26 DEBUG 1712725166, 05:59:26 DEBUG "0" 05:59:26 DEBUG ] 05:59:26 DEBUG } 05:59:26 DEBUG 05:59:26 DEBUG ] 05:59:26 DEBUG } 05:59:26 DEBUG } 05:59:26 DEBUG 05:59:26 DEBUG } 05:59:26 DEBUG ] 05:59:26 DEBUG 05:59:26 DEBUG } 05:59:26 DEBUG } 05:59:26 DEBUG [...] 05:59:26 DEBUG } 05:59:26 DEBUG 05:59:26 DEBUG } 05:59:26 DEBUG 05:59:26 DEBUG 05:59:26 DEBUG "pod": "ds-cts-2" 05:59:26 DEBUG }, 05:59:26 DEBUG "value": [ 05:59:26 DEBUG 1712725166, 05:59:26 DEBUG "0" 05:59:26 DEBUG ] 05:59:26 DEBUG } 05:59:26 DEBUG ] 05:59:26 DEBUG } 05:59:26 DEBUG } 05:59:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725167" 05:59:27 INFO [http_cmd]: http status code OK 05:59:27 DEBUG --- status code --- 05:59:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:27 DEBUG --- http response --- 05:59:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:59:27 DEBUG ----- output ----- 05:59:27 DEBUG { 05:59:27 DEBUG "status": "success", 05:59:27 DEBUG "data": { 05:59:27 DEBUG "resultType": "vector", 05:59:27 DEBUG "result": [ 05:59:27 DEBUG { 05:59:27 DEBUG "metric": { 05:59:27 DEBUG "pod": "ds-idrepo-0" 05:59:27 DEBUG }, 05:59:27 DEBUG "value": [ 05:59:27 DEBUG 05:59:27 DEBUG 05:59:27 DEBUG 05:59:27 DEBUG [...] 05:59:27 DEBUG 05:59:27 DEBUG 05:59:27 DEBUG 05:59:27 DEBUG "pod": "ds-idrepo-1" 05:59:27 DEBUG }, 05:59:27 DEBUG "value": [ 05:59:27 DEBUG 1712725167, 05:59:27 DEBUG "0" 05:59:27 DEBUG ] 05:59:27 DEBUG } 05:59:27 DEBUG ] 05:59:27 DEBUG } 05:59:27 DEBUG } 05:59:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725167" 05:59:27 INFO [http_cmd]: http status code OK 05:59:27 DEBUG --- status code --- 05:59:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:27 DEBUG --- http response --- 05:59:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:59:27 DEBUG ----- output ----- 05:59:27 DEBUG { 05:59:27 DEBUG "status": "success", 05:59:27 DEBUG "data": { 05:59:27 DEBUG "resultType": "vector", 05:59:27 DEBUG "result": [ 05:59:27 DEBUG { 05:59:27 DEBUG "metric": { 05:59:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:59:27 DEBUG }, 05:59:27 DEBUG "value": [ 05:59:27 DEBUG 05:59:27 DEBUG 05:59:27 DEBUG 05:59:27 DEBUG [...] 05:59:27 DEBUG 05:59:27 DEBUG 05:59:27 DEBUG 05:59:27 DEBUG "pod": "am-654c4fcbcf-487cv" 05:59:27 DEBUG }, 05:59:27 DEBUG "value": [ 05:59:27 DEBUG 1712725167, 05:59:27 DEBUG "1434.9" 05:59:27 DEBUG ] 05:59:27 DEBUG } 05:59:27 DEBUG ] 05:59:27 DEBUG } 05:59:27 DEBUG } 05:59:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725167" 05:59:27 INFO [http_cmd]: http status code OK 05:59:27 DEBUG --- status code --- 05:59:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:27 DEBUG --- http response --- 05:59:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:59:27 DEBUG ----- output ----- 05:59:27 DEBUG { 05:59:27 DEBUG "status": "success", 05:59:27 DEBUG "data": { 05:59:27 DEBUG "resultType": "vector", 05:59:27 DEBUG "result": [ 05:59:27 DEBUG { 05:59:27 DEBUG "metric": { 05:59:27 DEBUG "instance": "10.142.0.25:9100" 05:59:27 DEBUG }, 05:59:27 DEBUG "value": [ 05:59:27 DEBUG 05:59:27 DEBUG 05:59:27 DEBUG 05:59:27 DEBUG [...] 05:59:27 DEBUG 05:59:27 DEBUG 05:59:27 DEBUG 05:59:27 DEBUG "instance": "10.142.0.77:9100" 05:59:27 DEBUG }, 05:59:27 DEBUG "value": [ 05:59:27 DEBUG 1712725167, 05:59:27 DEBUG "0.14440000000000217" 05:59:27 DEBUG ] 05:59:27 DEBUG } 05:59:27 DEBUG ] 05:59:27 DEBUG } 05:59:27 DEBUG } 05:59:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725168" 05:59:28 INFO [http_cmd]: http status code OK 05:59:28 DEBUG --- status code --- 05:59:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:28 DEBUG --- http response --- 05:59:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:59:28 DEBUG ----- output ----- 05:59:28 DEBUG { 05:59:28 DEBUG "status": "success", 05:59:28 DEBUG "data": { 05:59:28 DEBUG "resultType": "vector", 05:59:28 DEBUG "result": [ 05:59:28 DEBUG { 05:59:28 DEBUG "metric": { 05:59:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:59:28 DEBUG }, 05:59:28 DEBUG "value": [ 05:59:28 DEBUG 05:59:28 DEBUG 05:59:28 DEBUG 05:59:28 DEBUG [...] 05:59:28 DEBUG 05:59:28 DEBUG 05:59:28 DEBUG 05:59:28 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:59:28 DEBUG }, 05:59:28 DEBUG "value": [ 05:59:28 DEBUG 1712725168, 05:59:28 DEBUG "6086.275900277007" 05:59:28 DEBUG ] 05:59:28 DEBUG } 05:59:28 DEBUG ] 05:59:28 DEBUG } 05:59:28 DEBUG } 05:59:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712725168" 05:59:28 INFO [http_cmd]: http status code OK 05:59:28 DEBUG --- status code --- 05:59:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:28 DEBUG --- http response --- 05:59:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:59:28 DEBUG ----- output ----- 05:59:28 DEBUG { 05:59:28 DEBUG "status": "success", 05:59:28 DEBUG "data": { 05:59:28 DEBUG "resultType": "vector", 05:59:28 DEBUG "result": [ 05:59:28 DEBUG { 05:59:28 DEBUG "metric": { 05:59:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:59:28 DEBUG }, 05:59:28 DEBUG "value": [ 05:59:28 DEBUG 05:59:28 DEBUG 05:59:28 DEBUG 05:59:28 DEBUG [...] 05:59:28 DEBUG 05:59:28 DEBUG 05:59:28 DEBUG 05:59:28 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:59:28 DEBUG }, 05:59:28 DEBUG "value": [ 05:59:28 DEBUG 1712725168, 05:59:28 DEBUG "4.129936801748352" 05:59:28 DEBUG ] 05:59:28 DEBUG } 05:59:28 DEBUG ] 05:59:28 DEBUG } 05:59:28 DEBUG } 05:59:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712725168" 05:59:28 INFO [http_cmd]: http status code OK 05:59:28 DEBUG --- status code --- 05:59:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:28 DEBUG --- http response --- 05:59:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:59:28 DEBUG ----- output ----- 05:59:28 DEBUG { 05:59:28 DEBUG "status": "success", 05:59:28 DEBUG "data": { 05:59:28 DEBUG "resultType": "vector", 05:59:28 DEBUG "result": [ 05:59:28 DEBUG { 05:59:28 DEBUG "metric": { 05:59:28 DEBUG "pod": "ds-idrepo-0" 05:59:28 DEBUG }, 05:59:28 DEBUG "value": [ 05:59:28 DEBUG 05:59:28 DEBUG 05:59:28 DEBUG 05:59:28 DEBUG [...] 05:59:28 DEBUG 05:59:28 DEBUG 05:59:28 DEBUG 05:59:28 DEBUG "pod": "ds-cts-2" 05:59:28 DEBUG }, 05:59:28 DEBUG "value": [ 05:59:28 DEBUG 1712725168, 05:59:28 DEBUG "0" 05:59:28 DEBUG ] 05:59:28 DEBUG } 05:59:28 DEBUG ] 05:59:28 DEBUG } 05:59:28 DEBUG } 05:59:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725168" 05:59:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725168" 05:59:28 INFO [http_cmd]: http status code OK 05:59:28 DEBUG --- status code --- 05:59:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:28 DEBUG --- http response --- 05:59:28 INFO [http_cmd]: http status code OK 05:59:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 05:59:28 DEBUG --- status code --- 05:59:28 DEBUG ----- output ----- 05:59:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:28 DEBUG { 05:59:28 DEBUG --- http response --- 05:59:28 DEBUG "status": "success", 05:59:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:59:28 DEBUG "data": { 05:59:28 DEBUG "resultType": "vector", 05:59:28 DEBUG ----- output ----- 05:59:28 DEBUG "result": [ 05:59:28 DEBUG { 05:59:28 DEBUG { 05:59:28 DEBUG "status": "success", 05:59:28 DEBUG "metric": { 05:59:28 DEBUG "data": { 05:59:28 DEBUG "pod": "ds-idrepo-2" 05:59:28 DEBUG "resultType": "vector", 05:59:28 DEBUG }, 05:59:28 DEBUG "result": [ 05:59:28 DEBUG "value": [ 05:59:28 DEBUG { 05:59:28 DEBUG 05:59:28 DEBUG "metric": { 05:59:28 DEBUG 05:59:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:59:28 DEBUG 05:59:28 DEBUG }, 05:59:28 DEBUG [...] 05:59:28 DEBUG "value": [ 05:59:28 DEBUG 05:59:28 DEBUG 05:59:28 DEBUG 05:59:28 DEBUG 05:59:28 DEBUG 05:59:28 DEBUG 05:59:28 DEBUG "pod": "ds-cts-2" 05:59:28 DEBUG [...] 05:59:28 DEBUG }, 05:59:28 DEBUG 05:59:28 DEBUG "value": [ 05:59:28 DEBUG 1712725168, 05:59:28 DEBUG "0.00006666666666666667" 05:59:28 DEBUG 05:59:28 DEBUG ] 05:59:28 DEBUG 05:59:28 DEBUG } 05:59:28 DEBUG "pod": "ds-idrepo-2" 05:59:28 DEBUG ] 05:59:28 DEBUG }, 05:59:28 DEBUG } 05:59:28 DEBUG } 05:59:28 DEBUG "value": [ 05:59:28 DEBUG 1712725168, 05:59:28 DEBUG "0.42383198146720524" 05:59:28 DEBUG ] 05:59:28 DEBUG } 05:59:28 DEBUG ] 05:59:28 DEBUG } 05:59:28 DEBUG } 05:59:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725168" 05:59:28 INFO [http_cmd]: http status code OK 05:59:28 DEBUG --- status code --- 05:59:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:28 DEBUG --- http response --- 05:59:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:59:28 DEBUG ----- output ----- 05:59:28 DEBUG { 05:59:28 DEBUG "status": "success", 05:59:28 DEBUG "data": { 05:59:28 DEBUG "resultType": "vector", 05:59:28 DEBUG "result": [ 05:59:28 DEBUG { 05:59:28 DEBUG "metric": { 05:59:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:59:28 DEBUG }, 05:59:28 DEBUG "value": [ 05:59:28 DEBUG 05:59:28 DEBUG 05:59:28 DEBUG 05:59:28 DEBUG [...] 05:59:28 DEBUG 05:59:28 DEBUG 05:59:28 DEBUG 05:59:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:59:28 DEBUG }, 05:59:28 DEBUG "value": [ 05:59:28 DEBUG 1712725168, 05:59:28 DEBUG "1434.5" 05:59:28 DEBUG ] 05:59:28 DEBUG } 05:59:28 DEBUG ] 05:59:28 DEBUG } 05:59:28 DEBUG } 05:59:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712725168" 05:59:28 INFO [http_cmd]: http status code OK 05:59:28 DEBUG --- status code --- 05:59:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:28 DEBUG --- http response --- 05:59:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:59:28 DEBUG ----- output ----- 05:59:28 DEBUG { 05:59:28 DEBUG "status": "success", 05:59:28 DEBUG "data": { 05:59:28 DEBUG "resultType": "vector", 05:59:28 DEBUG "result": [ 05:59:28 DEBUG { 05:59:28 DEBUG "metric": { 05:59:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:59:28 DEBUG }, 05:59:28 DEBUG "value": [ 05:59:28 DEBUG 05:59:28 DEBUG 05:59:28 DEBUG 05:59:28 DEBUG [...] 05:59:28 DEBUG 05:59:28 DEBUG 05:59:28 DEBUG 05:59:28 DEBUG "pod": "am-654c4fcbcf-487cv" 05:59:28 DEBUG }, 05:59:28 DEBUG "value": [ 05:59:28 DEBUG 1712725168, 05:59:28 DEBUG "0.0465" 05:59:28 DEBUG ] 05:59:28 DEBUG } 05:59:28 DEBUG ] 05:59:28 DEBUG } 05:59:28 DEBUG } 05:59:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725169" 05:59:29 INFO [http_cmd]: http status code OK 05:59:29 DEBUG --- status code --- 05:59:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:29 DEBUG --- http response --- 05:59:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:59:29 DEBUG ----- output ----- 05:59:29 DEBUG { 05:59:29 DEBUG "status": "success", 05:59:29 DEBUG "data": { 05:59:29 DEBUG "resultType": "vector", 05:59:29 DEBUG "result": [ 05:59:29 DEBUG { 05:59:29 DEBUG "metric": { 05:59:29 DEBUG "pod": "ds-idrepo-0" 05:59:29 DEBUG }, 05:59:29 DEBUG "value": [ 05:59:29 DEBUG 05:59:29 DEBUG 05:59:29 DEBUG 05:59:29 DEBUG [...] 05:59:29 DEBUG 05:59:29 DEBUG 05:59:29 DEBUG 05:59:29 DEBUG "pod": "ds-idrepo-1" 05:59:29 DEBUG }, 05:59:29 DEBUG "value": [ 05:59:29 DEBUG 1712725169, 05:59:29 DEBUG "0" 05:59:29 DEBUG ] 05:59:29 DEBUG } 05:59:29 DEBUG ] 05:59:29 DEBUG } 05:59:29 DEBUG } 05:59:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725169" 05:59:29 INFO [http_cmd]: http status code OK 05:59:29 DEBUG --- status code --- 05:59:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:29 DEBUG --- http response --- 05:59:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:59:29 DEBUG ----- output ----- 05:59:29 DEBUG { 05:59:29 DEBUG "status": "success", 05:59:29 DEBUG "data": { 05:59:29 DEBUG "resultType": "vector", 05:59:29 DEBUG "result": [ 05:59:29 DEBUG { 05:59:29 DEBUG "metric": { 05:59:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 05:59:29 DEBUG }, 05:59:29 DEBUG "value": [ 05:59:29 DEBUG 05:59:29 DEBUG 05:59:29 DEBUG 05:59:29 DEBUG [...] 05:59:29 DEBUG 05:59:29 DEBUG 05:59:29 DEBUG 05:59:29 DEBUG "pod": "am-654c4fcbcf-487cv" 05:59:29 DEBUG }, 05:59:29 DEBUG "value": [ 05:59:29 DEBUG 1712725169, 05:59:29 DEBUG "1461.4666666666667" 05:59:29 DEBUG ] 05:59:29 DEBUG } 05:59:29 DEBUG ] 05:59:29 DEBUG } 05:59:29 DEBUG } 05:59:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725169" 05:59:29 INFO [http_cmd]: http status code OK 05:59:29 DEBUG --- status code --- 05:59:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:29 DEBUG --- http response --- 05:59:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:59:29 DEBUG ----- output ----- 05:59:29 DEBUG { 05:59:29 DEBUG "status": "success", 05:59:29 DEBUG "data": { 05:59:29 DEBUG "resultType": "vector", 05:59:29 DEBUG "result": [ 05:59:29 DEBUG { 05:59:29 DEBUG "metric": { 05:59:29 DEBUG "pod": "ds-idrepo-0" 05:59:29 DEBUG }, 05:59:29 DEBUG "value": [ 05:59:29 DEBUG 05:59:29 DEBUG 05:59:29 DEBUG 05:59:29 DEBUG [...] 05:59:29 DEBUG 05:59:29 DEBUG 05:59:29 DEBUG 05:59:29 DEBUG "pod": "ds-idrepo-1" 05:59:29 DEBUG }, 05:59:29 DEBUG "value": [ 05:59:29 DEBUG 1712725169, 05:59:29 DEBUG "0" 05:59:29 DEBUG ] 05:59:29 DEBUG } 05:59:29 DEBUG ] 05:59:29 DEBUG } 05:59:29 DEBUG } 05:59:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712725169" 05:59:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725169" 05:59:29 INFO [http_cmd]: http status code OK 05:59:29 DEBUG --- status code --- 05:59:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:29 DEBUG --- http response --- 05:59:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:59:29 DEBUG ----- output ----- 05:59:29 DEBUG { 05:59:29 DEBUG "status": "success", 05:59:29 DEBUG "data": { 05:59:29 DEBUG "resultType": "vector", 05:59:29 DEBUG "result": [ 05:59:29 DEBUG { 05:59:29 DEBUG "metric": { 05:59:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 05:59:29 INFO [http_cmd]: http status code OK 05:59:29 DEBUG }, 05:59:29 DEBUG --- status code --- 05:59:29 DEBUG "value": [ 05:59:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:29 DEBUG 05:59:29 DEBUG --- http response --- 05:59:29 DEBUG 05:59:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:59:29 DEBUG 05:59:29 DEBUG ----- output ----- 05:59:29 DEBUG [...] 05:59:29 DEBUG { 05:59:29 DEBUG 05:59:29 DEBUG "status": "success", 05:59:29 DEBUG 05:59:29 DEBUG "data": { 05:59:29 DEBUG 05:59:29 DEBUG "resultType": "vector", 05:59:29 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 05:59:29 DEBUG "result": [ 05:59:29 DEBUG }, 05:59:29 DEBUG { 05:59:29 DEBUG "value": [ 05:59:29 DEBUG "metric": { 05:59:29 DEBUG 1712725169, 05:59:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 05:59:29 DEBUG }, 05:59:29 DEBUG "0" 05:59:29 DEBUG "value": [ 05:59:29 DEBUG ] 05:59:29 DEBUG 05:59:29 DEBUG } 05:59:29 DEBUG 05:59:29 DEBUG ] 05:59:29 DEBUG 05:59:29 DEBUG [...] 05:59:29 DEBUG } 05:59:29 DEBUG 05:59:29 DEBUG } 05:59:29 DEBUG 05:59:29 DEBUG 05:59:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 05:59:29 DEBUG }, 05:59:29 DEBUG "value": [ 05:59:29 DEBUG 1712725169, 05:59:29 DEBUG "2487386112" 05:59:29 DEBUG ] 05:59:29 DEBUG } 05:59:29 DEBUG ] 05:59:29 DEBUG } 05:59:29 DEBUG } 05:59:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725169" 05:59:29 INFO [http_cmd]: http status code OK 05:59:29 DEBUG --- status code --- 05:59:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:29 DEBUG --- http response --- 05:59:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:59:29 DEBUG ----- output ----- 05:59:29 DEBUG { 05:59:29 DEBUG "status": "success", 05:59:29 DEBUG "data": { 05:59:29 DEBUG "resultType": "vector", 05:59:29 DEBUG "result": [ 05:59:29 DEBUG { 05:59:29 DEBUG "metric": { 05:59:29 DEBUG "instance": "10.142.0.25:9100" 05:59:29 DEBUG }, 05:59:29 DEBUG "value": [ 05:59:29 DEBUG 05:59:29 DEBUG 05:59:29 DEBUG 05:59:29 DEBUG [...] 05:59:29 DEBUG 05:59:29 DEBUG 05:59:29 DEBUG 05:59:29 DEBUG "instance": "10.142.0.77:9100" 05:59:29 DEBUG }, 05:59:29 DEBUG "value": [ 05:59:29 DEBUG 1712725169, 05:59:29 DEBUG "42713715.03333333" 05:59:29 DEBUG ] 05:59:29 DEBUG } 05:59:29 DEBUG ] 05:59:29 DEBUG } 05:59:29 DEBUG } 05:59:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725169" 05:59:29 INFO [http_cmd]: http status code OK 05:59:29 DEBUG --- status code --- 05:59:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:29 DEBUG --- http response --- 05:59:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:59:29 DEBUG ----- output ----- 05:59:29 DEBUG { 05:59:29 DEBUG "status": "success", 05:59:29 DEBUG "data": { 05:59:29 DEBUG "resultType": "vector", 05:59:29 DEBUG "result": [ 05:59:29 DEBUG { 05:59:29 DEBUG "metric": { 05:59:29 DEBUG "instance": "10.142.0.25:9100" 05:59:29 DEBUG }, 05:59:29 DEBUG "value": [ 05:59:29 DEBUG 05:59:29 DEBUG 05:59:29 DEBUG 05:59:29 DEBUG [...] 05:59:29 DEBUG 05:59:29 DEBUG 05:59:29 DEBUG 05:59:29 DEBUG "instance": "10.142.0.77:9100" 05:59:29 DEBUG }, 05:59:29 DEBUG "value": [ 05:59:29 DEBUG 1712725169, 05:59:29 DEBUG "0.042333333333332314" 05:59:29 DEBUG ] 05:59:29 DEBUG } 05:59:29 DEBUG ] 05:59:29 DEBUG } 05:59:29 DEBUG } 05:59:46 INFO 05:59:46 INFO [loop_until]: kubectl --namespace=xlou top pods 05:59:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:59:46 INFO [loop_until]: OK (rc = 0) 05:59:46 DEBUG --- stdout --- 05:59:46 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10663m 8961Mi am-654c4fcbcf-9scrj 9905m 8438Mi am-654c4fcbcf-v5qb5 8187m 8428Mi ds-cts-0 8467m 13378Mi ds-cts-1 8510m 12931Mi ds-cts-2 8279m 13336Mi ds-idrepo-0 10600m 5622Mi ds-idrepo-1 375m 5671Mi ds-idrepo-2 417m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 2m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4102m 2231Mi 05:59:46 DEBUG --- stderr --- 05:59:46 DEBUG 05:59:54 INFO 05:59:54 INFO [loop_until]: kubectl --namespace=xlou top node 05:59:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:59:54 INFO [loop_until]: OK (rc = 0) 05:59:54 DEBUG --- stdout --- 05:59:54 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 92m 0% 2306Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9043m 56% 9304Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 80m 0% 3259Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13226m 83% 6569Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10953m 68% 9321Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11292m 71% 9787Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 79m 0% 851Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10840m 68% 6294Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8564m 53% 14040Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 7989m 50% 14028Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 503m 3% 6381Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8527m 53% 13532Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 423m 2% 6355Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4247m 26% 5884Mi 10% 05:59:54 DEBUG --- stderr --- 05:59:54 DEBUG 06:00:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712725226" 06:00:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725226" 06:00:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725226" 06:00:26 INFO [http_cmd]: http status code OK 06:00:26 DEBUG --- status code --- 06:00:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:26 DEBUG --- http response --- 06:00:26 ERROR [http_cmd]: ERROR 06:00:26 ERROR --- status code --- 06:00:26 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:00:26 ERROR --- http response --- 06:00:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:00:26 INFO [http_cmd]: http status code OK 06:00:26 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:00:26 DEBUG ----- output ----- 06:00:26 DEBUG --- status code --- 06:00:26 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:00:26 DEBUG { 06:00:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:26 DEBUG "status": "success", 06:00:26 DEBUG --- http response --- 06:00:26 DEBUG "data": { 06:00:26 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:00:26 DEBUG "resultType": "vector", 06:00:26 DEBUG "result": [ 06:00:26 DEBUG { 06:00:26 DEBUG "metric": { 06:00:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:00:26 DEBUG }, 06:00:26 DEBUG "value": [ 06:00:26 DEBUG 06:00:26 DEBUG 06:00:26 DEBUG 06:00:26 DEBUG [...] 06:00:26 DEBUG 06:00:26 DEBUG 06:00:26 DEBUG 06:00:26 DEBUG "pod": "am-654c4fcbcf-487cv" 06:00:26 DEBUG }, 06:00:26 DEBUG "value": [ 06:00:26 DEBUG 1712725226, 06:00:26 DEBUG "0" 06:00:26 DEBUG ] 06:00:26 DEBUG } 06:00:26 DEBUG ] 06:00:26 DEBUG } 06:00:26 DEBUG } 06:00:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725226" 06:00:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725226" 06:00:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725226" 06:00:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725226" 06:00:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725226" 06:00:26 INFO [http_cmd]: http status code OK 06:00:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725226" 06:00:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725226" 06:00:26 DEBUG --- status code --- 06:00:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:26 DEBUG --- http response --- 06:00:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:00:26 DEBUG ----- output ----- 06:00:26 DEBUG { 06:00:26 DEBUG "status": "success", 06:00:26 DEBUG "data": { 06:00:26 DEBUG "resultType": "vector", 06:00:26 DEBUG "result": [ 06:00:26 DEBUG { 06:00:26 DEBUG "metric": { 06:00:26 DEBUG "instance": "10.142.0.25:9100" 06:00:26 DEBUG }, 06:00:26 DEBUG "value": [ 06:00:26 DEBUG 06:00:26 DEBUG 06:00:26 DEBUG 06:00:26 DEBUG [...] 06:00:26 INFO [http_cmd]: http status code OK 06:00:26 DEBUG --- status code --- 06:00:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:26 DEBUG --- http response --- 06:00:26 DEBUG 06:00:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:00:26 DEBUG ----- output ----- 06:00:26 DEBUG 06:00:26 DEBUG { 06:00:26 DEBUG 06:00:26 DEBUG "status": "success", 06:00:26 DEBUG "instance": "10.142.0.77:9100" 06:00:26 DEBUG "data": { 06:00:26 DEBUG }, 06:00:26 DEBUG "resultType": "vector", 06:00:26 DEBUG "value": [ 06:00:26 DEBUG 1712725226, 06:00:26 DEBUG "42397234.66871084" 06:00:26 DEBUG ] 06:00:26 DEBUG } 06:00:26 DEBUG ] 06:00:26 DEBUG } 06:00:26 DEBUG } 06:00:26 DEBUG "result": [ 06:00:26 DEBUG { 06:00:26 DEBUG "metric": { 06:00:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:00:26 INFO [http_cmd]: http status code OK 06:00:26 DEBUG }, 06:00:26 DEBUG --- status code --- 06:00:26 DEBUG "value": [ 06:00:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:26 DEBUG 06:00:26 DEBUG --- http response --- 06:00:26 DEBUG 06:00:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:00:26 DEBUG ----- output ----- 06:00:26 DEBUG { 06:00:26 DEBUG 06:00:26 DEBUG "status": "success", 06:00:26 DEBUG [...] 06:00:26 DEBUG "data": { 06:00:26 DEBUG 06:00:26 DEBUG "resultType": "vector", 06:00:26 DEBUG 06:00:26 DEBUG "result": [ 06:00:26 DEBUG 06:00:26 DEBUG { 06:00:26 DEBUG "pod": "am-654c4fcbcf-487cv" 06:00:26 DEBUG "metric": { 06:00:26 DEBUG }, 06:00:26 DEBUG "value": [ 06:00:26 DEBUG "pod": "ds-idrepo-0" 06:00:26 DEBUG 1712725226, 06:00:26 DEBUG }, 06:00:26 DEBUG "1461.3666666666666" 06:00:26 DEBUG "value": [ 06:00:26 DEBUG ] 06:00:26 DEBUG 06:00:26 DEBUG } 06:00:26 DEBUG 06:00:26 DEBUG ] 06:00:26 DEBUG 06:00:26 DEBUG } 06:00:26 DEBUG [...] 06:00:26 DEBUG } 06:00:26 DEBUG 06:00:26 DEBUG 06:00:26 DEBUG 06:00:26 DEBUG "pod": "ds-cts-2" 06:00:26 INFO [http_cmd]: http status code OK 06:00:26 DEBUG --- status code --- 06:00:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:26 DEBUG --- http response --- 06:00:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:00:26 DEBUG ----- output ----- 06:00:26 DEBUG { 06:00:26 DEBUG "status": "success", 06:00:26 DEBUG "data": { 06:00:26 DEBUG "resultType": "vector", 06:00:26 DEBUG "result": [ 06:00:26 DEBUG { 06:00:26 DEBUG "metric": { 06:00:26 DEBUG "pod": "ds-idrepo-0" 06:00:26 DEBUG }, 06:00:26 DEBUG "value": [ 06:00:26 DEBUG 06:00:26 DEBUG 06:00:26 DEBUG 06:00:26 DEBUG [...] 06:00:26 DEBUG 06:00:26 DEBUG }, 06:00:26 INFO [http_cmd]: http status code OK 06:00:26 INFO [http_cmd]: http status code OK 06:00:26 DEBUG --- status code --- 06:00:26 DEBUG "value": [ 06:00:26 DEBUG --- status code --- 06:00:26 DEBUG 06:00:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:26 DEBUG 1712725226, 06:00:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:26 DEBUG 06:00:26 DEBUG --- http response --- 06:00:26 DEBUG "29.266666666666666" 06:00:26 DEBUG --- http response --- 06:00:26 DEBUG "pod": "ds-cts-2" 06:00:26 DEBUG ] 06:00:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 06:00:26 INFO [http_cmd]: http status code OK 06:00:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:00:26 DEBUG }, 06:00:26 DEBUG } 06:00:26 DEBUG ----- output ----- 06:00:26 DEBUG --- status code --- 06:00:26 DEBUG ----- output ----- 06:00:26 DEBUG "value": [ 06:00:26 DEBUG ] 06:00:26 DEBUG { 06:00:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:26 DEBUG { 06:00:26 DEBUG 1712725226, 06:00:26 DEBUG } 06:00:26 DEBUG "status": "success", 06:00:26 DEBUG --- http response --- 06:00:26 DEBUG "status": "success", 06:00:26 DEBUG "0" 06:00:26 DEBUG } 06:00:26 DEBUG "data": { 06:00:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:00:26 DEBUG "data": { 06:00:26 DEBUG "resultType": "vector", 06:00:26 DEBUG "result": [ 06:00:26 DEBUG ----- output ----- 06:00:26 DEBUG "resultType": "vector", 06:00:26 DEBUG "result": [ 06:00:26 DEBUG { 06:00:26 DEBUG { 06:00:26 DEBUG "metric": { 06:00:26 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:00:26 DEBUG ] 06:00:26 DEBUG "status": "success", 06:00:26 DEBUG { 06:00:26 DEBUG }, 06:00:26 DEBUG } 06:00:26 DEBUG "data": { 06:00:26 DEBUG "metric": { 06:00:26 DEBUG "value": [ 06:00:26 DEBUG ] 06:00:26 DEBUG "resultType": "vector", 06:00:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:00:26 DEBUG 06:00:26 DEBUG } 06:00:26 DEBUG "result": [ 06:00:26 DEBUG }, 06:00:26 DEBUG 06:00:26 DEBUG } 06:00:26 DEBUG { 06:00:26 DEBUG "metric": { 06:00:26 DEBUG "instance": "10.142.0.25:9100" 06:00:26 DEBUG "value": [ 06:00:26 DEBUG }, 06:00:26 DEBUG 06:00:26 DEBUG 06:00:26 DEBUG "value": [ 06:00:26 DEBUG [...] 06:00:26 DEBUG 06:00:26 DEBUG 06:00:26 DEBUG 06:00:26 DEBUG 06:00:26 DEBUG 06:00:26 DEBUG 06:00:26 DEBUG [...] 06:00:26 DEBUG 06:00:26 DEBUG 06:00:26 DEBUG [...] 06:00:26 DEBUG 06:00:26 DEBUG "instance": "10.142.0.77:9100" 06:00:26 DEBUG 06:00:26 DEBUG }, 06:00:26 DEBUG "value": [ 06:00:26 DEBUG 06:00:26 DEBUG 1712725226, 06:00:26 DEBUG "340247503872" 06:00:26 DEBUG ] 06:00:26 DEBUG 06:00:26 DEBUG } 06:00:26 DEBUG ] 06:00:26 DEBUG } 06:00:26 DEBUG } 06:00:26 DEBUG 06:00:26 DEBUG "pod": "am-654c4fcbcf-487cv" 06:00:26 DEBUG 06:00:26 DEBUG }, 06:00:26 DEBUG "value": [ 06:00:26 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:00:26 DEBUG 1712725226, 06:00:26 DEBUG }, 06:00:26 DEBUG "value": [ 06:00:26 DEBUG 1712725226, 06:00:26 DEBUG "0" 06:00:26 DEBUG ] 06:00:26 DEBUG } 06:00:26 DEBUG "258349" 06:00:26 DEBUG ] 06:00:26 DEBUG ] 06:00:26 DEBUG } 06:00:26 DEBUG } 06:00:26 DEBUG } 06:00:26 DEBUG ] 06:00:26 DEBUG } 06:00:26 DEBUG } 06:00:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725227" 06:00:27 INFO [http_cmd]: http status code OK 06:00:27 DEBUG --- status code --- 06:00:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:27 DEBUG --- http response --- 06:00:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:00:27 DEBUG ----- output ----- 06:00:27 DEBUG { 06:00:27 DEBUG "status": "success", 06:00:27 DEBUG "data": { 06:00:27 DEBUG "resultType": "vector", 06:00:27 DEBUG "result": [ 06:00:27 DEBUG { 06:00:27 DEBUG "metric": { 06:00:27 DEBUG "pod": "ds-idrepo-0" 06:00:27 DEBUG }, 06:00:27 DEBUG "value": [ 06:00:27 DEBUG 06:00:27 DEBUG 06:00:27 DEBUG 06:00:27 DEBUG [...] 06:00:27 DEBUG 06:00:27 DEBUG 06:00:27 DEBUG 06:00:27 DEBUG "pod": "ds-idrepo-1" 06:00:27 DEBUG }, 06:00:27 DEBUG "value": [ 06:00:27 DEBUG 1712725227, 06:00:27 DEBUG "0" 06:00:27 DEBUG ] 06:00:27 DEBUG } 06:00:27 DEBUG ] 06:00:27 DEBUG } 06:00:27 DEBUG } 06:00:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725227" 06:00:27 INFO [http_cmd]: http status code OK 06:00:27 DEBUG --- status code --- 06:00:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:27 DEBUG --- http response --- 06:00:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:00:27 DEBUG ----- output ----- 06:00:27 DEBUG { 06:00:27 DEBUG "status": "success", 06:00:27 DEBUG "data": { 06:00:27 DEBUG "resultType": "vector", 06:00:27 DEBUG "result": [ 06:00:27 DEBUG { 06:00:27 DEBUG "metric": { 06:00:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:00:27 DEBUG }, 06:00:27 DEBUG "value": [ 06:00:27 DEBUG 06:00:27 DEBUG 06:00:27 DEBUG 06:00:27 DEBUG [...] 06:00:27 DEBUG 06:00:27 DEBUG 06:00:27 DEBUG 06:00:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:00:27 DEBUG }, 06:00:27 DEBUG "value": [ 06:00:27 DEBUG 1712725227, 06:00:27 DEBUG "1463.3" 06:00:27 DEBUG ] 06:00:27 DEBUG } 06:00:27 DEBUG ] 06:00:27 DEBUG } 06:00:27 DEBUG } 06:00:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725227" 06:00:27 INFO [http_cmd]: http status code OK 06:00:27 DEBUG --- status code --- 06:00:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:27 DEBUG --- http response --- 06:00:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:00:27 DEBUG ----- output ----- 06:00:27 DEBUG { 06:00:27 DEBUG "status": "success", 06:00:27 DEBUG "data": { 06:00:27 DEBUG "resultType": "vector", 06:00:27 DEBUG "result": [ 06:00:27 DEBUG { 06:00:27 DEBUG "metric": { 06:00:27 DEBUG "instance": "10.142.0.25:9100" 06:00:27 DEBUG }, 06:00:27 DEBUG "value": [ 06:00:27 DEBUG 06:00:27 DEBUG 06:00:27 DEBUG 06:00:27 DEBUG [...] 06:00:27 DEBUG 06:00:27 DEBUG 06:00:27 DEBUG 06:00:27 DEBUG "instance": "10.142.0.77:9100" 06:00:27 DEBUG }, 06:00:27 DEBUG "value": [ 06:00:27 DEBUG 1712725227, 06:00:27 DEBUG "0.14754699373416558" 06:00:27 DEBUG ] 06:00:27 DEBUG } 06:00:27 DEBUG ] 06:00:27 DEBUG } 06:00:27 DEBUG } 06:00:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725228" 06:00:28 INFO [http_cmd]: http status code OK 06:00:28 DEBUG --- status code --- 06:00:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:28 DEBUG --- http response --- 06:00:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:00:28 DEBUG ----- output ----- 06:00:28 DEBUG { 06:00:28 DEBUG "status": "success", 06:00:28 DEBUG "data": { 06:00:28 DEBUG "resultType": "vector", 06:00:28 DEBUG "result": [ 06:00:28 DEBUG { 06:00:28 DEBUG "metric": { 06:00:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:00:28 DEBUG }, 06:00:28 DEBUG "value": [ 06:00:28 DEBUG 06:00:28 DEBUG 06:00:28 DEBUG 06:00:28 DEBUG [...] 06:00:28 DEBUG 06:00:28 DEBUG 06:00:28 DEBUG 06:00:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:00:28 DEBUG }, 06:00:28 DEBUG "value": [ 06:00:28 DEBUG 1712725228, 06:00:28 DEBUG "86757.2069456906" 06:00:28 DEBUG ] 06:00:28 DEBUG } 06:00:28 DEBUG ] 06:00:28 DEBUG } 06:00:28 DEBUG } 06:00:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712725228" 06:00:28 INFO [http_cmd]: http status code OK 06:00:28 DEBUG --- status code --- 06:00:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:28 DEBUG --- http response --- 06:00:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:00:28 DEBUG ----- output ----- 06:00:28 DEBUG { 06:00:28 DEBUG "status": "success", 06:00:28 DEBUG "data": { 06:00:28 DEBUG "resultType": "vector", 06:00:28 DEBUG "result": [ 06:00:28 DEBUG { 06:00:28 DEBUG "metric": { 06:00:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:00:28 DEBUG }, 06:00:28 DEBUG "value": [ 06:00:28 DEBUG 06:00:28 DEBUG 06:00:28 DEBUG 06:00:28 DEBUG [...] 06:00:28 DEBUG 06:00:28 DEBUG 06:00:28 DEBUG 06:00:28 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:00:28 DEBUG }, 06:00:28 DEBUG "value": [ 06:00:28 DEBUG 1712725228, 06:00:28 DEBUG "4.194708814930873" 06:00:28 DEBUG ] 06:00:28 DEBUG } 06:00:28 DEBUG ] 06:00:28 DEBUG } 06:00:28 DEBUG } 06:00:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712725228" 06:00:28 INFO [http_cmd]: http status code OK 06:00:28 DEBUG --- status code --- 06:00:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:28 DEBUG --- http response --- 06:00:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:00:28 DEBUG ----- output ----- 06:00:28 DEBUG { 06:00:28 DEBUG "status": "success", 06:00:28 DEBUG "data": { 06:00:28 DEBUG "resultType": "vector", 06:00:28 DEBUG "result": [ 06:00:28 DEBUG { 06:00:28 DEBUG "metric": { 06:00:28 DEBUG "pod": "ds-idrepo-0" 06:00:28 DEBUG }, 06:00:28 DEBUG "value": [ 06:00:28 DEBUG 06:00:28 DEBUG 06:00:28 DEBUG 06:00:28 DEBUG [...] 06:00:28 DEBUG 06:00:28 DEBUG 06:00:28 DEBUG 06:00:28 DEBUG "pod": "ds-cts-2" 06:00:28 DEBUG }, 06:00:28 DEBUG "value": [ 06:00:28 DEBUG 1712725228, 06:00:28 DEBUG "0" 06:00:28 DEBUG ] 06:00:28 DEBUG } 06:00:28 DEBUG ] 06:00:28 DEBUG } 06:00:28 DEBUG } 06:00:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725228" 06:00:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725228" 06:00:28 INFO [http_cmd]: http status code OK 06:00:28 DEBUG --- status code --- 06:00:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:28 INFO [http_cmd]: http status code OK 06:00:28 DEBUG --- http response --- 06:00:28 DEBUG --- status code --- 06:00:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:00:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:28 DEBUG --- http response --- 06:00:28 DEBUG ----- output ----- 06:00:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:00:28 DEBUG { 06:00:28 DEBUG ----- output ----- 06:00:28 DEBUG "status": "success", 06:00:28 DEBUG { 06:00:28 DEBUG "data": { 06:00:28 DEBUG "status": "success", 06:00:28 DEBUG "resultType": "vector", 06:00:28 DEBUG "data": { 06:00:28 DEBUG "result": [ 06:00:28 DEBUG "resultType": "vector", 06:00:28 DEBUG { 06:00:28 DEBUG "result": [ 06:00:28 DEBUG "metric": { 06:00:28 DEBUG { 06:00:28 DEBUG "pod": "ds-idrepo-2" 06:00:28 DEBUG "metric": { 06:00:28 DEBUG }, 06:00:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:00:28 DEBUG "value": [ 06:00:28 DEBUG }, 06:00:28 DEBUG 06:00:28 DEBUG "value": [ 06:00:28 DEBUG 06:00:28 DEBUG 06:00:28 DEBUG 06:00:28 DEBUG 06:00:28 DEBUG [...] 06:00:28 DEBUG 06:00:28 DEBUG 06:00:28 DEBUG [...] 06:00:28 DEBUG 06:00:28 DEBUG 06:00:28 DEBUG 06:00:28 DEBUG 06:00:28 DEBUG "pod": "ds-cts-2" 06:00:28 DEBUG 06:00:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:00:28 DEBUG }, 06:00:28 DEBUG }, 06:00:28 DEBUG "value": [ 06:00:28 DEBUG "value": [ 06:00:28 DEBUG 1712725228, 06:00:28 DEBUG "0.000033333333333333335" 06:00:28 DEBUG 1712725228, 06:00:28 DEBUG ] 06:00:28 DEBUG } 06:00:28 DEBUG "4.982752008258176" 06:00:28 DEBUG ] 06:00:28 DEBUG ] 06:00:28 DEBUG } 06:00:28 DEBUG } 06:00:28 DEBUG } 06:00:28 DEBUG ] 06:00:28 DEBUG } 06:00:28 DEBUG } 06:00:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725229" 06:00:29 INFO [http_cmd]: http status code OK 06:00:29 DEBUG --- status code --- 06:00:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:29 DEBUG --- http response --- 06:00:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:00:29 DEBUG ----- output ----- 06:00:29 DEBUG { 06:00:29 DEBUG "status": "success", 06:00:29 DEBUG "data": { 06:00:29 DEBUG "resultType": "vector", 06:00:29 DEBUG "result": [ 06:00:29 DEBUG { 06:00:29 DEBUG "metric": { 06:00:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:00:29 DEBUG }, 06:00:29 DEBUG "value": [ 06:00:29 DEBUG 06:00:29 DEBUG 06:00:29 DEBUG 06:00:29 DEBUG [...] 06:00:29 DEBUG 06:00:29 DEBUG 06:00:29 DEBUG 06:00:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:00:29 DEBUG }, 06:00:29 DEBUG "value": [ 06:00:29 DEBUG 1712725229, 06:00:29 DEBUG "1463.8333333333333" 06:00:29 DEBUG ] 06:00:29 DEBUG } 06:00:29 DEBUG ] 06:00:29 DEBUG } 06:00:29 DEBUG } 06:00:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712725229" 06:00:29 INFO [http_cmd]: http status code OK 06:00:29 DEBUG --- status code --- 06:00:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:29 DEBUG --- http response --- 06:00:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:00:29 DEBUG ----- output ----- 06:00:29 DEBUG { 06:00:29 DEBUG "status": "success", 06:00:29 DEBUG "data": { 06:00:29 DEBUG "resultType": "vector", 06:00:29 DEBUG "result": [ 06:00:29 DEBUG { 06:00:29 DEBUG "metric": { 06:00:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:00:29 DEBUG }, 06:00:29 DEBUG "value": [ 06:00:29 DEBUG 06:00:29 DEBUG 06:00:29 DEBUG 06:00:29 DEBUG [...] 06:00:29 DEBUG 06:00:29 DEBUG 06:00:29 DEBUG 06:00:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:00:29 DEBUG }, 06:00:29 DEBUG "value": [ 06:00:29 DEBUG 1712725229, 06:00:29 DEBUG "0.0475" 06:00:29 DEBUG ] 06:00:29 DEBUG } 06:00:29 DEBUG ] 06:00:29 DEBUG } 06:00:29 DEBUG } 06:00:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725229" 06:00:29 INFO [http_cmd]: http status code OK 06:00:29 DEBUG --- status code --- 06:00:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:29 DEBUG --- http response --- 06:00:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:00:29 DEBUG ----- output ----- 06:00:29 DEBUG { 06:00:29 DEBUG "status": "success", 06:00:29 DEBUG "data": { 06:00:29 DEBUG "resultType": "vector", 06:00:29 DEBUG "result": [ 06:00:29 DEBUG { 06:00:29 DEBUG "metric": { 06:00:29 DEBUG "pod": "ds-idrepo-0" 06:00:29 DEBUG }, 06:00:29 DEBUG "value": [ 06:00:29 DEBUG 06:00:29 DEBUG 06:00:29 DEBUG 06:00:29 DEBUG [...] 06:00:29 DEBUG 06:00:29 DEBUG 06:00:29 DEBUG 06:00:29 DEBUG "pod": "ds-idrepo-1" 06:00:29 DEBUG }, 06:00:29 DEBUG "value": [ 06:00:29 DEBUG 1712725229, 06:00:29 DEBUG "0" 06:00:29 DEBUG ] 06:00:29 DEBUG } 06:00:29 DEBUG ] 06:00:29 DEBUG } 06:00:29 DEBUG } 06:00:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725229" 06:00:29 INFO [http_cmd]: http status code OK 06:00:29 DEBUG --- status code --- 06:00:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:29 DEBUG --- http response --- 06:00:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:00:29 DEBUG ----- output ----- 06:00:29 DEBUG { 06:00:29 DEBUG "status": "success", 06:00:29 DEBUG "data": { 06:00:29 DEBUG "resultType": "vector", 06:00:29 DEBUG "result": [ 06:00:29 DEBUG { 06:00:29 DEBUG "metric": { 06:00:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:00:29 DEBUG }, 06:00:29 DEBUG "value": [ 06:00:29 DEBUG 06:00:29 DEBUG 06:00:29 DEBUG 06:00:29 DEBUG [...] 06:00:29 DEBUG 06:00:29 DEBUG 06:00:29 DEBUG 06:00:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:00:29 DEBUG }, 06:00:29 DEBUG "value": [ 06:00:29 DEBUG 1712725229, 06:00:29 DEBUG "1439.1666666666667" 06:00:29 DEBUG ] 06:00:29 DEBUG } 06:00:29 DEBUG ] 06:00:29 DEBUG } 06:00:29 DEBUG } 06:00:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725229" 06:00:29 INFO [http_cmd]: http status code OK 06:00:29 DEBUG --- status code --- 06:00:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:29 DEBUG --- http response --- 06:00:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:00:29 DEBUG ----- output ----- 06:00:29 DEBUG { 06:00:29 DEBUG "status": "success", 06:00:29 DEBUG "data": { 06:00:29 DEBUG "resultType": "vector", 06:00:29 DEBUG "result": [ 06:00:29 DEBUG { 06:00:29 DEBUG "metric": { 06:00:29 DEBUG "pod": "ds-idrepo-0" 06:00:29 DEBUG }, 06:00:29 DEBUG "value": [ 06:00:29 DEBUG 06:00:29 DEBUG 06:00:29 DEBUG 06:00:29 DEBUG [...] 06:00:29 DEBUG 06:00:29 DEBUG 06:00:29 DEBUG 06:00:29 DEBUG "pod": "ds-idrepo-1" 06:00:29 DEBUG }, 06:00:29 DEBUG "value": [ 06:00:29 DEBUG 1712725229, 06:00:29 DEBUG "0" 06:00:29 DEBUG ] 06:00:29 DEBUG } 06:00:29 DEBUG ] 06:00:29 DEBUG } 06:00:29 DEBUG } 06:00:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725229" 06:00:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712725229" 06:00:29 INFO [http_cmd]: http status code OK 06:00:29 DEBUG --- status code --- 06:00:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:29 DEBUG --- http response --- 06:00:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:00:29 DEBUG ----- output ----- 06:00:29 DEBUG { 06:00:29 DEBUG "status": "success", 06:00:29 DEBUG "data": { 06:00:29 DEBUG "resultType": "vector", 06:00:29 DEBUG "result": [ 06:00:29 DEBUG { 06:00:29 DEBUG "metric": { 06:00:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:00:29 DEBUG }, 06:00:29 DEBUG "value": [ 06:00:29 DEBUG 06:00:29 DEBUG 06:00:29 DEBUG 06:00:29 DEBUG [...] 06:00:29 DEBUG 06:00:29 DEBUG 06:00:29 DEBUG 06:00:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:00:29 DEBUG }, 06:00:29 DEBUG "value": [ 06:00:29 DEBUG 1712725229, 06:00:29 DEBUG "0" 06:00:29 DEBUG ] 06:00:29 DEBUG } 06:00:29 DEBUG ] 06:00:29 DEBUG } 06:00:29 DEBUG } 06:00:29 INFO [http_cmd]: http status code OK 06:00:29 DEBUG --- status code --- 06:00:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:29 DEBUG --- http response --- 06:00:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:00:29 DEBUG ----- output ----- 06:00:29 DEBUG { 06:00:29 DEBUG "status": "success", 06:00:29 DEBUG "data": { 06:00:29 DEBUG "resultType": "vector", 06:00:29 DEBUG "result": [ 06:00:29 DEBUG { 06:00:29 DEBUG "metric": { 06:00:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:00:29 DEBUG }, 06:00:29 DEBUG "value": [ 06:00:29 DEBUG 06:00:29 DEBUG 06:00:29 DEBUG 06:00:29 DEBUG [...] 06:00:29 DEBUG 06:00:29 DEBUG 06:00:29 DEBUG 06:00:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:00:29 DEBUG }, 06:00:29 DEBUG "value": [ 06:00:29 DEBUG 1712725229, 06:00:29 DEBUG "2490052608" 06:00:29 DEBUG ] 06:00:29 DEBUG } 06:00:29 DEBUG ] 06:00:29 DEBUG } 06:00:29 DEBUG } 06:00:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725229" 06:00:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725230" 06:00:30 INFO [http_cmd]: http status code OK 06:00:30 DEBUG --- status code --- 06:00:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:30 DEBUG --- http response --- 06:00:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:00:30 DEBUG ----- output ----- 06:00:30 DEBUG { 06:00:30 DEBUG "status": "success", 06:00:30 DEBUG "data": { 06:00:30 DEBUG "resultType": "vector", 06:00:30 DEBUG "result": [ 06:00:30 DEBUG { 06:00:30 DEBUG "metric": { 06:00:30 DEBUG "instance": "10.142.0.25:9100" 06:00:30 DEBUG }, 06:00:30 DEBUG "value": [ 06:00:30 DEBUG 06:00:30 DEBUG 06:00:30 DEBUG 06:00:30 DEBUG [...] 06:00:30 DEBUG 06:00:30 DEBUG 06:00:30 DEBUG 06:00:30 DEBUG "instance": "10.142.0.77:9100" 06:00:30 DEBUG }, 06:00:30 DEBUG "value": [ 06:00:30 DEBUG 1712725229, 06:00:30 DEBUG "42402150.21330489" 06:00:30 DEBUG ] 06:00:30 DEBUG } 06:00:30 DEBUG ] 06:00:30 DEBUG } 06:00:30 DEBUG } 06:00:30 INFO [http_cmd]: http status code OK 06:00:30 DEBUG --- status code --- 06:00:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:30 DEBUG --- http response --- 06:00:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:00:30 DEBUG ----- output ----- 06:00:30 DEBUG { 06:00:30 DEBUG "status": "success", 06:00:30 DEBUG "data": { 06:00:30 DEBUG "resultType": "vector", 06:00:30 DEBUG "result": [ 06:00:30 DEBUG { 06:00:30 DEBUG "metric": { 06:00:30 DEBUG "instance": "10.142.0.25:9100" 06:00:30 DEBUG }, 06:00:30 DEBUG "value": [ 06:00:30 DEBUG 06:00:30 DEBUG 06:00:30 DEBUG 06:00:30 DEBUG [...] 06:00:30 DEBUG 06:00:30 DEBUG 06:00:30 DEBUG 06:00:30 DEBUG "instance": "10.142.0.77:9100" 06:00:30 DEBUG }, 06:00:30 DEBUG "value": [ 06:00:30 DEBUG 1712725230, 06:00:30 DEBUG "0.0493267564324758" 06:00:30 DEBUG ] 06:00:30 DEBUG } 06:00:30 DEBUG ] 06:00:30 DEBUG } 06:00:30 DEBUG } 06:00:46 INFO 06:00:46 INFO [loop_until]: kubectl --namespace=xlou top pods 06:00:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:00:46 INFO [loop_until]: OK (rc = 0) 06:00:46 DEBUG --- stdout --- 06:00:46 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10392m 8968Mi am-654c4fcbcf-9scrj 9860m 8439Mi am-654c4fcbcf-v5qb5 8367m 8427Mi ds-cts-0 8329m 12911Mi ds-cts-1 8574m 12434Mi ds-cts-2 8105m 12663Mi ds-idrepo-0 10734m 5622Mi ds-idrepo-1 353m 5671Mi ds-idrepo-2 402m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 9m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4039m 2232Mi 06:00:46 DEBUG --- stderr --- 06:00:46 DEBUG 06:00:54 INFO 06:00:54 INFO [loop_until]: kubectl --namespace=xlou top node 06:00:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:00:54 INFO [loop_until]: OK (rc = 0) 06:00:54 DEBUG --- stdout --- 06:00:54 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 96m 0% 2303Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9294m 58% 9304Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 81m 0% 3264Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13825m 87% 6496Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10924m 68% 9325Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11527m 72% 9799Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 78m 0% 857Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10861m 68% 6295Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8347m 52% 13528Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 8184m 51% 13362Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 452m 2% 6383Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8569m 53% 13101Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 419m 2% 6344Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4256m 26% 5881Mi 10% 06:00:54 DEBUG --- stderr --- 06:00:54 DEBUG 06:01:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712725286" 06:01:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725286" 06:01:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725286" 06:01:26 INFO [http_cmd]: http status code OK 06:01:26 ERROR [http_cmd]: ERROR 06:01:26 INFO [http_cmd]: http status code OK 06:01:26 DEBUG --- status code --- 06:01:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:26 DEBUG --- status code --- 06:01:26 DEBUG --- http response --- 06:01:26 ERROR --- status code --- 06:01:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:26 DEBUG --- http response --- 06:01:26 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:01:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:01:26 ERROR --- http response --- 06:01:26 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:01:26 DEBUG ----- output ----- 06:01:26 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:01:26 DEBUG { 06:01:26 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:01:26 DEBUG "status": "success", 06:01:26 DEBUG "data": { 06:01:26 DEBUG "resultType": "vector", 06:01:26 DEBUG "result": [ 06:01:26 DEBUG { 06:01:26 DEBUG "metric": { 06:01:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:01:26 DEBUG }, 06:01:26 DEBUG "value": [ 06:01:26 DEBUG 06:01:26 DEBUG 06:01:26 DEBUG 06:01:26 DEBUG [...] 06:01:26 DEBUG 06:01:26 DEBUG 06:01:26 DEBUG 06:01:26 DEBUG "pod": "am-654c4fcbcf-487cv" 06:01:26 DEBUG }, 06:01:26 DEBUG "value": [ 06:01:26 DEBUG 1712725286, 06:01:26 DEBUG "0" 06:01:26 DEBUG ] 06:01:26 DEBUG } 06:01:26 DEBUG ] 06:01:26 DEBUG } 06:01:26 DEBUG } 06:01:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725286" 06:01:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725286" 06:01:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725287" 06:01:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725287" 06:01:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725287" 06:01:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725287" 06:01:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725287" 06:01:27 INFO [http_cmd]: http status code OK 06:01:27 DEBUG --- status code --- 06:01:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:27 DEBUG --- http response --- 06:01:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:01:27 DEBUG ----- output ----- 06:01:27 DEBUG { 06:01:27 DEBUG "status": "success", 06:01:27 DEBUG "data": { 06:01:27 INFO [http_cmd]: http status code OK 06:01:27 DEBUG "resultType": "vector", 06:01:27 DEBUG "result": [ 06:01:27 DEBUG --- status code --- 06:01:27 DEBUG { 06:01:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:27 DEBUG "metric": { 06:01:27 DEBUG --- http response --- 06:01:27 DEBUG "instance": "10.142.0.25:9100" 06:01:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:01:27 DEBUG }, 06:01:27 DEBUG ----- output ----- 06:01:27 DEBUG { 06:01:27 DEBUG "status": "success", 06:01:27 INFO [http_cmd]: http status code OK 06:01:27 DEBUG "data": { 06:01:27 INFO [http_cmd]: http status code OK 06:01:27 DEBUG "value": [ 06:01:27 DEBUG --- status code --- 06:01:27 DEBUG "resultType": "vector", 06:01:27 DEBUG --- status code --- 06:01:27 DEBUG 06:01:27 DEBUG 06:01:27 DEBUG "result": [ 06:01:27 DEBUG { 06:01:27 DEBUG "metric": { 06:01:27 DEBUG 06:01:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:01:27 DEBUG [...] 06:01:27 DEBUG --- http response --- 06:01:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:01:27 DEBUG }, 06:01:27 DEBUG 06:01:27 DEBUG 06:01:27 DEBUG 06:01:27 DEBUG --- http response --- 06:01:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:01:27 DEBUG "value": [ 06:01:27 DEBUG "instance": "10.142.0.77:9100" 06:01:27 DEBUG }, 06:01:27 INFO [http_cmd]: http status code OK 06:01:27 DEBUG --- status code --- 06:01:27 DEBUG 06:01:27 DEBUG "value": [ 06:01:27 DEBUG 1712725286, 06:01:27 DEBUG "42375505.166666664" 06:01:27 DEBUG ] 06:01:27 DEBUG 06:01:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:27 DEBUG --- http response --- 06:01:27 DEBUG ----- output ----- 06:01:27 DEBUG { 06:01:27 DEBUG 06:01:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:01:27 DEBUG ----- output ----- 06:01:27 DEBUG { 06:01:27 DEBUG "status": "success", 06:01:27 DEBUG [...] 06:01:27 DEBUG 06:01:27 DEBUG "data": { 06:01:27 DEBUG "resultType": "vector", 06:01:27 DEBUG "result": [ 06:01:27 DEBUG 06:01:27 DEBUG 06:01:27 DEBUG { 06:01:27 DEBUG "metric": { 06:01:27 DEBUG "pod": "ds-idrepo-0" 06:01:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:01:27 DEBUG }, 06:01:27 DEBUG }, 06:01:27 DEBUG "status": "success", 06:01:27 DEBUG } 06:01:27 DEBUG "value": [ 06:01:27 DEBUG 1712725286, 06:01:27 DEBUG "value": [ 06:01:27 DEBUG "data": { 06:01:27 DEBUG "resultType": "vector", 06:01:27 DEBUG "1429.9" 06:01:27 DEBUG ] 06:01:27 INFO [http_cmd]: http status code OK 06:01:27 DEBUG --- status code --- 06:01:27 DEBUG "result": [ 06:01:27 DEBUG ] 06:01:27 DEBUG ----- output ----- 06:01:27 INFO [http_cmd]: http status code OK 06:01:27 DEBUG } 06:01:27 DEBUG ] 06:01:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:27 DEBUG { 06:01:27 DEBUG } 06:01:27 DEBUG { 06:01:27 DEBUG --- status code --- 06:01:27 DEBUG 06:01:27 DEBUG } 06:01:27 DEBUG } 06:01:27 DEBUG "metric": { 06:01:27 DEBUG "pod": "ds-idrepo-0" 06:01:27 DEBUG "status": "success", 06:01:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:27 DEBUG 06:01:27 DEBUG --- http response --- 06:01:27 DEBUG } 06:01:27 DEBUG }, 06:01:27 DEBUG "value": [ 06:01:27 DEBUG --- http response --- 06:01:27 DEBUG 06:01:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:01:27 DEBUG "data": { 06:01:27 DEBUG "resultType": "vector", 06:01:27 DEBUG "result": [ 06:01:27 DEBUG { 06:01:27 DEBUG "metric": { 06:01:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:01:27 DEBUG }, 06:01:27 DEBUG "value": [ 06:01:27 DEBUG 06:01:27 DEBUG 06:01:27 DEBUG 06:01:27 DEBUG [...] 06:01:27 DEBUG 06:01:27 DEBUG 06:01:27 DEBUG 06:01:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:01:27 DEBUG }, 06:01:27 DEBUG "value": [ 06:01:27 DEBUG 1712725287, 06:01:27 DEBUG "259184" 06:01:27 DEBUG ] 06:01:27 DEBUG } 06:01:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:01:27 DEBUG ----- output ----- 06:01:27 DEBUG ] 06:01:27 DEBUG } 06:01:27 DEBUG [...] 06:01:27 DEBUG 06:01:27 DEBUG 06:01:27 DEBUG 06:01:27 DEBUG 06:01:27 DEBUG 06:01:27 DEBUG [...] 06:01:27 DEBUG 06:01:27 DEBUG 06:01:27 DEBUG 06:01:27 DEBUG "pod": "ds-cts-2" 06:01:27 DEBUG ----- output ----- 06:01:27 DEBUG { 06:01:27 DEBUG 06:01:27 DEBUG }, 06:01:27 DEBUG "value": [ 06:01:27 DEBUG { 06:01:27 DEBUG "status": "success", 06:01:27 DEBUG "pod": "ds-cts-2" 06:01:27 DEBUG 1712725287, 06:01:27 DEBUG "0" 06:01:27 DEBUG ] 06:01:27 DEBUG } 06:01:27 DEBUG ] 06:01:27 DEBUG "status": "success", 06:01:27 DEBUG }, 06:01:27 DEBUG "value": [ 06:01:27 DEBUG 1712725287, 06:01:27 DEBUG "29.966666666666665" 06:01:27 DEBUG ] 06:01:27 DEBUG "data": { 06:01:27 DEBUG } 06:01:27 DEBUG ] 06:01:27 DEBUG } 06:01:27 DEBUG } 06:01:27 DEBUG "resultType": "vector", 06:01:27 DEBUG "data": { 06:01:27 DEBUG "resultType": "vector", 06:01:27 DEBUG "result": [ 06:01:27 DEBUG { 06:01:27 DEBUG "metric": { 06:01:27 DEBUG "result": [ 06:01:27 DEBUG "instance": "10.142.0.25:9100" 06:01:27 DEBUG }, 06:01:27 DEBUG "value": [ 06:01:27 DEBUG { 06:01:27 DEBUG 06:01:27 DEBUG 06:01:27 DEBUG 06:01:27 DEBUG "metric": { 06:01:27 DEBUG [...] 06:01:27 DEBUG 06:01:27 DEBUG 06:01:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:01:27 DEBUG } 06:01:27 DEBUG 06:01:27 DEBUG "instance": "10.142.0.77:9100" 06:01:27 DEBUG }, 06:01:27 DEBUG }, 06:01:27 DEBUG "value": [ 06:01:27 DEBUG 1712725287, 06:01:27 DEBUG "value": [ 06:01:27 DEBUG "342327356416" 06:01:27 DEBUG ] 06:01:27 DEBUG 06:01:27 DEBUG } 06:01:27 DEBUG ] 06:01:27 DEBUG 06:01:27 DEBUG } 06:01:27 DEBUG } 06:01:27 DEBUG 06:01:27 DEBUG } 06:01:27 DEBUG [...] 06:01:27 DEBUG } 06:01:27 DEBUG 06:01:27 DEBUG 06:01:27 DEBUG 06:01:27 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:01:27 DEBUG }, 06:01:27 DEBUG "value": [ 06:01:27 DEBUG 1712725287, 06:01:27 DEBUG "0" 06:01:27 DEBUG ] 06:01:27 DEBUG } 06:01:27 DEBUG ] 06:01:27 DEBUG } 06:01:27 DEBUG } 06:01:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725287" 06:01:27 INFO [http_cmd]: http status code OK 06:01:27 DEBUG --- status code --- 06:01:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:27 DEBUG --- http response --- 06:01:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:01:27 DEBUG ----- output ----- 06:01:27 DEBUG { 06:01:27 DEBUG "status": "success", 06:01:27 DEBUG "data": { 06:01:27 DEBUG "resultType": "vector", 06:01:27 DEBUG "result": [ 06:01:27 DEBUG { 06:01:27 DEBUG "metric": { 06:01:27 DEBUG "pod": "ds-idrepo-0" 06:01:27 DEBUG }, 06:01:27 DEBUG "value": [ 06:01:27 DEBUG 06:01:27 DEBUG 06:01:27 DEBUG 06:01:27 DEBUG [...] 06:01:27 DEBUG 06:01:27 DEBUG 06:01:27 DEBUG 06:01:27 DEBUG "pod": "ds-idrepo-1" 06:01:27 DEBUG }, 06:01:27 DEBUG "value": [ 06:01:27 DEBUG 1712725287, 06:01:27 DEBUG "0" 06:01:27 DEBUG ] 06:01:27 DEBUG } 06:01:27 DEBUG ] 06:01:27 DEBUG } 06:01:27 DEBUG } 06:01:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725287" 06:01:27 INFO [http_cmd]: http status code OK 06:01:27 DEBUG --- status code --- 06:01:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:27 DEBUG --- http response --- 06:01:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:01:27 DEBUG ----- output ----- 06:01:27 DEBUG { 06:01:27 DEBUG "status": "success", 06:01:27 DEBUG "data": { 06:01:27 DEBUG "resultType": "vector", 06:01:27 DEBUG "result": [ 06:01:27 DEBUG { 06:01:27 DEBUG "metric": { 06:01:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:01:27 DEBUG }, 06:01:27 DEBUG "value": [ 06:01:27 DEBUG 06:01:27 DEBUG 06:01:27 DEBUG 06:01:27 DEBUG [...] 06:01:27 DEBUG 06:01:27 DEBUG 06:01:27 DEBUG 06:01:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:01:27 DEBUG }, 06:01:27 DEBUG "value": [ 06:01:27 DEBUG 1712725287, 06:01:27 DEBUG "1413.9333333333334" 06:01:27 DEBUG ] 06:01:27 DEBUG } 06:01:27 DEBUG ] 06:01:27 DEBUG } 06:01:27 DEBUG } 06:01:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725287" 06:01:27 INFO [http_cmd]: http status code OK 06:01:27 DEBUG --- status code --- 06:01:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:27 DEBUG --- http response --- 06:01:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:01:27 DEBUG ----- output ----- 06:01:27 DEBUG { 06:01:27 DEBUG "status": "success", 06:01:27 DEBUG "data": { 06:01:27 DEBUG "resultType": "vector", 06:01:27 DEBUG "result": [ 06:01:27 DEBUG { 06:01:27 DEBUG "metric": { 06:01:27 DEBUG "instance": "10.142.0.25:9100" 06:01:27 DEBUG }, 06:01:27 DEBUG "value": [ 06:01:27 DEBUG 06:01:27 DEBUG 06:01:27 DEBUG 06:01:27 DEBUG [...] 06:01:27 DEBUG 06:01:27 DEBUG 06:01:27 DEBUG 06:01:27 DEBUG "instance": "10.142.0.77:9100" 06:01:27 DEBUG }, 06:01:27 DEBUG "value": [ 06:01:27 DEBUG 1712725287, 06:01:27 DEBUG "0.04863333333333249" 06:01:27 DEBUG ] 06:01:27 DEBUG } 06:01:27 DEBUG ] 06:01:27 DEBUG } 06:01:27 DEBUG } 06:01:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725288" 06:01:28 INFO [http_cmd]: http status code OK 06:01:28 DEBUG --- status code --- 06:01:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:28 DEBUG --- http response --- 06:01:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 06:01:28 DEBUG ----- output ----- 06:01:28 DEBUG { 06:01:28 DEBUG "status": "success", 06:01:28 DEBUG "data": { 06:01:28 DEBUG "resultType": "vector", 06:01:28 DEBUG "result": [ 06:01:28 DEBUG { 06:01:28 DEBUG "metric": { 06:01:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:01:28 DEBUG }, 06:01:28 DEBUG "value": [ 06:01:28 DEBUG 06:01:28 DEBUG 06:01:28 DEBUG 06:01:28 DEBUG [...] 06:01:28 DEBUG 06:01:28 DEBUG 06:01:28 DEBUG 06:01:28 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:01:28 DEBUG }, 06:01:28 DEBUG "value": [ 06:01:28 DEBUG 1712725288, 06:01:28 DEBUG "80237606.49782379" 06:01:28 DEBUG ] 06:01:28 DEBUG } 06:01:28 DEBUG ] 06:01:28 DEBUG } 06:01:28 DEBUG } 06:01:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712725288" 06:01:28 INFO [http_cmd]: http status code OK 06:01:28 DEBUG --- status code --- 06:01:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:28 DEBUG --- http response --- 06:01:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:01:28 DEBUG ----- output ----- 06:01:28 DEBUG { 06:01:28 DEBUG "status": "success", 06:01:28 DEBUG "data": { 06:01:28 DEBUG "resultType": "vector", 06:01:28 DEBUG "result": [ 06:01:28 DEBUG { 06:01:28 DEBUG "metric": { 06:01:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:01:28 DEBUG }, 06:01:28 DEBUG "value": [ 06:01:28 DEBUG 06:01:28 DEBUG 06:01:28 DEBUG 06:01:28 DEBUG [...] 06:01:28 DEBUG 06:01:28 DEBUG 06:01:28 DEBUG 06:01:28 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:01:28 DEBUG }, 06:01:28 DEBUG "value": [ 06:01:28 DEBUG 1712725288, 06:01:28 DEBUG "4.1194080742609405" 06:01:28 DEBUG ] 06:01:28 DEBUG } 06:01:28 DEBUG ] 06:01:28 DEBUG } 06:01:28 DEBUG } 06:01:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712725288" 06:01:28 INFO [http_cmd]: http status code OK 06:01:28 DEBUG --- status code --- 06:01:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:28 DEBUG --- http response --- 06:01:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:01:28 DEBUG ----- output ----- 06:01:28 DEBUG { 06:01:28 DEBUG "status": "success", 06:01:28 DEBUG "data": { 06:01:28 DEBUG "resultType": "vector", 06:01:28 DEBUG "result": [ 06:01:28 DEBUG { 06:01:28 DEBUG "metric": { 06:01:28 DEBUG "pod": "ds-idrepo-0" 06:01:28 DEBUG }, 06:01:28 DEBUG "value": [ 06:01:28 DEBUG 06:01:28 DEBUG 06:01:28 DEBUG 06:01:28 DEBUG [...] 06:01:28 DEBUG 06:01:28 DEBUG 06:01:28 DEBUG 06:01:28 DEBUG "pod": "ds-cts-2" 06:01:28 DEBUG }, 06:01:28 DEBUG "value": [ 06:01:28 DEBUG 1712725288, 06:01:28 DEBUG "0" 06:01:28 DEBUG ] 06:01:28 DEBUG } 06:01:28 DEBUG ] 06:01:28 DEBUG } 06:01:28 DEBUG } 06:01:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725288" 06:01:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725288" 06:01:29 INFO [http_cmd]: http status code OK 06:01:29 INFO [http_cmd]: http status code OK 06:01:29 DEBUG --- status code --- 06:01:29 DEBUG --- status code --- 06:01:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:29 DEBUG --- http response --- 06:01:29 DEBUG --- http response --- 06:01:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:01:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:01:29 DEBUG ----- output ----- 06:01:29 DEBUG ----- output ----- 06:01:29 DEBUG { 06:01:29 DEBUG { 06:01:29 DEBUG "status": "success", 06:01:29 DEBUG "status": "success", 06:01:29 DEBUG "data": { 06:01:29 DEBUG "data": { 06:01:29 DEBUG "resultType": "vector", 06:01:29 DEBUG "resultType": "vector", 06:01:29 DEBUG "result": [ 06:01:29 DEBUG "result": [ 06:01:29 DEBUG { 06:01:29 DEBUG { 06:01:29 DEBUG "metric": { 06:01:29 DEBUG "metric": { 06:01:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:01:29 DEBUG }, 06:01:29 DEBUG "pod": "ds-idrepo-2" 06:01:29 DEBUG "value": [ 06:01:29 DEBUG }, 06:01:29 DEBUG 06:01:29 DEBUG "value": [ 06:01:29 DEBUG 06:01:29 DEBUG 06:01:29 DEBUG 06:01:29 DEBUG 06:01:29 DEBUG [...] 06:01:29 DEBUG 06:01:29 DEBUG 06:01:29 DEBUG [...] 06:01:29 DEBUG 06:01:29 DEBUG 06:01:29 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:01:29 DEBUG }, 06:01:29 DEBUG 06:01:29 DEBUG "value": [ 06:01:29 DEBUG 06:01:29 DEBUG 1712725288, 06:01:29 DEBUG 06:01:29 DEBUG "4.0920291259124175" 06:01:29 DEBUG "pod": "ds-cts-2" 06:01:29 DEBUG ] 06:01:29 DEBUG }, 06:01:29 DEBUG } 06:01:29 DEBUG "value": [ 06:01:29 DEBUG ] 06:01:29 DEBUG 1712725288, 06:01:29 DEBUG } 06:01:29 DEBUG "0.00011532333333333334" 06:01:29 DEBUG } 06:01:29 DEBUG ] 06:01:29 DEBUG } 06:01:29 DEBUG ] 06:01:29 DEBUG } 06:01:29 DEBUG } 06:01:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725289" 06:01:29 INFO [http_cmd]: http status code OK 06:01:29 DEBUG --- status code --- 06:01:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:29 DEBUG --- http response --- 06:01:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:01:29 DEBUG ----- output ----- 06:01:29 DEBUG { 06:01:29 DEBUG "status": "success", 06:01:29 DEBUG "data": { 06:01:29 DEBUG "resultType": "vector", 06:01:29 DEBUG "result": [ 06:01:29 DEBUG { 06:01:29 DEBUG "metric": { 06:01:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:01:29 DEBUG }, 06:01:29 DEBUG "value": [ 06:01:29 DEBUG 06:01:29 DEBUG 06:01:29 DEBUG 06:01:29 DEBUG [...] 06:01:29 DEBUG 06:01:29 DEBUG 06:01:29 DEBUG 06:01:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:01:29 DEBUG }, 06:01:29 DEBUG "value": [ 06:01:29 DEBUG 1712725289, 06:01:29 DEBUG "1413.6" 06:01:29 DEBUG ] 06:01:29 DEBUG } 06:01:29 DEBUG ] 06:01:29 DEBUG } 06:01:29 DEBUG } 06:01:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712725289" 06:01:29 INFO [http_cmd]: http status code OK 06:01:29 DEBUG --- status code --- 06:01:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:29 DEBUG --- http response --- 06:01:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:01:29 DEBUG ----- output ----- 06:01:29 DEBUG { 06:01:29 DEBUG "status": "success", 06:01:29 DEBUG "data": { 06:01:29 DEBUG "resultType": "vector", 06:01:29 DEBUG "result": [ 06:01:29 DEBUG { 06:01:29 DEBUG "metric": { 06:01:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:01:29 DEBUG }, 06:01:29 DEBUG "value": [ 06:01:29 DEBUG 06:01:29 DEBUG 06:01:29 DEBUG 06:01:29 DEBUG [...] 06:01:29 DEBUG 06:01:29 DEBUG 06:01:29 DEBUG 06:01:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:01:29 DEBUG }, 06:01:29 DEBUG "value": [ 06:01:29 DEBUG 1712725289, 06:01:29 DEBUG "0.048" 06:01:29 DEBUG ] 06:01:29 DEBUG } 06:01:29 DEBUG ] 06:01:29 DEBUG } 06:01:29 DEBUG } 06:01:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725289" 06:01:29 INFO [http_cmd]: http status code OK 06:01:29 DEBUG --- status code --- 06:01:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:29 DEBUG --- http response --- 06:01:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:01:29 DEBUG ----- output ----- 06:01:29 DEBUG { 06:01:29 DEBUG "status": "success", 06:01:29 DEBUG "data": { 06:01:29 DEBUG "resultType": "vector", 06:01:29 DEBUG "result": [ 06:01:29 DEBUG { 06:01:29 DEBUG "metric": { 06:01:29 DEBUG "pod": "ds-idrepo-0" 06:01:29 DEBUG }, 06:01:29 DEBUG "value": [ 06:01:29 DEBUG 06:01:29 DEBUG 06:01:29 DEBUG 06:01:29 DEBUG [...] 06:01:29 DEBUG 06:01:29 DEBUG 06:01:29 DEBUG 06:01:29 DEBUG "pod": "ds-idrepo-1" 06:01:29 DEBUG }, 06:01:29 DEBUG "value": [ 06:01:29 DEBUG 1712725289, 06:01:29 DEBUG "0" 06:01:29 DEBUG ] 06:01:29 DEBUG } 06:01:29 DEBUG ] 06:01:29 DEBUG } 06:01:29 DEBUG } 06:01:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725289" 06:01:29 INFO [http_cmd]: http status code OK 06:01:29 DEBUG --- status code --- 06:01:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:29 DEBUG --- http response --- 06:01:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:01:29 DEBUG ----- output ----- 06:01:29 DEBUG { 06:01:29 DEBUG "status": "success", 06:01:29 DEBUG "data": { 06:01:29 DEBUG "resultType": "vector", 06:01:29 DEBUG "result": [ 06:01:29 DEBUG { 06:01:29 DEBUG "metric": { 06:01:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:01:29 DEBUG }, 06:01:29 DEBUG "value": [ 06:01:29 DEBUG 06:01:29 DEBUG 06:01:29 DEBUG 06:01:29 DEBUG [...] 06:01:29 DEBUG 06:01:29 DEBUG 06:01:29 DEBUG 06:01:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:01:29 DEBUG }, 06:01:29 DEBUG "value": [ 06:01:29 DEBUG 1712725289, 06:01:29 DEBUG "1429.3" 06:01:29 DEBUG ] 06:01:29 DEBUG } 06:01:29 DEBUG ] 06:01:29 DEBUG } 06:01:29 DEBUG } 06:01:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725289" 06:01:29 INFO [http_cmd]: http status code OK 06:01:29 DEBUG --- status code --- 06:01:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:29 DEBUG --- http response --- 06:01:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:01:29 DEBUG ----- output ----- 06:01:29 DEBUG { 06:01:29 DEBUG "status": "success", 06:01:29 DEBUG "data": { 06:01:29 DEBUG "resultType": "vector", 06:01:29 DEBUG "result": [ 06:01:29 DEBUG { 06:01:29 DEBUG "metric": { 06:01:29 DEBUG "pod": "ds-idrepo-0" 06:01:29 DEBUG }, 06:01:29 DEBUG "value": [ 06:01:29 DEBUG 06:01:29 DEBUG 06:01:29 DEBUG 06:01:29 DEBUG [...] 06:01:29 DEBUG 06:01:29 DEBUG 06:01:29 DEBUG 06:01:29 DEBUG "pod": "ds-idrepo-1" 06:01:29 DEBUG }, 06:01:29 DEBUG "value": [ 06:01:29 DEBUG 1712725289, 06:01:29 DEBUG "0" 06:01:29 DEBUG ] 06:01:29 DEBUG } 06:01:29 DEBUG ] 06:01:29 DEBUG } 06:01:29 DEBUG } 06:01:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725290" 06:01:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712725290" 06:01:30 INFO [http_cmd]: http status code OK 06:01:30 DEBUG --- status code --- 06:01:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:30 DEBUG --- http response --- 06:01:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:01:30 DEBUG ----- output ----- 06:01:30 DEBUG { 06:01:30 DEBUG "status": "success", 06:01:30 DEBUG "data": { 06:01:30 DEBUG "resultType": "vector", 06:01:30 DEBUG "result": [ 06:01:30 DEBUG { 06:01:30 DEBUG "metric": { 06:01:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:01:30 DEBUG }, 06:01:30 DEBUG "value": [ 06:01:30 DEBUG 06:01:30 DEBUG 06:01:30 DEBUG 06:01:30 DEBUG [...] 06:01:30 DEBUG 06:01:30 DEBUG 06:01:30 DEBUG 06:01:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:01:30 DEBUG }, 06:01:30 DEBUG "value": [ 06:01:30 DEBUG 1712725290, 06:01:30 DEBUG "0" 06:01:30 DEBUG ] 06:01:30 DEBUG } 06:01:30 DEBUG ] 06:01:30 DEBUG } 06:01:30 DEBUG } 06:01:30 INFO [http_cmd]: http status code OK 06:01:30 DEBUG --- status code --- 06:01:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:30 DEBUG --- http response --- 06:01:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:01:30 DEBUG ----- output ----- 06:01:30 DEBUG { 06:01:30 DEBUG "status": "success", 06:01:30 DEBUG "data": { 06:01:30 DEBUG "resultType": "vector", 06:01:30 DEBUG "result": [ 06:01:30 DEBUG { 06:01:30 DEBUG "metric": { 06:01:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:01:30 DEBUG }, 06:01:30 DEBUG "value": [ 06:01:30 DEBUG 06:01:30 DEBUG 06:01:30 DEBUG 06:01:30 DEBUG [...] 06:01:30 DEBUG 06:01:30 DEBUG 06:01:30 DEBUG 06:01:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:01:30 DEBUG }, 06:01:30 DEBUG "value": [ 06:01:30 DEBUG 1712725290, 06:01:30 DEBUG "2489565184" 06:01:30 DEBUG ] 06:01:30 DEBUG } 06:01:30 DEBUG ] 06:01:30 DEBUG } 06:01:30 DEBUG } 06:01:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725290" 06:01:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725290" 06:01:30 INFO [http_cmd]: http status code OK 06:01:30 DEBUG --- status code --- 06:01:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:30 DEBUG --- http response --- 06:01:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:01:30 DEBUG ----- output ----- 06:01:30 DEBUG { 06:01:30 DEBUG "status": "success", 06:01:30 DEBUG "data": { 06:01:30 DEBUG "resultType": "vector", 06:01:30 DEBUG "result": [ 06:01:30 DEBUG { 06:01:30 DEBUG "metric": { 06:01:30 DEBUG "instance": "10.142.0.25:9100" 06:01:30 DEBUG }, 06:01:30 DEBUG "value": [ 06:01:30 DEBUG 06:01:30 DEBUG 06:01:30 DEBUG 06:01:30 DEBUG [...] 06:01:30 DEBUG 06:01:30 DEBUG 06:01:30 DEBUG 06:01:30 DEBUG "instance": "10.142.0.77:9100" 06:01:30 DEBUG }, 06:01:30 DEBUG "value": [ 06:01:30 DEBUG 1712725290, 06:01:30 DEBUG "42377163.333333336" 06:01:30 DEBUG ] 06:01:30 DEBUG } 06:01:30 DEBUG ] 06:01:30 DEBUG } 06:01:30 DEBUG } 06:01:30 INFO [http_cmd]: http status code OK 06:01:30 DEBUG --- status code --- 06:01:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:30 DEBUG --- http response --- 06:01:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:01:30 DEBUG ----- output ----- 06:01:30 DEBUG { 06:01:30 DEBUG "status": "success", 06:01:30 DEBUG "data": { 06:01:30 DEBUG "resultType": "vector", 06:01:30 DEBUG "result": [ 06:01:30 DEBUG { 06:01:30 DEBUG "metric": { 06:01:30 DEBUG "instance": "10.142.0.25:9100" 06:01:30 DEBUG }, 06:01:30 DEBUG "value": [ 06:01:30 DEBUG 06:01:30 DEBUG 06:01:30 DEBUG 06:01:30 DEBUG [...] 06:01:30 DEBUG 06:01:30 DEBUG 06:01:30 DEBUG 06:01:30 DEBUG "instance": "10.142.0.77:9100" 06:01:30 DEBUG }, 06:01:30 DEBUG "value": [ 06:01:30 DEBUG 1712725290, 06:01:30 DEBUG "0.019333333333333154" 06:01:30 DEBUG ] 06:01:30 DEBUG } 06:01:30 DEBUG ] 06:01:30 DEBUG } 06:01:30 DEBUG } 06:01:46 INFO 06:01:46 INFO [loop_until]: kubectl --namespace=xlou top pods 06:01:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:01:46 INFO [loop_until]: OK (rc = 0) 06:01:46 DEBUG --- stdout --- 06:01:46 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10179m 8960Mi am-654c4fcbcf-9scrj 9691m 8442Mi am-654c4fcbcf-v5qb5 8514m 8454Mi ds-cts-0 8605m 13379Mi ds-cts-1 8224m 12986Mi ds-cts-2 7906m 13393Mi ds-idrepo-0 10706m 5622Mi ds-idrepo-1 348m 5671Mi ds-idrepo-2 427m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 8m 2213Mi lodemon-b776b7d88-thrqf 5m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4163m 2234Mi 06:01:46 DEBUG --- stderr --- 06:01:46 DEBUG 06:01:54 INFO 06:01:54 INFO [loop_until]: kubectl --namespace=xlou top node 06:01:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:01:54 INFO [loop_until]: OK (rc = 0) 06:01:54 DEBUG --- stdout --- 06:01:54 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 93m 0% 2305Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9235m 58% 9317Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 82m 0% 3262Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 12705m 79% 6521Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10992m 69% 9321Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11464m 72% 9797Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 78m 0% 855Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10795m 67% 6296Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8632m 54% 14039Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 8073m 50% 14048Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 469m 2% 6381Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8489m 53% 13710Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 438m 2% 6344Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4226m 26% 5883Mi 10% 06:01:54 DEBUG --- stderr --- 06:01:54 DEBUG 06:02:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725346" 06:02:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712725346" 06:02:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725346" 06:02:26 ERROR [http_cmd]: ERROR 06:02:26 INFO [http_cmd]: http status code OK 06:02:26 ERROR --- status code --- 06:02:26 DEBUG --- status code --- 06:02:26 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:02:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:26 ERROR --- http response --- 06:02:26 DEBUG --- http response --- 06:02:26 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:02:26 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:02:26 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:02:26 INFO [http_cmd]: http status code OK 06:02:26 DEBUG --- status code --- 06:02:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:26 DEBUG --- http response --- 06:02:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:02:26 DEBUG ----- output ----- 06:02:26 DEBUG { 06:02:26 DEBUG "status": "success", 06:02:26 DEBUG "data": { 06:02:26 DEBUG "resultType": "vector", 06:02:26 DEBUG "result": [ 06:02:26 DEBUG { 06:02:26 DEBUG "metric": { 06:02:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:02:26 DEBUG }, 06:02:26 DEBUG "value": [ 06:02:26 DEBUG 06:02:26 DEBUG 06:02:26 DEBUG 06:02:26 DEBUG [...] 06:02:26 DEBUG 06:02:26 DEBUG 06:02:26 DEBUG 06:02:26 DEBUG "pod": "am-654c4fcbcf-487cv" 06:02:26 DEBUG }, 06:02:26 DEBUG "value": [ 06:02:26 DEBUG 1712725346, 06:02:26 DEBUG "0" 06:02:26 DEBUG ] 06:02:26 DEBUG } 06:02:26 DEBUG ] 06:02:26 DEBUG } 06:02:26 DEBUG } 06:02:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725347" 06:02:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725347" 06:02:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725347" 06:02:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725347" 06:02:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725347" 06:02:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725347" 06:02:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725347" 06:02:27 INFO [http_cmd]: http status code OK 06:02:27 DEBUG --- status code --- 06:02:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:27 DEBUG --- http response --- 06:02:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:02:27 INFO [http_cmd]: http status code OK 06:02:27 DEBUG ----- output ----- 06:02:27 INFO [http_cmd]: http status code OK 06:02:27 DEBUG --- status code --- 06:02:27 INFO [http_cmd]: http status code OK 06:02:27 DEBUG { 06:02:27 DEBUG --- status code --- 06:02:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:27 DEBUG --- http response --- 06:02:27 DEBUG "status": "success", 06:02:27 DEBUG --- status code --- 06:02:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:27 DEBUG --- http response --- 06:02:27 DEBUG "data": { 06:02:27 DEBUG "resultType": "vector", 06:02:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:02:27 INFO [http_cmd]: http status code OK 06:02:27 INFO [http_cmd]: http status code OK 06:02:27 DEBUG --- status code --- 06:02:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:27 DEBUG --- http response --- 06:02:27 DEBUG ----- output ----- 06:02:27 DEBUG --- status code --- 06:02:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:02:27 DEBUG ----- output ----- 06:02:27 DEBUG { 06:02:27 DEBUG "status": "success", 06:02:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:27 DEBUG --- http response --- 06:02:27 DEBUG "result": [ 06:02:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:02:27 DEBUG { 06:02:27 DEBUG "status": "success", 06:02:27 DEBUG --- http response --- 06:02:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:02:27 DEBUG { 06:02:27 DEBUG ----- output ----- 06:02:27 DEBUG { 06:02:27 DEBUG "data": { 06:02:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:02:27 DEBUG ----- output ----- 06:02:27 DEBUG { 06:02:27 DEBUG "status": "success", 06:02:27 DEBUG "data": { 06:02:27 DEBUG "resultType": "vector", 06:02:27 DEBUG "result": [ 06:02:27 DEBUG "metric": { 06:02:27 DEBUG "status": "success", 06:02:27 DEBUG "data": { 06:02:27 DEBUG "resultType": "vector", 06:02:27 DEBUG "result": [ 06:02:27 DEBUG { 06:02:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:02:27 DEBUG "metric": { 06:02:27 DEBUG "instance": "10.142.0.25:9100" 06:02:27 DEBUG }, 06:02:27 DEBUG "value": [ 06:02:27 DEBUG "resultType": "vector", 06:02:27 DEBUG "result": [ 06:02:27 DEBUG ----- output ----- 06:02:27 DEBUG { 06:02:27 DEBUG "status": "success", 06:02:27 DEBUG "data": { 06:02:27 DEBUG "resultType": "vector", 06:02:27 DEBUG "result": [ 06:02:27 DEBUG { 06:02:27 DEBUG "metric": { 06:02:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:02:27 DEBUG }, 06:02:27 DEBUG "value": [ 06:02:27 DEBUG "data": { 06:02:27 DEBUG "resultType": "vector", 06:02:27 DEBUG "result": [ 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG [...] 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG { 06:02:27 DEBUG "metric": { 06:02:27 DEBUG "pod": "ds-idrepo-0" 06:02:27 DEBUG }, 06:02:27 DEBUG "value": [ 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 INFO [http_cmd]: http status code OK 06:02:27 DEBUG --- status code --- 06:02:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:27 DEBUG --- http response --- 06:02:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 06:02:27 DEBUG ----- output ----- 06:02:27 DEBUG { 06:02:27 DEBUG "status": "success", 06:02:27 DEBUG "data": { 06:02:27 DEBUG "resultType": "vector", 06:02:27 DEBUG "result": [ 06:02:27 DEBUG { 06:02:27 DEBUG "metric": { 06:02:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:02:27 DEBUG }, 06:02:27 DEBUG "value": [ 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG [...] 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG { 06:02:27 DEBUG "metric": { 06:02:27 DEBUG "instance": "10.142.0.25:9100" 06:02:27 DEBUG }, 06:02:27 DEBUG 06:02:27 DEBUG { 06:02:27 DEBUG "metric": { 06:02:27 DEBUG "pod": "ds-idrepo-0" 06:02:27 DEBUG }, 06:02:27 DEBUG "value": [ 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:02:27 DEBUG }, 06:02:27 DEBUG "value": [ 06:02:27 DEBUG 1712725347, 06:02:27 DEBUG "value": [ 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG [...] 06:02:27 DEBUG 06:02:27 DEBUG [...] 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG "0" 06:02:27 DEBUG ] 06:02:27 DEBUG } 06:02:27 DEBUG 06:02:27 DEBUG [...] 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG "pod": "ds-cts-2" 06:02:27 DEBUG }, 06:02:27 DEBUG "value": [ 06:02:27 DEBUG 1712725347, 06:02:27 DEBUG "0" 06:02:27 DEBUG ] 06:02:27 DEBUG } 06:02:27 DEBUG ] 06:02:27 DEBUG } 06:02:27 DEBUG } 06:02:27 DEBUG 06:02:27 DEBUG "instance": "10.142.0.77:9100" 06:02:27 DEBUG }, 06:02:27 DEBUG "value": [ 06:02:27 DEBUG 1712725347, 06:02:27 DEBUG }, 06:02:27 DEBUG 06:02:27 DEBUG [...] 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG "value": [ 06:02:27 DEBUG "pod": "ds-cts-2" 06:02:27 DEBUG }, 06:02:27 DEBUG "value": [ 06:02:27 DEBUG 1712725347, 06:02:27 DEBUG "29.863680298636805" 06:02:27 DEBUG 06:02:27 DEBUG ] 06:02:27 DEBUG } 06:02:27 DEBUG ] 06:02:27 DEBUG } 06:02:27 DEBUG } 06:02:27 DEBUG 06:02:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:02:27 DEBUG }, 06:02:27 DEBUG "value": [ 06:02:27 DEBUG 1712725347, 06:02:27 DEBUG 06:02:27 DEBUG "259713.5" 06:02:27 DEBUG ] 06:02:27 DEBUG } 06:02:27 DEBUG ] 06:02:27 DEBUG [...] 06:02:27 DEBUG } 06:02:27 DEBUG ] 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG } 06:02:27 DEBUG } 06:02:27 DEBUG } 06:02:27 DEBUG "344518808576" 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG "instance": "10.142.0.77:9100" 06:02:27 DEBUG 06:02:27 DEBUG ] 06:02:27 DEBUG } 06:02:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:02:27 DEBUG }, 06:02:27 DEBUG "value": [ 06:02:27 DEBUG }, 06:02:27 DEBUG "value": [ 06:02:27 DEBUG 1712725347, 06:02:27 DEBUG "42539776.86666667" 06:02:27 DEBUG ] 06:02:27 DEBUG ] 06:02:27 DEBUG } 06:02:27 DEBUG ] 06:02:27 DEBUG } 06:02:27 DEBUG } 06:02:27 DEBUG } 06:02:27 DEBUG 1712725347, 06:02:27 DEBUG "1442.3333333333333" 06:02:27 DEBUG ] 06:02:27 DEBUG } 06:02:27 DEBUG ] 06:02:27 DEBUG } 06:02:27 DEBUG } 06:02:27 DEBUG } 06:02:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725347" 06:02:27 INFO [http_cmd]: http status code OK 06:02:27 DEBUG --- status code --- 06:02:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:27 DEBUG --- http response --- 06:02:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:02:27 DEBUG ----- output ----- 06:02:27 DEBUG { 06:02:27 DEBUG "status": "success", 06:02:27 DEBUG "data": { 06:02:27 DEBUG "resultType": "vector", 06:02:27 DEBUG "result": [ 06:02:27 DEBUG { 06:02:27 DEBUG "metric": { 06:02:27 DEBUG "pod": "ds-idrepo-0" 06:02:27 DEBUG }, 06:02:27 DEBUG "value": [ 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG [...] 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG "pod": "ds-idrepo-1" 06:02:27 DEBUG }, 06:02:27 DEBUG "value": [ 06:02:27 DEBUG 1712725347, 06:02:27 DEBUG "0" 06:02:27 DEBUG ] 06:02:27 DEBUG } 06:02:27 DEBUG ] 06:02:27 DEBUG } 06:02:27 DEBUG } 06:02:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725347" 06:02:27 INFO [http_cmd]: http status code OK 06:02:27 DEBUG --- status code --- 06:02:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:27 DEBUG --- http response --- 06:02:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:02:27 DEBUG ----- output ----- 06:02:27 DEBUG { 06:02:27 DEBUG "status": "success", 06:02:27 DEBUG "data": { 06:02:27 DEBUG "resultType": "vector", 06:02:27 DEBUG "result": [ 06:02:27 DEBUG { 06:02:27 DEBUG "metric": { 06:02:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:02:27 DEBUG }, 06:02:27 DEBUG "value": [ 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG [...] 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:02:27 DEBUG }, 06:02:27 DEBUG "value": [ 06:02:27 DEBUG 1712725347, 06:02:27 DEBUG "1429.0666666666666" 06:02:27 DEBUG ] 06:02:27 DEBUG } 06:02:27 DEBUG ] 06:02:27 DEBUG } 06:02:27 DEBUG } 06:02:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725347" 06:02:27 INFO [http_cmd]: http status code OK 06:02:27 DEBUG --- status code --- 06:02:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:27 DEBUG --- http response --- 06:02:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:02:27 DEBUG ----- output ----- 06:02:27 DEBUG { 06:02:27 DEBUG "status": "success", 06:02:27 DEBUG "data": { 06:02:27 DEBUG "resultType": "vector", 06:02:27 DEBUG "result": [ 06:02:27 DEBUG { 06:02:27 DEBUG "metric": { 06:02:27 DEBUG "instance": "10.142.0.25:9100" 06:02:27 DEBUG }, 06:02:27 DEBUG "value": [ 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG [...] 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG 06:02:27 DEBUG "instance": "10.142.0.77:9100" 06:02:27 DEBUG }, 06:02:27 DEBUG "value": [ 06:02:27 DEBUG 1712725347, 06:02:27 DEBUG "0.16313333333333233" 06:02:27 DEBUG ] 06:02:27 DEBUG } 06:02:27 DEBUG ] 06:02:27 DEBUG } 06:02:27 DEBUG } 06:02:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725348" 06:02:28 INFO [http_cmd]: http status code OK 06:02:28 DEBUG --- status code --- 06:02:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:28 DEBUG --- http response --- 06:02:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:02:28 DEBUG ----- output ----- 06:02:28 DEBUG { 06:02:28 DEBUG "status": "success", 06:02:28 DEBUG "data": { 06:02:28 DEBUG "resultType": "vector", 06:02:28 DEBUG "result": [ 06:02:28 DEBUG { 06:02:28 DEBUG "metric": { 06:02:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:02:28 DEBUG }, 06:02:28 DEBUG "value": [ 06:02:28 DEBUG 06:02:28 DEBUG 06:02:28 DEBUG 06:02:28 DEBUG [...] 06:02:28 DEBUG 06:02:28 DEBUG 06:02:28 DEBUG 06:02:28 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:02:28 DEBUG }, 06:02:28 DEBUG "value": [ 06:02:28 DEBUG 1712725348, 06:02:28 DEBUG "0" 06:02:28 DEBUG ] 06:02:28 DEBUG } 06:02:28 DEBUG ] 06:02:28 DEBUG } 06:02:28 DEBUG } 06:02:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712725348" 06:02:28 INFO [http_cmd]: http status code OK 06:02:28 DEBUG --- status code --- 06:02:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:28 DEBUG --- http response --- 06:02:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:02:28 DEBUG ----- output ----- 06:02:28 DEBUG { 06:02:28 DEBUG "status": "success", 06:02:28 DEBUG "data": { 06:02:28 DEBUG "resultType": "vector", 06:02:28 DEBUG "result": [ 06:02:28 DEBUG { 06:02:28 DEBUG "metric": { 06:02:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:02:28 DEBUG }, 06:02:28 DEBUG "value": [ 06:02:28 DEBUG 06:02:28 DEBUG 06:02:28 DEBUG 06:02:28 DEBUG [...] 06:02:28 DEBUG 06:02:28 DEBUG 06:02:28 DEBUG 06:02:28 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:02:28 DEBUG }, 06:02:28 DEBUG "value": [ 06:02:28 DEBUG 1712725348, 06:02:28 DEBUG "4.141842662712871" 06:02:28 DEBUG ] 06:02:28 DEBUG } 06:02:28 DEBUG ] 06:02:28 DEBUG } 06:02:28 DEBUG } 06:02:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712725348" 06:02:28 INFO [http_cmd]: http status code OK 06:02:28 DEBUG --- status code --- 06:02:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:28 DEBUG --- http response --- 06:02:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:02:28 DEBUG ----- output ----- 06:02:28 DEBUG { 06:02:28 DEBUG "status": "success", 06:02:28 DEBUG "data": { 06:02:28 DEBUG "resultType": "vector", 06:02:28 DEBUG "result": [ 06:02:28 DEBUG { 06:02:28 DEBUG "metric": { 06:02:28 DEBUG "pod": "ds-idrepo-0" 06:02:28 DEBUG }, 06:02:28 DEBUG "value": [ 06:02:28 DEBUG 06:02:28 DEBUG 06:02:28 DEBUG 06:02:28 DEBUG [...] 06:02:28 DEBUG 06:02:28 DEBUG 06:02:28 DEBUG 06:02:28 DEBUG "pod": "ds-cts-2" 06:02:28 DEBUG }, 06:02:28 DEBUG "value": [ 06:02:28 DEBUG 1712725348, 06:02:28 DEBUG "0" 06:02:28 DEBUG ] 06:02:28 DEBUG } 06:02:28 DEBUG ] 06:02:28 DEBUG } 06:02:28 DEBUG } 06:02:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725349" 06:02:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725349" 06:02:29 INFO [http_cmd]: http status code OK 06:02:29 DEBUG --- status code --- 06:02:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:29 DEBUG --- http response --- 06:02:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:02:29 DEBUG ----- output ----- 06:02:29 DEBUG { 06:02:29 DEBUG "status": "success", 06:02:29 DEBUG "data": { 06:02:29 DEBUG "resultType": "vector", 06:02:29 DEBUG "result": [ 06:02:29 DEBUG { 06:02:29 DEBUG "metric": { 06:02:29 DEBUG "pod": "ds-idrepo-2" 06:02:29 DEBUG }, 06:02:29 DEBUG "value": [ 06:02:29 DEBUG 06:02:29 DEBUG 06:02:29 INFO [http_cmd]: http status code OK 06:02:29 DEBUG 06:02:29 DEBUG --- status code --- 06:02:29 DEBUG [...] 06:02:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:29 DEBUG 06:02:29 DEBUG --- http response --- 06:02:29 DEBUG 06:02:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:02:29 DEBUG 06:02:29 DEBUG ----- output ----- 06:02:29 DEBUG "pod": "ds-cts-2" 06:02:29 DEBUG { 06:02:29 DEBUG }, 06:02:29 DEBUG "status": "success", 06:02:29 DEBUG "value": [ 06:02:29 DEBUG "data": { 06:02:29 DEBUG 1712725349, 06:02:29 DEBUG "resultType": "vector", 06:02:29 DEBUG "0" 06:02:29 DEBUG ] 06:02:29 DEBUG "result": [ 06:02:29 DEBUG } 06:02:29 DEBUG { 06:02:29 DEBUG ] 06:02:29 DEBUG "metric": { 06:02:29 DEBUG } 06:02:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:02:29 DEBUG } 06:02:29 DEBUG }, 06:02:29 DEBUG "value": [ 06:02:29 DEBUG 06:02:29 DEBUG 06:02:29 DEBUG 06:02:29 DEBUG [...] 06:02:29 DEBUG 06:02:29 DEBUG 06:02:29 DEBUG 06:02:29 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:02:29 DEBUG }, 06:02:29 DEBUG "value": [ 06:02:29 DEBUG 1712725349, 06:02:29 DEBUG "0" 06:02:29 DEBUG ] 06:02:29 DEBUG } 06:02:29 DEBUG ] 06:02:29 DEBUG } 06:02:29 DEBUG } 06:02:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725349" 06:02:29 INFO [http_cmd]: http status code OK 06:02:29 DEBUG --- status code --- 06:02:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:29 DEBUG --- http response --- 06:02:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:02:29 DEBUG ----- output ----- 06:02:29 DEBUG { 06:02:29 DEBUG "status": "success", 06:02:29 DEBUG "data": { 06:02:29 DEBUG "resultType": "vector", 06:02:29 DEBUG "result": [ 06:02:29 DEBUG { 06:02:29 DEBUG "metric": { 06:02:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:02:29 DEBUG }, 06:02:29 DEBUG "value": [ 06:02:29 DEBUG 06:02:29 DEBUG 06:02:29 DEBUG 06:02:29 DEBUG [...] 06:02:29 DEBUG 06:02:29 DEBUG 06:02:29 DEBUG 06:02:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:02:29 DEBUG }, 06:02:29 DEBUG "value": [ 06:02:29 DEBUG 1712725349, 06:02:29 DEBUG "1428.8333333333333" 06:02:29 DEBUG ] 06:02:29 DEBUG } 06:02:29 DEBUG ] 06:02:29 DEBUG } 06:02:29 DEBUG } 06:02:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712725349" 06:02:29 INFO [http_cmd]: http status code OK 06:02:29 DEBUG --- status code --- 06:02:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:29 DEBUG --- http response --- 06:02:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:02:29 DEBUG ----- output ----- 06:02:29 DEBUG { 06:02:29 DEBUG "status": "success", 06:02:29 DEBUG "data": { 06:02:29 DEBUG "resultType": "vector", 06:02:29 DEBUG "result": [ 06:02:29 DEBUG { 06:02:29 DEBUG "metric": { 06:02:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:02:29 DEBUG }, 06:02:29 DEBUG "value": [ 06:02:29 DEBUG 06:02:29 DEBUG 06:02:29 DEBUG 06:02:29 DEBUG [...] 06:02:29 DEBUG 06:02:29 DEBUG 06:02:29 DEBUG 06:02:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:02:29 DEBUG }, 06:02:29 DEBUG "value": [ 06:02:29 DEBUG 1712725349, 06:02:29 DEBUG "0.048" 06:02:29 DEBUG ] 06:02:29 DEBUG } 06:02:29 DEBUG ] 06:02:29 DEBUG } 06:02:29 DEBUG } 06:02:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725349" 06:02:29 INFO [http_cmd]: http status code OK 06:02:29 DEBUG --- status code --- 06:02:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:29 DEBUG --- http response --- 06:02:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:02:29 DEBUG ----- output ----- 06:02:29 DEBUG { 06:02:29 DEBUG "status": "success", 06:02:29 DEBUG "data": { 06:02:29 DEBUG "resultType": "vector", 06:02:29 DEBUG "result": [ 06:02:29 DEBUG { 06:02:29 DEBUG "metric": { 06:02:29 DEBUG "pod": "ds-idrepo-0" 06:02:29 DEBUG }, 06:02:29 DEBUG "value": [ 06:02:29 DEBUG 06:02:29 DEBUG 06:02:29 DEBUG 06:02:29 DEBUG [...] 06:02:29 DEBUG 06:02:29 DEBUG 06:02:29 DEBUG 06:02:29 DEBUG "pod": "ds-idrepo-1" 06:02:29 DEBUG }, 06:02:29 DEBUG "value": [ 06:02:29 DEBUG 1712725349, 06:02:29 DEBUG "0" 06:02:29 DEBUG ] 06:02:29 DEBUG } 06:02:29 DEBUG ] 06:02:29 DEBUG } 06:02:29 DEBUG } 06:02:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725349" 06:02:29 INFO [http_cmd]: http status code OK 06:02:29 DEBUG --- status code --- 06:02:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:29 DEBUG --- http response --- 06:02:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:02:29 DEBUG ----- output ----- 06:02:29 DEBUG { 06:02:29 DEBUG "status": "success", 06:02:29 DEBUG "data": { 06:02:29 DEBUG "resultType": "vector", 06:02:29 DEBUG "result": [ 06:02:29 DEBUG { 06:02:29 DEBUG "metric": { 06:02:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:02:29 DEBUG }, 06:02:29 DEBUG "value": [ 06:02:29 DEBUG 06:02:29 DEBUG 06:02:29 DEBUG 06:02:29 DEBUG [...] 06:02:29 DEBUG 06:02:29 DEBUG 06:02:29 DEBUG 06:02:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:02:29 DEBUG }, 06:02:29 DEBUG "value": [ 06:02:29 DEBUG 1712725349, 06:02:29 DEBUG "1452.5666666666666" 06:02:29 DEBUG ] 06:02:29 DEBUG } 06:02:29 DEBUG ] 06:02:29 DEBUG } 06:02:29 DEBUG } 06:02:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725349" 06:02:29 INFO [http_cmd]: http status code OK 06:02:29 DEBUG --- status code --- 06:02:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:29 DEBUG --- http response --- 06:02:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:02:29 DEBUG ----- output ----- 06:02:29 DEBUG { 06:02:29 DEBUG "status": "success", 06:02:29 DEBUG "data": { 06:02:29 DEBUG "resultType": "vector", 06:02:29 DEBUG "result": [ 06:02:29 DEBUG { 06:02:29 DEBUG "metric": { 06:02:29 DEBUG "pod": "ds-idrepo-0" 06:02:29 DEBUG }, 06:02:29 DEBUG "value": [ 06:02:29 DEBUG 06:02:29 DEBUG 06:02:29 DEBUG 06:02:29 DEBUG [...] 06:02:29 DEBUG 06:02:29 DEBUG 06:02:29 DEBUG 06:02:29 DEBUG "pod": "ds-idrepo-1" 06:02:29 DEBUG }, 06:02:29 DEBUG "value": [ 06:02:29 DEBUG 1712725349, 06:02:29 DEBUG "0" 06:02:29 DEBUG ] 06:02:29 DEBUG } 06:02:29 DEBUG ] 06:02:29 DEBUG } 06:02:29 DEBUG } 06:02:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725350" 06:02:30 INFO [http_cmd]: http status code OK 06:02:30 DEBUG --- status code --- 06:02:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:30 DEBUG --- http response --- 06:02:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:02:30 DEBUG ----- output ----- 06:02:30 DEBUG { 06:02:30 DEBUG "status": "success", 06:02:30 DEBUG "data": { 06:02:30 DEBUG "resultType": "vector", 06:02:30 DEBUG "result": [ 06:02:30 DEBUG { 06:02:30 DEBUG "metric": { 06:02:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:02:30 DEBUG }, 06:02:30 DEBUG "value": [ 06:02:30 DEBUG 06:02:30 DEBUG 06:02:30 DEBUG 06:02:30 DEBUG [...] 06:02:30 DEBUG 06:02:30 DEBUG 06:02:30 DEBUG 06:02:30 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:02:30 DEBUG }, 06:02:30 DEBUG "value": [ 06:02:30 DEBUG 1712725350, 06:02:30 DEBUG "0" 06:02:30 DEBUG ] 06:02:30 DEBUG } 06:02:30 DEBUG ] 06:02:30 DEBUG } 06:02:30 DEBUG } 06:02:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712725350" 06:02:30 INFO [http_cmd]: http status code OK 06:02:30 DEBUG --- status code --- 06:02:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:30 DEBUG --- http response --- 06:02:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:02:30 DEBUG ----- output ----- 06:02:30 DEBUG { 06:02:30 DEBUG "status": "success", 06:02:30 DEBUG "data": { 06:02:30 DEBUG "resultType": "vector", 06:02:30 DEBUG "result": [ 06:02:30 DEBUG { 06:02:30 DEBUG "metric": { 06:02:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:02:30 DEBUG }, 06:02:30 DEBUG "value": [ 06:02:30 DEBUG 06:02:30 DEBUG 06:02:30 DEBUG 06:02:30 DEBUG [...] 06:02:30 DEBUG 06:02:30 DEBUG 06:02:30 DEBUG 06:02:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:02:30 DEBUG }, 06:02:30 DEBUG "value": [ 06:02:30 DEBUG 1712725350, 06:02:30 DEBUG "2488823808" 06:02:30 DEBUG ] 06:02:30 DEBUG } 06:02:30 DEBUG ] 06:02:30 DEBUG } 06:02:30 DEBUG } 06:02:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725350" 06:02:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725350" 06:02:30 INFO [http_cmd]: http status code OK 06:02:30 DEBUG --- status code --- 06:02:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:30 DEBUG --- http response --- 06:02:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:02:30 DEBUG ----- output ----- 06:02:30 DEBUG { 06:02:30 DEBUG "status": "success", 06:02:30 DEBUG "data": { 06:02:30 DEBUG "resultType": "vector", 06:02:30 DEBUG "result": [ 06:02:30 DEBUG { 06:02:30 DEBUG "metric": { 06:02:30 DEBUG "instance": "10.142.0.25:9100" 06:02:30 DEBUG }, 06:02:30 DEBUG "value": [ 06:02:30 DEBUG 06:02:30 DEBUG 06:02:30 DEBUG 06:02:30 DEBUG [...] 06:02:30 DEBUG 06:02:30 DEBUG 06:02:30 DEBUG 06:02:30 DEBUG "instance": "10.142.0.77:9100" 06:02:30 DEBUG }, 06:02:30 DEBUG "value": [ 06:02:30 DEBUG 1712725350, 06:02:30 DEBUG "42541712.93333333" 06:02:30 DEBUG ] 06:02:30 DEBUG } 06:02:30 DEBUG ] 06:02:30 DEBUG } 06:02:30 DEBUG } 06:02:30 INFO [http_cmd]: http status code OK 06:02:30 DEBUG --- status code --- 06:02:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:30 DEBUG --- http response --- 06:02:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:02:30 DEBUG ----- output ----- 06:02:30 DEBUG { 06:02:30 DEBUG "status": "success", 06:02:30 DEBUG "data": { 06:02:30 DEBUG "resultType": "vector", 06:02:30 DEBUG "result": [ 06:02:30 DEBUG { 06:02:30 DEBUG "metric": { 06:02:30 DEBUG "instance": "10.142.0.25:9100" 06:02:30 DEBUG }, 06:02:30 DEBUG "value": [ 06:02:30 DEBUG 06:02:30 DEBUG 06:02:30 DEBUG 06:02:30 DEBUG [...] 06:02:30 DEBUG 06:02:30 DEBUG 06:02:30 DEBUG 06:02:30 DEBUG "instance": "10.142.0.77:9100" 06:02:30 DEBUG }, 06:02:30 DEBUG "value": [ 06:02:30 DEBUG 1712725350, 06:02:30 DEBUG "0.050333333333333674" 06:02:30 DEBUG ] 06:02:30 DEBUG } 06:02:30 DEBUG ] 06:02:30 DEBUG } 06:02:30 DEBUG } 06:02:47 INFO 06:02:47 INFO [loop_until]: kubectl --namespace=xlou top pods 06:02:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:02:47 INFO [loop_until]: OK (rc = 0) 06:02:47 DEBUG --- stdout --- 06:02:47 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10228m 8978Mi am-654c4fcbcf-9scrj 9779m 8443Mi am-654c4fcbcf-v5qb5 8402m 8444Mi ds-cts-0 7970m 13380Mi ds-cts-1 8614m 12602Mi ds-cts-2 7860m 12826Mi ds-idrepo-0 10773m 5622Mi ds-idrepo-1 373m 5672Mi ds-idrepo-2 371m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1191Mi idm-6bfb7c96cc-vsb58 7m 2213Mi lodemon-b776b7d88-thrqf 8m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4140m 2235Mi 06:02:47 DEBUG --- stderr --- 06:02:47 DEBUG 06:02:54 INFO 06:02:54 INFO [loop_until]: kubectl --namespace=xlou top node 06:02:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:02:54 INFO [loop_until]: OK (rc = 0) 06:02:54 DEBUG --- stdout --- 06:02:54 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 97m 0% 2305Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9217m 58% 9310Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 80m 0% 3258Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13165m 82% 6513Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10907m 68% 9323Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11483m 72% 9793Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 77m 0% 858Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10857m 68% 6297Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8154m 51% 14053Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 8015m 50% 13533Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 432m 2% 6378Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8757m 55% 13351Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 420m 2% 6345Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4250m 26% 5883Mi 10% 06:02:54 DEBUG --- stderr --- 06:02:54 DEBUG 06:03:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725406" 06:03:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712725406" 06:03:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725406" 06:03:26 INFO [http_cmd]: http status code OK 06:03:26 DEBUG --- status code --- 06:03:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:26 DEBUG --- http response --- 06:03:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:03:26 DEBUG ----- output ----- 06:03:26 DEBUG { 06:03:26 DEBUG "status": "success", 06:03:26 DEBUG "data": { 06:03:26 DEBUG "resultType": "vector", 06:03:26 DEBUG "result": [ 06:03:26 DEBUG { 06:03:26 DEBUG "metric": { 06:03:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:03:26 DEBUG }, 06:03:26 DEBUG "value": [ 06:03:26 DEBUG 06:03:26 DEBUG 06:03:26 DEBUG 06:03:26 DEBUG [...] 06:03:26 DEBUG 06:03:26 DEBUG 06:03:26 DEBUG 06:03:26 DEBUG "pod": "am-654c4fcbcf-487cv" 06:03:26 DEBUG }, 06:03:26 DEBUG "value": [ 06:03:26 DEBUG 1712725406, 06:03:26 DEBUG "0" 06:03:26 DEBUG ] 06:03:26 DEBUG } 06:03:26 DEBUG ] 06:03:26 DEBUG } 06:03:26 DEBUG } 06:03:26 ERROR [http_cmd]: ERROR 06:03:26 ERROR --- status code --- 06:03:26 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:03:26 ERROR --- http response --- 06:03:26 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:03:26 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:03:26 INFO [http_cmd]: http status code OK 06:03:26 DEBUG --- status code --- 06:03:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:26 DEBUG --- http response --- 06:03:26 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:03:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725407" 06:03:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725407" 06:03:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725407" 06:03:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725407" 06:03:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725407" 06:03:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725407" 06:03:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725407" 06:03:27 INFO [http_cmd]: http status code OK 06:03:27 DEBUG --- status code --- 06:03:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:27 DEBUG --- http response --- 06:03:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:03:27 DEBUG ----- output ----- 06:03:27 DEBUG { 06:03:27 DEBUG "status": "success", 06:03:27 DEBUG "data": { 06:03:27 DEBUG "resultType": "vector", 06:03:27 DEBUG "result": [ 06:03:27 DEBUG { 06:03:27 DEBUG "metric": { 06:03:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:03:27 DEBUG }, 06:03:27 DEBUG "value": [ 06:03:27 DEBUG 06:03:27 DEBUG 06:03:27 DEBUG 06:03:27 DEBUG [...] 06:03:27 DEBUG 06:03:27 INFO [http_cmd]: http status code OK 06:03:27 INFO [http_cmd]: http status code OK 06:03:27 DEBUG 06:03:27 DEBUG --- status code --- 06:03:27 DEBUG --- status code --- 06:03:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:27 DEBUG --- http response --- 06:03:27 DEBUG 06:03:27 DEBUG --- http response --- 06:03:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:03:27 DEBUG ----- output ----- 06:03:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:03:27 DEBUG { 06:03:27 DEBUG "status": "success", 06:03:27 DEBUG "data": { 06:03:27 DEBUG ----- output ----- 06:03:27 DEBUG "resultType": "vector", 06:03:27 DEBUG "result": [ 06:03:27 DEBUG { 06:03:27 DEBUG { 06:03:27 DEBUG "metric": { 06:03:27 DEBUG "status": "success", 06:03:27 DEBUG "pod": "ds-idrepo-0" 06:03:27 DEBUG }, 06:03:27 DEBUG "data": { 06:03:27 DEBUG "value": [ 06:03:27 DEBUG 06:03:27 DEBUG "resultType": "vector", 06:03:27 DEBUG 06:03:27 DEBUG 06:03:27 DEBUG "result": [ 06:03:27 DEBUG [...] 06:03:27 DEBUG 06:03:27 DEBUG { 06:03:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:03:27 DEBUG 06:03:27 DEBUG 06:03:27 INFO [http_cmd]: http status code OK 06:03:27 DEBUG }, 06:03:27 DEBUG "metric": { 06:03:27 DEBUG "pod": "ds-idrepo-0" 06:03:27 DEBUG }, 06:03:27 DEBUG "value": [ 06:03:27 DEBUG --- status code --- 06:03:27 DEBUG "value": [ 06:03:27 INFO [http_cmd]: http status code OK 06:03:27 INFO [http_cmd]: http status code OK 06:03:27 DEBUG 06:03:27 DEBUG "pod": "ds-cts-2" 06:03:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:27 DEBUG 1712725407, 06:03:27 DEBUG --- status code --- 06:03:27 DEBUG --- status code --- 06:03:27 DEBUG 06:03:27 DEBUG 06:03:27 INFO [http_cmd]: http status code OK 06:03:27 DEBUG --- http response --- 06:03:27 DEBUG "259745" 06:03:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:27 DEBUG [...] 06:03:27 DEBUG }, 06:03:27 DEBUG --- status code --- 06:03:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 06:03:27 DEBUG ] 06:03:27 DEBUG --- http response --- 06:03:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:03:27 DEBUG ----- output ----- 06:03:27 DEBUG "value": [ 06:03:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:27 DEBUG ----- output ----- 06:03:27 DEBUG } 06:03:27 DEBUG 06:03:27 DEBUG { 06:03:27 DEBUG --- http response --- 06:03:27 DEBUG 1712725407, 06:03:27 DEBUG --- http response --- 06:03:27 DEBUG { 06:03:27 DEBUG ] 06:03:27 DEBUG 06:03:27 DEBUG "status": "success", 06:03:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:03:27 DEBUG "29.663700296637003" 06:03:27 DEBUG "status": "success", 06:03:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:03:27 DEBUG } 06:03:27 DEBUG 06:03:27 DEBUG "data": { 06:03:27 DEBUG ----- output ----- 06:03:27 DEBUG ] 06:03:27 DEBUG "data": { 06:03:27 DEBUG ----- output ----- 06:03:27 DEBUG } 06:03:27 DEBUG "pod": "ds-cts-2" 06:03:27 DEBUG "resultType": "vector", 06:03:27 DEBUG { 06:03:27 DEBUG } 06:03:27 DEBUG "resultType": "vector", 06:03:27 DEBUG "result": [ 06:03:27 DEBUG { 06:03:27 DEBUG "status": "success", 06:03:27 DEBUG "data": { 06:03:27 DEBUG ] 06:03:27 DEBUG { 06:03:27 DEBUG "metric": { 06:03:27 DEBUG "status": "success", 06:03:27 DEBUG "data": { 06:03:27 DEBUG "resultType": "vector", 06:03:27 DEBUG } 06:03:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:03:27 DEBUG }, 06:03:27 DEBUG "result": [ 06:03:27 DEBUG { 06:03:27 DEBUG "metric": { 06:03:27 DEBUG } 06:03:27 DEBUG "value": [ 06:03:27 DEBUG 06:03:27 DEBUG "instance": "10.142.0.25:9100" 06:03:27 DEBUG }, 06:03:27 DEBUG "result": [ 06:03:27 DEBUG { 06:03:27 DEBUG }, 06:03:27 DEBUG "resultType": "vector", 06:03:27 DEBUG "result": [ 06:03:27 DEBUG "metric": { 06:03:27 DEBUG "instance": "10.142.0.25:9100" 06:03:27 DEBUG "value": [ 06:03:27 DEBUG { 06:03:27 DEBUG "metric": { 06:03:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:03:27 DEBUG 06:03:27 DEBUG }, 06:03:27 DEBUG }, 06:03:27 DEBUG "value": [ 06:03:27 DEBUG 06:03:27 DEBUG [...] 06:03:27 DEBUG 06:03:27 DEBUG 06:03:27 DEBUG 06:03:27 DEBUG 06:03:27 DEBUG 06:03:27 DEBUG 1712725407, 06:03:27 DEBUG [...] 06:03:27 DEBUG "value": [ 06:03:27 DEBUG "value": [ 06:03:27 DEBUG 06:03:27 DEBUG "0" 06:03:27 DEBUG 06:03:27 DEBUG 06:03:27 DEBUG 06:03:27 DEBUG 06:03:27 DEBUG ] 06:03:27 DEBUG 06:03:27 DEBUG 06:03:27 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:03:27 DEBUG 06:03:27 DEBUG } 06:03:27 DEBUG 06:03:27 DEBUG 06:03:27 DEBUG }, 06:03:27 DEBUG [...] 06:03:27 DEBUG ] 06:03:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:03:27 DEBUG [...] 06:03:27 DEBUG "value": [ 06:03:27 DEBUG 06:03:27 DEBUG } 06:03:27 DEBUG }, 06:03:27 DEBUG 06:03:27 DEBUG 1712725407, 06:03:27 DEBUG "0" 06:03:27 DEBUG } 06:03:27 DEBUG "value": [ 06:03:27 DEBUG 1712725407, 06:03:27 DEBUG 06:03:27 DEBUG ] 06:03:27 DEBUG } 06:03:27 DEBUG 06:03:27 DEBUG 06:03:27 DEBUG "1449.6" 06:03:27 DEBUG ] 06:03:27 DEBUG 06:03:27 DEBUG "instance": "10.142.0.77:9100" 06:03:27 DEBUG ] 06:03:27 DEBUG } 06:03:27 DEBUG "instance": "10.142.0.77:9100" 06:03:27 DEBUG }, 06:03:27 DEBUG } 06:03:27 DEBUG } 06:03:27 DEBUG }, 06:03:27 DEBUG "value": [ 06:03:27 DEBUG ] 06:03:27 DEBUG "value": [ 06:03:27 DEBUG 1712725407, 06:03:27 DEBUG } 06:03:27 DEBUG "42567083.86666667" 06:03:27 DEBUG 1712725407, 06:03:27 DEBUG } 06:03:27 DEBUG ] 06:03:27 DEBUG "346530173952" 06:03:27 DEBUG } 06:03:27 DEBUG ] 06:03:27 DEBUG ] 06:03:27 DEBUG } 06:03:27 DEBUG } 06:03:27 DEBUG ] 06:03:27 DEBUG } 06:03:27 DEBUG } 06:03:27 DEBUG } 06:03:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725407" 06:03:27 INFO [http_cmd]: http status code OK 06:03:27 DEBUG --- status code --- 06:03:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:27 DEBUG --- http response --- 06:03:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:03:27 DEBUG ----- output ----- 06:03:27 DEBUG { 06:03:27 DEBUG "status": "success", 06:03:27 DEBUG "data": { 06:03:27 DEBUG "resultType": "vector", 06:03:27 DEBUG "result": [ 06:03:27 DEBUG { 06:03:27 DEBUG "metric": { 06:03:27 DEBUG "pod": "ds-idrepo-0" 06:03:27 DEBUG }, 06:03:27 DEBUG "value": [ 06:03:27 DEBUG 06:03:27 DEBUG 06:03:27 DEBUG 06:03:27 DEBUG [...] 06:03:27 DEBUG 06:03:27 DEBUG 06:03:27 DEBUG 06:03:27 DEBUG "pod": "ds-idrepo-1" 06:03:27 DEBUG }, 06:03:27 DEBUG "value": [ 06:03:27 DEBUG 1712725407, 06:03:27 DEBUG "0" 06:03:27 DEBUG ] 06:03:27 DEBUG } 06:03:27 DEBUG ] 06:03:27 DEBUG } 06:03:27 DEBUG } 06:03:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725407" 06:03:27 INFO [http_cmd]: http status code OK 06:03:27 DEBUG --- status code --- 06:03:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:27 DEBUG --- http response --- 06:03:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:03:27 DEBUG ----- output ----- 06:03:27 DEBUG { 06:03:27 DEBUG "status": "success", 06:03:27 DEBUG "data": { 06:03:27 DEBUG "resultType": "vector", 06:03:27 DEBUG "result": [ 06:03:27 DEBUG { 06:03:27 DEBUG "metric": { 06:03:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:03:27 DEBUG }, 06:03:27 DEBUG "value": [ 06:03:27 DEBUG 06:03:27 DEBUG 06:03:27 DEBUG 06:03:27 DEBUG [...] 06:03:27 DEBUG 06:03:27 DEBUG 06:03:27 DEBUG 06:03:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:03:27 DEBUG }, 06:03:27 DEBUG "value": [ 06:03:27 DEBUG 1712725407, 06:03:27 DEBUG "1455.8666666666666" 06:03:27 DEBUG ] 06:03:27 DEBUG } 06:03:27 DEBUG ] 06:03:27 DEBUG } 06:03:27 DEBUG } 06:03:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725407" 06:03:27 INFO [http_cmd]: http status code OK 06:03:27 DEBUG --- status code --- 06:03:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:27 DEBUG --- http response --- 06:03:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:03:27 DEBUG ----- output ----- 06:03:27 DEBUG { 06:03:27 DEBUG "status": "success", 06:03:27 DEBUG "data": { 06:03:27 DEBUG "resultType": "vector", 06:03:27 DEBUG "result": [ 06:03:27 DEBUG { 06:03:27 DEBUG "metric": { 06:03:27 DEBUG "instance": "10.142.0.25:9100" 06:03:27 DEBUG }, 06:03:27 DEBUG "value": [ 06:03:27 DEBUG 06:03:27 DEBUG 06:03:27 DEBUG 06:03:27 DEBUG [...] 06:03:27 DEBUG 06:03:27 DEBUG 06:03:27 DEBUG 06:03:27 DEBUG "instance": "10.142.0.77:9100" 06:03:27 DEBUG }, 06:03:27 DEBUG "value": [ 06:03:27 DEBUG 1712725407, 06:03:27 DEBUG "0.04313333333333288" 06:03:27 DEBUG ] 06:03:27 DEBUG } 06:03:27 DEBUG ] 06:03:27 DEBUG } 06:03:27 DEBUG } 06:03:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725408" 06:03:28 INFO [http_cmd]: http status code OK 06:03:28 DEBUG --- status code --- 06:03:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:28 DEBUG --- http response --- 06:03:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:03:28 DEBUG ----- output ----- 06:03:28 DEBUG { 06:03:28 DEBUG "status": "success", 06:03:28 DEBUG "data": { 06:03:28 DEBUG "resultType": "vector", 06:03:28 DEBUG "result": [ 06:03:28 DEBUG { 06:03:28 DEBUG "metric": { 06:03:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:03:28 DEBUG }, 06:03:28 DEBUG "value": [ 06:03:28 DEBUG 06:03:28 DEBUG 06:03:28 DEBUG 06:03:28 DEBUG [...] 06:03:28 DEBUG 06:03:28 DEBUG 06:03:28 DEBUG 06:03:28 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:03:28 DEBUG }, 06:03:28 DEBUG "value": [ 06:03:28 DEBUG 1712725408, 06:03:28 DEBUG "0" 06:03:28 DEBUG ] 06:03:28 DEBUG } 06:03:28 DEBUG ] 06:03:28 DEBUG } 06:03:28 DEBUG } 06:03:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712725408" 06:03:28 INFO [http_cmd]: http status code OK 06:03:28 DEBUG --- status code --- 06:03:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:28 DEBUG --- http response --- 06:03:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:03:28 DEBUG ----- output ----- 06:03:28 DEBUG { 06:03:28 DEBUG "status": "success", 06:03:28 DEBUG "data": { 06:03:28 DEBUG "resultType": "vector", 06:03:28 DEBUG "result": [ 06:03:28 DEBUG { 06:03:28 DEBUG "metric": { 06:03:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:03:28 DEBUG }, 06:03:28 DEBUG "value": [ 06:03:28 DEBUG 06:03:28 DEBUG 06:03:28 DEBUG 06:03:28 DEBUG [...] 06:03:28 DEBUG 06:03:28 DEBUG 06:03:28 DEBUG 06:03:28 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:03:28 DEBUG }, 06:03:28 DEBUG "value": [ 06:03:28 DEBUG 1712725408, 06:03:28 DEBUG "4.1388565350664415" 06:03:28 DEBUG ] 06:03:28 DEBUG } 06:03:28 DEBUG ] 06:03:28 DEBUG } 06:03:28 DEBUG } 06:03:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712725408" 06:03:28 INFO [http_cmd]: http status code OK 06:03:28 DEBUG --- status code --- 06:03:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:28 DEBUG --- http response --- 06:03:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:03:28 DEBUG ----- output ----- 06:03:28 DEBUG { 06:03:28 DEBUG "status": "success", 06:03:28 DEBUG "data": { 06:03:28 DEBUG "resultType": "vector", 06:03:28 DEBUG "result": [ 06:03:28 DEBUG { 06:03:28 DEBUG "metric": { 06:03:28 DEBUG "pod": "ds-idrepo-0" 06:03:28 DEBUG }, 06:03:28 DEBUG "value": [ 06:03:28 DEBUG 06:03:28 DEBUG 06:03:28 DEBUG 06:03:28 DEBUG [...] 06:03:28 DEBUG 06:03:28 DEBUG 06:03:28 DEBUG 06:03:28 DEBUG "pod": "ds-cts-2" 06:03:28 DEBUG }, 06:03:28 DEBUG "value": [ 06:03:28 DEBUG 1712725408, 06:03:28 DEBUG "0" 06:03:28 DEBUG ] 06:03:28 DEBUG } 06:03:28 DEBUG ] 06:03:28 DEBUG } 06:03:28 DEBUG } 06:03:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725409" 06:03:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725409" 06:03:29 INFO [http_cmd]: http status code OK 06:03:29 INFO [http_cmd]: http status code OK 06:03:29 DEBUG --- status code --- 06:03:29 DEBUG --- status code --- 06:03:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:29 DEBUG --- http response --- 06:03:29 DEBUG --- http response --- 06:03:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:03:29 DEBUG ----- output ----- 06:03:29 DEBUG { 06:03:29 DEBUG "status": "success", 06:03:29 DEBUG "data": { 06:03:29 DEBUG "resultType": "vector", 06:03:29 DEBUG "result": [ 06:03:29 DEBUG { 06:03:29 DEBUG "metric": { 06:03:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:03:29 DEBUG }, 06:03:29 DEBUG "value": [ 06:03:29 DEBUG 06:03:29 DEBUG 06:03:29 DEBUG 06:03:29 DEBUG [...] 06:03:29 DEBUG 06:03:29 DEBUG 06:03:29 DEBUG 06:03:29 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:03:29 DEBUG }, 06:03:29 DEBUG "value": [ 06:03:29 DEBUG 1712725409, 06:03:29 DEBUG "0.003619899575584924" 06:03:29 DEBUG ] 06:03:29 DEBUG } 06:03:29 DEBUG ] 06:03:29 DEBUG } 06:03:29 DEBUG } 06:03:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:03:29 DEBUG ----- output ----- 06:03:29 DEBUG { 06:03:29 DEBUG "status": "success", 06:03:29 DEBUG "data": { 06:03:29 DEBUG "resultType": "vector", 06:03:29 DEBUG "result": [ 06:03:29 DEBUG { 06:03:29 DEBUG "metric": { 06:03:29 DEBUG "pod": "ds-idrepo-2" 06:03:29 DEBUG }, 06:03:29 DEBUG "value": [ 06:03:29 DEBUG 06:03:29 DEBUG 06:03:29 DEBUG 06:03:29 DEBUG [...] 06:03:29 DEBUG 06:03:29 DEBUG 06:03:29 DEBUG 06:03:29 DEBUG "pod": "ds-cts-2" 06:03:29 DEBUG }, 06:03:29 DEBUG "value": [ 06:03:29 DEBUG 1712725409, 06:03:29 DEBUG "0" 06:03:29 DEBUG ] 06:03:29 DEBUG } 06:03:29 DEBUG ] 06:03:29 DEBUG } 06:03:29 DEBUG } 06:03:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725409" 06:03:29 INFO [http_cmd]: http status code OK 06:03:29 DEBUG --- status code --- 06:03:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:29 DEBUG --- http response --- 06:03:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:03:29 DEBUG ----- output ----- 06:03:29 DEBUG { 06:03:29 DEBUG "status": "success", 06:03:29 DEBUG "data": { 06:03:29 DEBUG "resultType": "vector", 06:03:29 DEBUG "result": [ 06:03:29 DEBUG { 06:03:29 DEBUG "metric": { 06:03:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:03:29 DEBUG }, 06:03:29 DEBUG "value": [ 06:03:29 DEBUG 06:03:29 DEBUG 06:03:29 DEBUG 06:03:29 DEBUG [...] 06:03:29 DEBUG 06:03:29 DEBUG 06:03:29 DEBUG 06:03:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:03:29 DEBUG }, 06:03:29 DEBUG "value": [ 06:03:29 DEBUG 1712725409, 06:03:29 DEBUG "1455.7333333333333" 06:03:29 DEBUG ] 06:03:29 DEBUG } 06:03:29 DEBUG ] 06:03:29 DEBUG } 06:03:29 DEBUG } 06:03:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712725409" 06:03:29 INFO [http_cmd]: http status code OK 06:03:29 DEBUG --- status code --- 06:03:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:29 DEBUG --- http response --- 06:03:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:03:29 DEBUG ----- output ----- 06:03:29 DEBUG { 06:03:29 DEBUG "status": "success", 06:03:29 DEBUG "data": { 06:03:29 DEBUG "resultType": "vector", 06:03:29 DEBUG "result": [ 06:03:29 DEBUG { 06:03:29 DEBUG "metric": { 06:03:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:03:29 DEBUG }, 06:03:29 DEBUG "value": [ 06:03:29 DEBUG 06:03:29 DEBUG 06:03:29 DEBUG 06:03:29 DEBUG [...] 06:03:29 DEBUG 06:03:29 DEBUG 06:03:29 DEBUG 06:03:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:03:29 DEBUG }, 06:03:29 DEBUG "value": [ 06:03:29 DEBUG 1712725409, 06:03:29 DEBUG "0.048" 06:03:29 DEBUG ] 06:03:29 DEBUG } 06:03:29 DEBUG ] 06:03:29 DEBUG } 06:03:29 DEBUG } 06:03:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725409" 06:03:29 INFO [http_cmd]: http status code OK 06:03:29 DEBUG --- status code --- 06:03:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:29 DEBUG --- http response --- 06:03:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:03:29 DEBUG ----- output ----- 06:03:29 DEBUG { 06:03:29 DEBUG "status": "success", 06:03:29 DEBUG "data": { 06:03:29 DEBUG "resultType": "vector", 06:03:29 DEBUG "result": [ 06:03:29 DEBUG { 06:03:29 DEBUG "metric": { 06:03:29 DEBUG "pod": "ds-idrepo-0" 06:03:29 DEBUG }, 06:03:29 DEBUG "value": [ 06:03:29 DEBUG 06:03:29 DEBUG 06:03:29 DEBUG 06:03:29 DEBUG [...] 06:03:29 DEBUG 06:03:29 DEBUG 06:03:29 DEBUG 06:03:29 DEBUG "pod": "ds-idrepo-1" 06:03:29 DEBUG }, 06:03:29 DEBUG "value": [ 06:03:29 DEBUG 1712725409, 06:03:29 DEBUG "0" 06:03:29 DEBUG ] 06:03:29 DEBUG } 06:03:29 DEBUG ] 06:03:29 DEBUG } 06:03:29 DEBUG } 06:03:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725409" 06:03:29 INFO [http_cmd]: http status code OK 06:03:29 DEBUG --- status code --- 06:03:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:29 DEBUG --- http response --- 06:03:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:03:29 DEBUG ----- output ----- 06:03:29 DEBUG { 06:03:29 DEBUG "status": "success", 06:03:29 DEBUG "data": { 06:03:29 DEBUG "resultType": "vector", 06:03:29 DEBUG "result": [ 06:03:29 DEBUG { 06:03:29 DEBUG "metric": { 06:03:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:03:29 DEBUG }, 06:03:29 DEBUG "value": [ 06:03:29 DEBUG 06:03:29 DEBUG 06:03:29 DEBUG 06:03:29 DEBUG [...] 06:03:29 DEBUG 06:03:29 DEBUG 06:03:29 DEBUG 06:03:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:03:29 DEBUG }, 06:03:29 DEBUG "value": [ 06:03:29 DEBUG 1712725409, 06:03:29 DEBUG "1468.1" 06:03:29 DEBUG ] 06:03:29 DEBUG } 06:03:29 DEBUG ] 06:03:29 DEBUG } 06:03:29 DEBUG } 06:03:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725409" 06:03:29 INFO [http_cmd]: http status code OK 06:03:29 DEBUG --- status code --- 06:03:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:29 DEBUG --- http response --- 06:03:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:03:29 DEBUG ----- output ----- 06:03:29 DEBUG { 06:03:29 DEBUG "status": "success", 06:03:29 DEBUG "data": { 06:03:29 DEBUG "resultType": "vector", 06:03:29 DEBUG "result": [ 06:03:29 DEBUG { 06:03:29 DEBUG "metric": { 06:03:29 DEBUG "pod": "ds-idrepo-0" 06:03:29 DEBUG }, 06:03:29 DEBUG "value": [ 06:03:29 DEBUG 06:03:29 DEBUG 06:03:29 DEBUG 06:03:29 DEBUG [...] 06:03:29 DEBUG 06:03:29 DEBUG 06:03:29 DEBUG 06:03:29 DEBUG "pod": "ds-idrepo-1" 06:03:29 DEBUG }, 06:03:29 DEBUG "value": [ 06:03:29 DEBUG 1712725409, 06:03:29 DEBUG "0" 06:03:29 DEBUG ] 06:03:29 DEBUG } 06:03:29 DEBUG ] 06:03:29 DEBUG } 06:03:29 DEBUG } 06:03:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725410" 06:03:30 INFO [http_cmd]: http status code OK 06:03:30 DEBUG --- status code --- 06:03:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:30 DEBUG --- http response --- 06:03:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:03:30 DEBUG ----- output ----- 06:03:30 DEBUG { 06:03:30 DEBUG "status": "success", 06:03:30 DEBUG "data": { 06:03:30 DEBUG "resultType": "vector", 06:03:30 DEBUG "result": [ 06:03:30 DEBUG { 06:03:30 DEBUG "metric": { 06:03:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:03:30 DEBUG }, 06:03:30 DEBUG "value": [ 06:03:30 DEBUG 06:03:30 DEBUG 06:03:30 DEBUG 06:03:30 DEBUG [...] 06:03:30 DEBUG 06:03:30 DEBUG 06:03:30 DEBUG 06:03:30 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:03:30 DEBUG }, 06:03:30 DEBUG "value": [ 06:03:30 DEBUG 1712725410, 06:03:30 DEBUG "0" 06:03:30 DEBUG ] 06:03:30 DEBUG } 06:03:30 DEBUG ] 06:03:30 DEBUG } 06:03:30 DEBUG } 06:03:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712725410" 06:03:30 INFO [http_cmd]: http status code OK 06:03:30 DEBUG --- status code --- 06:03:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:30 DEBUG --- http response --- 06:03:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:03:30 DEBUG ----- output ----- 06:03:30 DEBUG { 06:03:30 DEBUG "status": "success", 06:03:30 DEBUG "data": { 06:03:30 DEBUG "resultType": "vector", 06:03:30 DEBUG "result": [ 06:03:30 DEBUG { 06:03:30 DEBUG "metric": { 06:03:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:03:30 DEBUG }, 06:03:30 DEBUG "value": [ 06:03:30 DEBUG 06:03:30 DEBUG 06:03:30 DEBUG 06:03:30 DEBUG [...] 06:03:30 DEBUG 06:03:30 DEBUG 06:03:30 DEBUG 06:03:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:03:30 DEBUG }, 06:03:30 DEBUG "value": [ 06:03:30 DEBUG 1712725410, 06:03:30 DEBUG "2488582144" 06:03:30 DEBUG ] 06:03:30 DEBUG } 06:03:30 DEBUG ] 06:03:30 DEBUG } 06:03:30 DEBUG } 06:03:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725410" 06:03:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725410" 06:03:30 INFO [http_cmd]: http status code OK 06:03:30 DEBUG --- status code --- 06:03:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:30 DEBUG --- http response --- 06:03:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:03:30 DEBUG ----- output ----- 06:03:30 DEBUG { 06:03:30 DEBUG "status": "success", 06:03:30 DEBUG "data": { 06:03:30 DEBUG "resultType": "vector", 06:03:30 DEBUG "result": [ 06:03:30 DEBUG { 06:03:30 DEBUG "metric": { 06:03:30 DEBUG "instance": "10.142.0.25:9100" 06:03:30 DEBUG }, 06:03:30 DEBUG "value": [ 06:03:30 DEBUG 06:03:30 DEBUG 06:03:30 DEBUG 06:03:30 DEBUG [...] 06:03:30 DEBUG 06:03:30 DEBUG 06:03:30 DEBUG 06:03:30 DEBUG "instance": "10.142.0.77:9100" 06:03:30 DEBUG }, 06:03:30 DEBUG "value": [ 06:03:30 DEBUG 1712725410, 06:03:30 DEBUG "42568891.900000006" 06:03:30 DEBUG ] 06:03:30 DEBUG } 06:03:30 DEBUG ] 06:03:30 DEBUG } 06:03:30 DEBUG } 06:03:30 INFO [http_cmd]: http status code OK 06:03:30 DEBUG --- status code --- 06:03:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:30 DEBUG --- http response --- 06:03:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:03:30 DEBUG ----- output ----- 06:03:30 DEBUG { 06:03:30 DEBUG "status": "success", 06:03:30 DEBUG "data": { 06:03:30 DEBUG "resultType": "vector", 06:03:30 DEBUG "result": [ 06:03:30 DEBUG { 06:03:30 DEBUG "metric": { 06:03:30 DEBUG "instance": "10.142.0.25:9100" 06:03:30 DEBUG }, 06:03:30 DEBUG "value": [ 06:03:30 DEBUG 06:03:30 DEBUG 06:03:30 DEBUG 06:03:30 DEBUG [...] 06:03:30 DEBUG 06:03:30 DEBUG 06:03:30 DEBUG 06:03:30 DEBUG "instance": "10.142.0.77:9100" 06:03:30 DEBUG }, 06:03:30 DEBUG "value": [ 06:03:30 DEBUG 1712725410, 06:03:30 DEBUG "0.019666666666666777" 06:03:30 DEBUG ] 06:03:30 DEBUG } 06:03:30 DEBUG ] 06:03:30 DEBUG } 06:03:30 DEBUG } 06:03:47 INFO 06:03:47 INFO [loop_until]: kubectl --namespace=xlou top pods 06:03:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:03:47 INFO [loop_until]: OK (rc = 0) 06:03:47 DEBUG --- stdout --- 06:03:47 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10429m 8966Mi am-654c4fcbcf-9scrj 9863m 8442Mi am-654c4fcbcf-v5qb5 8131m 8443Mi ds-cts-0 8458m 12962Mi ds-cts-1 8840m 13373Mi ds-cts-2 8153m 13394Mi ds-idrepo-0 10785m 5622Mi ds-idrepo-1 429m 5672Mi ds-idrepo-2 368m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 6m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4232m 2234Mi 06:03:47 DEBUG --- stderr --- 06:03:47 DEBUG 06:03:54 INFO 06:03:54 INFO [loop_until]: kubectl --namespace=xlou top node 06:03:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:03:54 INFO [loop_until]: OK (rc = 0) 06:03:54 DEBUG --- stdout --- 06:03:54 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 97m 0% 2304Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9069m 57% 9312Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 81m 0% 3257Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 12639m 79% 6521Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10468m 65% 9322Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11059m 69% 9788Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 79m 0% 856Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10845m 68% 6296Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8536m 53% 13655Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 7798m 49% 13104Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 428m 2% 6392Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8708m 54% 14062Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 469m 2% 6345Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4237m 26% 5885Mi 10% 06:03:54 DEBUG --- stderr --- 06:03:54 DEBUG 06:04:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725466" 06:04:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712725466" 06:04:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725466" 06:04:26 ERROR [http_cmd]: ERROR 06:04:26 INFO [http_cmd]: http status code OK 06:04:26 ERROR --- status code --- 06:04:26 INFO [http_cmd]: http status code OK 06:04:26 DEBUG --- status code --- 06:04:26 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:04:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:26 DEBUG --- http response --- 06:04:26 ERROR --- http response --- 06:04:26 DEBUG --- status code --- 06:04:26 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:04:26 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:04:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:26 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:04:26 DEBUG --- http response --- 06:04:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:04:26 DEBUG ----- output ----- 06:04:26 DEBUG { 06:04:26 DEBUG "status": "success", 06:04:26 DEBUG "data": { 06:04:26 DEBUG "resultType": "vector", 06:04:26 DEBUG "result": [ 06:04:26 DEBUG { 06:04:26 DEBUG "metric": { 06:04:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:04:26 DEBUG }, 06:04:26 DEBUG "value": [ 06:04:26 DEBUG 06:04:26 DEBUG 06:04:26 DEBUG 06:04:26 DEBUG [...] 06:04:26 DEBUG 06:04:26 DEBUG 06:04:26 DEBUG 06:04:26 DEBUG "pod": "am-654c4fcbcf-487cv" 06:04:26 DEBUG }, 06:04:26 DEBUG "value": [ 06:04:26 DEBUG 1712725466, 06:04:26 DEBUG "0" 06:04:26 DEBUG ] 06:04:26 DEBUG } 06:04:26 DEBUG ] 06:04:26 DEBUG } 06:04:26 DEBUG } 06:04:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725467" 06:04:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725467" 06:04:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725467" 06:04:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725467" 06:04:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725467" 06:04:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725467" 06:04:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725467" 06:04:27 INFO [http_cmd]: http status code OK 06:04:27 DEBUG --- status code --- 06:04:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:27 DEBUG --- http response --- 06:04:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:04:27 DEBUG ----- output ----- 06:04:27 DEBUG { 06:04:27 DEBUG "status": "success", 06:04:27 DEBUG "data": { 06:04:27 DEBUG "resultType": "vector", 06:04:27 DEBUG "result": [ 06:04:27 DEBUG { 06:04:27 DEBUG "metric": { 06:04:27 INFO [http_cmd]: http status code OK 06:04:27 DEBUG --- status code --- 06:04:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:27 DEBUG --- http response --- 06:04:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:04:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:04:27 DEBUG ----- output ----- 06:04:27 DEBUG { 06:04:27 DEBUG "status": "success", 06:04:27 DEBUG "data": { 06:04:27 DEBUG "resultType": "vector", 06:04:27 DEBUG "result": [ 06:04:27 DEBUG { 06:04:27 DEBUG "metric": { 06:04:27 DEBUG "pod": "ds-idrepo-0" 06:04:27 DEBUG }, 06:04:27 DEBUG "value": [ 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG [...] 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG "pod": "ds-cts-2" 06:04:27 DEBUG }, 06:04:27 DEBUG "value": [ 06:04:27 DEBUG 1712725467, 06:04:27 DEBUG }, 06:04:27 DEBUG "29.9" 06:04:27 DEBUG "value": [ 06:04:27 INFO [http_cmd]: http status code OK 06:04:27 DEBUG ] 06:04:27 DEBUG 06:04:27 DEBUG --- status code --- 06:04:27 DEBUG } 06:04:27 DEBUG 06:04:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:27 DEBUG ] 06:04:27 DEBUG } 06:04:27 DEBUG --- http response --- 06:04:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:04:27 DEBUG } 06:04:27 DEBUG 06:04:27 INFO [http_cmd]: http status code OK 06:04:27 DEBUG ----- output ----- 06:04:27 INFO [http_cmd]: http status code OK 06:04:27 INFO [http_cmd]: http status code OK 06:04:27 DEBUG --- status code --- 06:04:27 INFO [http_cmd]: http status code OK 06:04:27 DEBUG --- status code --- 06:04:27 DEBUG { 06:04:27 DEBUG --- status code --- 06:04:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:27 DEBUG [...] 06:04:27 DEBUG --- status code --- 06:04:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:27 DEBUG --- http response --- 06:04:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:27 DEBUG --- http response --- 06:04:27 DEBUG 06:04:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:27 DEBUG --- http response --- 06:04:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:04:27 DEBUG --- http response --- 06:04:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:04:27 DEBUG }, 06:04:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:04:27 DEBUG ----- output ----- 06:04:27 DEBUG { 06:04:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:04:27 DEBUG ----- output ----- 06:04:27 DEBUG { 06:04:27 DEBUG "status": "success", 06:04:27 DEBUG ----- output ----- 06:04:27 DEBUG { 06:04:27 DEBUG "status": "success", 06:04:27 DEBUG "data": { 06:04:27 DEBUG "resultType": "vector", 06:04:27 DEBUG "result": [ 06:04:27 DEBUG { 06:04:27 DEBUG "metric": { 06:04:27 DEBUG "data": { 06:04:27 DEBUG "status": "success", 06:04:27 DEBUG "data": { 06:04:27 DEBUG "instance": "10.142.0.25:9100" 06:04:27 DEBUG ----- output ----- 06:04:27 DEBUG "status": "success", 06:04:27 DEBUG "resultType": "vector", 06:04:27 DEBUG "result": [ 06:04:27 DEBUG { 06:04:27 DEBUG }, 06:04:27 DEBUG { 06:04:27 DEBUG "data": { 06:04:27 DEBUG "resultType": "vector", 06:04:27 DEBUG "result": [ 06:04:27 DEBUG { 06:04:27 DEBUG "value": [ 06:04:27 DEBUG "status": "success", 06:04:27 DEBUG "resultType": "vector", 06:04:27 DEBUG "metric": { 06:04:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:04:27 DEBUG }, 06:04:27 DEBUG 06:04:27 DEBUG "data": { 06:04:27 DEBUG "result": [ 06:04:27 DEBUG { 06:04:27 DEBUG "metric": { 06:04:27 DEBUG "instance": "10.142.0.25:9100" 06:04:27 DEBUG 06:04:27 DEBUG "resultType": "vector", 06:04:27 DEBUG "value": [ 06:04:27 DEBUG "metric": { 06:04:27 DEBUG }, 06:04:27 DEBUG "value": [ 06:04:27 DEBUG 06:04:27 DEBUG "result": [ 06:04:27 DEBUG 06:04:27 DEBUG "pod": "ds-idrepo-0" 06:04:27 DEBUG }, 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG [...] 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG "value": [ 06:04:27 DEBUG { 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG [...] 06:04:27 DEBUG "metric": { 06:04:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:04:27 DEBUG }, 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG [...] 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG "value": [ 06:04:27 DEBUG "value": [ 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG [...] 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG [...] 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG "instance": "10.142.0.77:9100" 06:04:27 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:04:27 DEBUG "instance": "10.142.0.77:9100" 06:04:27 DEBUG }, 06:04:27 DEBUG 06:04:27 DEBUG 1712725467, 06:04:27 DEBUG }, 06:04:27 DEBUG }, 06:04:27 DEBUG "value": [ 06:04:27 DEBUG "value": [ 06:04:27 DEBUG "pod": "ds-cts-2" 06:04:27 DEBUG "259817.5" 06:04:27 DEBUG "value": [ 06:04:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:04:27 DEBUG 1712725467, 06:04:27 DEBUG "0" 06:04:27 DEBUG ] 06:04:27 DEBUG } 06:04:27 DEBUG 1712725467, 06:04:27 DEBUG }, 06:04:27 DEBUG "value": [ 06:04:27 DEBUG 1712725467, 06:04:27 DEBUG "1450.4333333333334" 06:04:27 DEBUG ] 06:04:27 DEBUG } 06:04:27 DEBUG ] 06:04:27 DEBUG } 06:04:27 DEBUG } 06:04:27 DEBUG 1712725467, 06:04:27 DEBUG "348702182400" 06:04:27 DEBUG ] 06:04:27 DEBUG } 06:04:27 DEBUG ] 06:04:27 DEBUG } 06:04:27 DEBUG ] 06:04:27 DEBUG } 06:04:27 DEBUG } 06:04:27 DEBUG ] 06:04:27 DEBUG }, 06:04:27 DEBUG "42512268.46666667" 06:04:27 DEBUG ] 06:04:27 DEBUG } 06:04:27 DEBUG "value": [ 06:04:27 DEBUG 1712725467, 06:04:27 DEBUG } 06:04:27 DEBUG } 06:04:27 DEBUG "0" 06:04:27 DEBUG ] 06:04:27 DEBUG } 06:04:27 DEBUG ] 06:04:27 DEBUG ] 06:04:27 DEBUG ] 06:04:27 DEBUG } 06:04:27 DEBUG } 06:04:27 DEBUG } 06:04:27 DEBUG } 06:04:27 DEBUG } 06:04:27 DEBUG } 06:04:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725467" 06:04:27 INFO [http_cmd]: http status code OK 06:04:27 DEBUG --- status code --- 06:04:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:27 DEBUG --- http response --- 06:04:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:04:27 DEBUG ----- output ----- 06:04:27 DEBUG { 06:04:27 DEBUG "status": "success", 06:04:27 DEBUG "data": { 06:04:27 DEBUG "resultType": "vector", 06:04:27 DEBUG "result": [ 06:04:27 DEBUG { 06:04:27 DEBUG "metric": { 06:04:27 DEBUG "pod": "ds-idrepo-0" 06:04:27 DEBUG }, 06:04:27 DEBUG "value": [ 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG [...] 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG "pod": "ds-idrepo-1" 06:04:27 DEBUG }, 06:04:27 DEBUG "value": [ 06:04:27 DEBUG 1712725467, 06:04:27 DEBUG "0" 06:04:27 DEBUG ] 06:04:27 DEBUG } 06:04:27 DEBUG ] 06:04:27 DEBUG } 06:04:27 DEBUG } 06:04:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725467" 06:04:27 INFO [http_cmd]: http status code OK 06:04:27 DEBUG --- status code --- 06:04:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:27 DEBUG --- http response --- 06:04:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:04:27 DEBUG ----- output ----- 06:04:27 DEBUG { 06:04:27 DEBUG "status": "success", 06:04:27 DEBUG "data": { 06:04:27 DEBUG "resultType": "vector", 06:04:27 DEBUG "result": [ 06:04:27 DEBUG { 06:04:27 DEBUG "metric": { 06:04:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:04:27 DEBUG }, 06:04:27 DEBUG "value": [ 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG [...] 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:04:27 DEBUG }, 06:04:27 DEBUG "value": [ 06:04:27 DEBUG 1712725467, 06:04:27 DEBUG "1449.8666666666666" 06:04:27 DEBUG ] 06:04:27 DEBUG } 06:04:27 DEBUG ] 06:04:27 DEBUG } 06:04:27 DEBUG } 06:04:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725467" 06:04:27 INFO [http_cmd]: http status code OK 06:04:27 DEBUG --- status code --- 06:04:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:27 DEBUG --- http response --- 06:04:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:04:27 DEBUG ----- output ----- 06:04:27 DEBUG { 06:04:27 DEBUG "status": "success", 06:04:27 DEBUG "data": { 06:04:27 DEBUG "resultType": "vector", 06:04:27 DEBUG "result": [ 06:04:27 DEBUG { 06:04:27 DEBUG "metric": { 06:04:27 DEBUG "instance": "10.142.0.25:9100" 06:04:27 DEBUG }, 06:04:27 DEBUG "value": [ 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG [...] 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG 06:04:27 DEBUG "instance": "10.142.0.77:9100" 06:04:27 DEBUG }, 06:04:27 DEBUG "value": [ 06:04:27 DEBUG 1712725467, 06:04:27 DEBUG "0.15440000000000217" 06:04:27 DEBUG ] 06:04:27 DEBUG } 06:04:27 DEBUG ] 06:04:27 DEBUG } 06:04:27 DEBUG } 06:04:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725468" 06:04:28 INFO [http_cmd]: http status code OK 06:04:28 DEBUG --- status code --- 06:04:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:28 DEBUG --- http response --- 06:04:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 06:04:28 DEBUG ----- output ----- 06:04:28 DEBUG { 06:04:28 DEBUG "status": "success", 06:04:28 DEBUG "data": { 06:04:28 DEBUG "resultType": "vector", 06:04:28 DEBUG "result": [ 06:04:28 DEBUG { 06:04:28 DEBUG "metric": { 06:04:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:04:28 DEBUG }, 06:04:28 DEBUG "value": [ 06:04:28 DEBUG 06:04:28 DEBUG 06:04:28 DEBUG 06:04:28 DEBUG [...] 06:04:28 DEBUG 06:04:28 DEBUG 06:04:28 DEBUG 06:04:28 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:04:28 DEBUG }, 06:04:28 DEBUG "value": [ 06:04:28 DEBUG 1712725468, 06:04:28 DEBUG "52244568.379272714" 06:04:28 DEBUG ] 06:04:28 DEBUG } 06:04:28 DEBUG ] 06:04:28 DEBUG } 06:04:28 DEBUG } 06:04:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712725468" 06:04:28 INFO [http_cmd]: http status code OK 06:04:28 DEBUG --- status code --- 06:04:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:28 DEBUG --- http response --- 06:04:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:04:28 DEBUG ----- output ----- 06:04:28 DEBUG { 06:04:28 DEBUG "status": "success", 06:04:28 DEBUG "data": { 06:04:28 DEBUG "resultType": "vector", 06:04:28 DEBUG "result": [ 06:04:28 DEBUG { 06:04:28 DEBUG "metric": { 06:04:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:04:28 DEBUG }, 06:04:28 DEBUG "value": [ 06:04:28 DEBUG 06:04:28 DEBUG 06:04:28 DEBUG 06:04:28 DEBUG [...] 06:04:28 DEBUG 06:04:28 DEBUG 06:04:28 DEBUG 06:04:28 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:04:28 DEBUG }, 06:04:28 DEBUG "value": [ 06:04:28 DEBUG 1712725468, 06:04:28 DEBUG "4.163775231765703" 06:04:28 DEBUG ] 06:04:28 DEBUG } 06:04:28 DEBUG ] 06:04:28 DEBUG } 06:04:28 DEBUG } 06:04:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712725468" 06:04:28 INFO [http_cmd]: http status code OK 06:04:28 DEBUG --- status code --- 06:04:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:28 DEBUG --- http response --- 06:04:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:04:28 DEBUG ----- output ----- 06:04:28 DEBUG { 06:04:28 DEBUG "status": "success", 06:04:28 DEBUG "data": { 06:04:28 DEBUG "resultType": "vector", 06:04:28 DEBUG "result": [ 06:04:28 DEBUG { 06:04:28 DEBUG "metric": { 06:04:28 DEBUG "pod": "ds-idrepo-0" 06:04:28 DEBUG }, 06:04:28 DEBUG "value": [ 06:04:28 DEBUG 06:04:28 DEBUG 06:04:28 DEBUG 06:04:28 DEBUG [...] 06:04:28 DEBUG 06:04:28 DEBUG 06:04:28 DEBUG 06:04:28 DEBUG "pod": "ds-cts-2" 06:04:28 DEBUG }, 06:04:28 DEBUG "value": [ 06:04:28 DEBUG 1712725468, 06:04:28 DEBUG "0" 06:04:28 DEBUG ] 06:04:28 DEBUG } 06:04:28 DEBUG ] 06:04:28 DEBUG } 06:04:28 DEBUG } 06:04:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725469" 06:04:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725469" 06:04:29 INFO [http_cmd]: http status code OK 06:04:29 DEBUG --- status code --- 06:04:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:29 DEBUG --- http response --- 06:04:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:04:29 DEBUG ----- output ----- 06:04:29 DEBUG { 06:04:29 DEBUG "status": "success", 06:04:29 DEBUG "data": { 06:04:29 DEBUG "resultType": "vector", 06:04:29 DEBUG "result": [ 06:04:29 DEBUG { 06:04:29 DEBUG "metric": { 06:04:29 DEBUG "pod": "ds-idrepo-2" 06:04:29 DEBUG }, 06:04:29 DEBUG "value": [ 06:04:29 DEBUG 06:04:29 DEBUG 06:04:29 DEBUG 06:04:29 DEBUG [...] 06:04:29 DEBUG 06:04:29 DEBUG 06:04:29 DEBUG 06:04:29 DEBUG "pod": "ds-cts-2" 06:04:29 DEBUG }, 06:04:29 DEBUG "value": [ 06:04:29 DEBUG 1712725469, 06:04:29 DEBUG "0.00006132666666666667" 06:04:29 DEBUG ] 06:04:29 DEBUG } 06:04:29 DEBUG ] 06:04:29 DEBUG } 06:04:29 DEBUG } 06:04:29 INFO [http_cmd]: http status code OK 06:04:29 DEBUG --- status code --- 06:04:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:29 DEBUG --- http response --- 06:04:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:04:29 DEBUG ----- output ----- 06:04:29 DEBUG { 06:04:29 DEBUG "status": "success", 06:04:29 DEBUG "data": { 06:04:29 DEBUG "resultType": "vector", 06:04:29 DEBUG "result": [ 06:04:29 DEBUG { 06:04:29 DEBUG "metric": { 06:04:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:04:29 DEBUG }, 06:04:29 DEBUG "value": [ 06:04:29 DEBUG 06:04:29 DEBUG 06:04:29 DEBUG 06:04:29 DEBUG [...] 06:04:29 DEBUG 06:04:29 DEBUG 06:04:29 DEBUG 06:04:29 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:04:29 DEBUG }, 06:04:29 DEBUG "value": [ 06:04:29 DEBUG 1712725469, 06:04:29 DEBUG "0.004447447359674084" 06:04:29 DEBUG ] 06:04:29 DEBUG } 06:04:29 DEBUG ] 06:04:29 DEBUG } 06:04:29 DEBUG } 06:04:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725469" 06:04:29 INFO [http_cmd]: http status code OK 06:04:29 DEBUG --- status code --- 06:04:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:29 DEBUG --- http response --- 06:04:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:04:29 DEBUG ----- output ----- 06:04:29 DEBUG { 06:04:29 DEBUG "status": "success", 06:04:29 DEBUG "data": { 06:04:29 DEBUG "resultType": "vector", 06:04:29 DEBUG "result": [ 06:04:29 DEBUG { 06:04:29 DEBUG "metric": { 06:04:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:04:29 DEBUG }, 06:04:29 DEBUG "value": [ 06:04:29 DEBUG 06:04:29 DEBUG 06:04:29 DEBUG 06:04:29 DEBUG [...] 06:04:29 DEBUG 06:04:29 DEBUG 06:04:29 DEBUG 06:04:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:04:29 DEBUG }, 06:04:29 DEBUG "value": [ 06:04:29 DEBUG 1712725469, 06:04:29 DEBUG "1449.8666666666666" 06:04:29 DEBUG ] 06:04:29 DEBUG } 06:04:29 DEBUG ] 06:04:29 DEBUG } 06:04:29 DEBUG } 06:04:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712725469" 06:04:29 INFO [http_cmd]: http status code OK 06:04:29 DEBUG --- status code --- 06:04:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:29 DEBUG --- http response --- 06:04:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:04:29 DEBUG ----- output ----- 06:04:29 DEBUG { 06:04:29 DEBUG "status": "success", 06:04:29 DEBUG "data": { 06:04:29 DEBUG "resultType": "vector", 06:04:29 DEBUG "result": [ 06:04:29 DEBUG { 06:04:29 DEBUG "metric": { 06:04:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:04:29 DEBUG }, 06:04:29 DEBUG "value": [ 06:04:29 DEBUG 06:04:29 DEBUG 06:04:29 DEBUG 06:04:29 DEBUG [...] 06:04:29 DEBUG 06:04:29 DEBUG 06:04:29 DEBUG 06:04:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:04:29 DEBUG }, 06:04:29 DEBUG "value": [ 06:04:29 DEBUG 1712725469, 06:04:29 DEBUG "0.048" 06:04:29 DEBUG ] 06:04:29 DEBUG } 06:04:29 DEBUG ] 06:04:29 DEBUG } 06:04:29 DEBUG } 06:04:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725469" 06:04:29 INFO [http_cmd]: http status code OK 06:04:29 DEBUG --- status code --- 06:04:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:29 DEBUG --- http response --- 06:04:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:04:29 DEBUG ----- output ----- 06:04:29 DEBUG { 06:04:29 DEBUG "status": "success", 06:04:29 DEBUG "data": { 06:04:29 DEBUG "resultType": "vector", 06:04:29 DEBUG "result": [ 06:04:29 DEBUG { 06:04:29 DEBUG "metric": { 06:04:29 DEBUG "pod": "ds-idrepo-0" 06:04:29 DEBUG }, 06:04:29 DEBUG "value": [ 06:04:29 DEBUG 06:04:29 DEBUG 06:04:29 DEBUG 06:04:29 DEBUG [...] 06:04:29 DEBUG 06:04:29 DEBUG 06:04:29 DEBUG 06:04:29 DEBUG "pod": "ds-idrepo-1" 06:04:29 DEBUG }, 06:04:29 DEBUG "value": [ 06:04:29 DEBUG 1712725469, 06:04:29 DEBUG "0" 06:04:29 DEBUG ] 06:04:29 DEBUG } 06:04:29 DEBUG ] 06:04:29 DEBUG } 06:04:29 DEBUG } 06:04:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725469" 06:04:29 INFO [http_cmd]: http status code OK 06:04:29 DEBUG --- status code --- 06:04:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:29 DEBUG --- http response --- 06:04:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:04:29 DEBUG ----- output ----- 06:04:29 DEBUG { 06:04:29 DEBUG "status": "success", 06:04:29 DEBUG "data": { 06:04:29 DEBUG "resultType": "vector", 06:04:29 DEBUG "result": [ 06:04:29 DEBUG { 06:04:29 DEBUG "metric": { 06:04:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:04:29 DEBUG }, 06:04:29 DEBUG "value": [ 06:04:29 DEBUG 06:04:29 DEBUG 06:04:29 DEBUG 06:04:29 DEBUG [...] 06:04:29 DEBUG 06:04:29 DEBUG 06:04:29 DEBUG 06:04:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:04:29 DEBUG }, 06:04:29 DEBUG "value": [ 06:04:29 DEBUG 1712725469, 06:04:29 DEBUG "1416.5333333333333" 06:04:29 DEBUG ] 06:04:29 DEBUG } 06:04:29 DEBUG ] 06:04:29 DEBUG } 06:04:29 DEBUG } 06:04:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725470" 06:04:30 INFO [http_cmd]: http status code OK 06:04:30 DEBUG --- status code --- 06:04:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:30 DEBUG --- http response --- 06:04:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:04:30 DEBUG ----- output ----- 06:04:30 DEBUG { 06:04:30 DEBUG "status": "success", 06:04:30 DEBUG "data": { 06:04:30 DEBUG "resultType": "vector", 06:04:30 DEBUG "result": [ 06:04:30 DEBUG { 06:04:30 DEBUG "metric": { 06:04:30 DEBUG "pod": "ds-idrepo-0" 06:04:30 DEBUG }, 06:04:30 DEBUG "value": [ 06:04:30 DEBUG 06:04:30 DEBUG 06:04:30 DEBUG 06:04:30 DEBUG [...] 06:04:30 DEBUG 06:04:30 DEBUG 06:04:30 DEBUG 06:04:30 DEBUG "pod": "ds-idrepo-1" 06:04:30 DEBUG }, 06:04:30 DEBUG "value": [ 06:04:30 DEBUG 1712725470, 06:04:30 DEBUG "0" 06:04:30 DEBUG ] 06:04:30 DEBUG } 06:04:30 DEBUG ] 06:04:30 DEBUG } 06:04:30 DEBUG } 06:04:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725470" 06:04:30 INFO [http_cmd]: http status code OK 06:04:30 DEBUG --- status code --- 06:04:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:30 DEBUG --- http response --- 06:04:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:04:30 DEBUG ----- output ----- 06:04:30 DEBUG { 06:04:30 DEBUG "status": "success", 06:04:30 DEBUG "data": { 06:04:30 DEBUG "resultType": "vector", 06:04:30 DEBUG "result": [ 06:04:30 DEBUG { 06:04:30 DEBUG "metric": { 06:04:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:04:30 DEBUG }, 06:04:30 DEBUG "value": [ 06:04:30 DEBUG 06:04:30 DEBUG 06:04:30 DEBUG 06:04:30 DEBUG [...] 06:04:30 DEBUG 06:04:30 DEBUG 06:04:30 DEBUG 06:04:30 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:04:30 DEBUG }, 06:04:30 DEBUG "value": [ 06:04:30 DEBUG 1712725470, 06:04:30 DEBUG "0" 06:04:30 DEBUG ] 06:04:30 DEBUG } 06:04:30 DEBUG ] 06:04:30 DEBUG } 06:04:30 DEBUG } 06:04:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712725470" 06:04:30 INFO [http_cmd]: http status code OK 06:04:30 DEBUG --- status code --- 06:04:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:30 DEBUG --- http response --- 06:04:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:04:30 DEBUG ----- output ----- 06:04:30 DEBUG { 06:04:30 DEBUG "status": "success", 06:04:30 DEBUG "data": { 06:04:30 DEBUG "resultType": "vector", 06:04:30 DEBUG "result": [ 06:04:30 DEBUG { 06:04:30 DEBUG "metric": { 06:04:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:04:30 DEBUG }, 06:04:30 DEBUG "value": [ 06:04:30 DEBUG 06:04:30 DEBUG 06:04:30 DEBUG 06:04:30 DEBUG [...] 06:04:30 DEBUG 06:04:30 DEBUG 06:04:30 DEBUG 06:04:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:04:30 DEBUG }, 06:04:30 DEBUG "value": [ 06:04:30 DEBUG 1712725470, 06:04:30 DEBUG "2485571584" 06:04:30 DEBUG ] 06:04:30 DEBUG } 06:04:30 DEBUG ] 06:04:30 DEBUG } 06:04:30 DEBUG } 06:04:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725470" 06:04:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725470" 06:04:30 INFO [http_cmd]: http status code OK 06:04:30 DEBUG --- status code --- 06:04:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:30 DEBUG --- http response --- 06:04:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:04:30 DEBUG ----- output ----- 06:04:30 DEBUG { 06:04:30 DEBUG "status": "success", 06:04:30 DEBUG "data": { 06:04:30 DEBUG "resultType": "vector", 06:04:30 DEBUG "result": [ 06:04:30 DEBUG { 06:04:30 DEBUG "metric": { 06:04:30 DEBUG "instance": "10.142.0.25:9100" 06:04:30 DEBUG }, 06:04:30 DEBUG "value": [ 06:04:30 DEBUG 06:04:30 DEBUG 06:04:30 DEBUG 06:04:30 DEBUG [...] 06:04:30 DEBUG 06:04:30 DEBUG 06:04:30 DEBUG 06:04:30 DEBUG "instance": "10.142.0.77:9100" 06:04:30 DEBUG }, 06:04:30 DEBUG "value": [ 06:04:30 DEBUG 1712725470, 06:04:30 DEBUG "42515624.8" 06:04:30 DEBUG ] 06:04:30 DEBUG } 06:04:30 DEBUG ] 06:04:30 DEBUG } 06:04:30 DEBUG } 06:04:30 INFO [http_cmd]: http status code OK 06:04:30 DEBUG --- status code --- 06:04:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:30 DEBUG --- http response --- 06:04:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:04:30 DEBUG ----- output ----- 06:04:30 DEBUG { 06:04:30 DEBUG "status": "success", 06:04:30 DEBUG "data": { 06:04:30 DEBUG "resultType": "vector", 06:04:30 DEBUG "result": [ 06:04:30 DEBUG { 06:04:30 DEBUG "metric": { 06:04:30 DEBUG "instance": "10.142.0.25:9100" 06:04:30 DEBUG }, 06:04:30 DEBUG "value": [ 06:04:30 DEBUG 06:04:30 DEBUG 06:04:30 DEBUG 06:04:30 DEBUG [...] 06:04:30 DEBUG 06:04:30 DEBUG 06:04:30 DEBUG 06:04:30 DEBUG "instance": "10.142.0.77:9100" 06:04:30 DEBUG }, 06:04:30 DEBUG "value": [ 06:04:30 DEBUG 1712725470, 06:04:30 DEBUG "0.052666666666666424" 06:04:30 DEBUG ] 06:04:30 DEBUG } 06:04:30 DEBUG ] 06:04:30 DEBUG } 06:04:30 DEBUG } 06:04:47 INFO 06:04:47 INFO [loop_until]: kubectl --namespace=xlou top pods 06:04:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:04:47 INFO [loop_until]: OK (rc = 0) 06:04:47 DEBUG --- stdout --- 06:04:47 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10402m 8965Mi am-654c4fcbcf-9scrj 9841m 8446Mi am-654c4fcbcf-v5qb5 8523m 8441Mi ds-cts-0 8430m 13382Mi ds-cts-1 8475m 12867Mi ds-cts-2 7835m 13060Mi ds-idrepo-0 11145m 5622Mi ds-idrepo-1 381m 5672Mi ds-idrepo-2 385m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 8m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3998m 2234Mi 06:04:47 DEBUG --- stderr --- 06:04:47 DEBUG 06:04:54 INFO 06:04:54 INFO [loop_until]: kubectl --namespace=xlou top node 06:04:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:04:55 INFO [loop_until]: OK (rc = 0) 06:04:55 DEBUG --- stdout --- 06:04:55 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 95m 0% 2308Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9316m 58% 9312Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 78m 0% 3257Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13183m 82% 6511Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10913m 68% 9328Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11548m 72% 9785Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 77m 0% 858Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10880m 68% 6296Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8619m 54% 14036Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 7981m 50% 13721Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 439m 2% 6382Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8483m 53% 13513Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 440m 2% 6348Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4251m 26% 5887Mi 10% 06:04:55 DEBUG --- stderr --- 06:04:55 DEBUG 06:05:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712725526" 06:05:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725526" 06:05:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725526" 06:05:26 INFO [http_cmd]: http status code OK 06:05:26 ERROR [http_cmd]: ERROR 06:05:26 DEBUG --- status code --- 06:05:26 INFO [http_cmd]: http status code OK 06:05:26 ERROR --- status code --- 06:05:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:26 DEBUG --- status code --- 06:05:26 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:05:26 DEBUG --- http response --- 06:05:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:26 DEBUG --- http response --- 06:05:26 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:05:26 ERROR --- http response --- 06:05:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:05:26 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:05:26 DEBUG ----- output ----- 06:05:26 DEBUG { 06:05:26 DEBUG "status": "success", 06:05:26 DEBUG "data": { 06:05:26 DEBUG "resultType": "vector", 06:05:26 DEBUG "result": [ 06:05:26 DEBUG { 06:05:26 DEBUG "metric": { 06:05:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:05:26 DEBUG }, 06:05:26 DEBUG "value": [ 06:05:26 DEBUG 06:05:26 DEBUG 06:05:26 DEBUG 06:05:26 DEBUG [...] 06:05:26 DEBUG 06:05:26 DEBUG 06:05:26 DEBUG 06:05:26 DEBUG "pod": "am-654c4fcbcf-487cv" 06:05:26 DEBUG }, 06:05:26 DEBUG "value": [ 06:05:26 DEBUG 1712725526, 06:05:26 DEBUG "0" 06:05:26 DEBUG ] 06:05:26 DEBUG } 06:05:26 DEBUG ] 06:05:26 DEBUG } 06:05:26 DEBUG } 06:05:26 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:05:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725527" 06:05:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725527" 06:05:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725527" 06:05:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725527" 06:05:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725527" 06:05:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725527" 06:05:27 INFO [http_cmd]: http status code OK 06:05:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725527" 06:05:27 DEBUG --- status code --- 06:05:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:27 DEBUG --- http response --- 06:05:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:05:27 DEBUG ----- output ----- 06:05:27 DEBUG { 06:05:27 DEBUG "status": "success", 06:05:27 DEBUG "data": { 06:05:27 DEBUG "resultType": "vector", 06:05:27 DEBUG "result": [ 06:05:27 DEBUG { 06:05:27 DEBUG "metric": { 06:05:27 DEBUG "pod": "ds-idrepo-0" 06:05:27 DEBUG }, 06:05:27 DEBUG "value": [ 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG [...] 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG "pod": "ds-cts-2" 06:05:27 DEBUG }, 06:05:27 DEBUG "value": [ 06:05:27 DEBUG 1712725527, 06:05:27 DEBUG "29.7" 06:05:27 DEBUG ] 06:05:27 DEBUG } 06:05:27 DEBUG ] 06:05:27 DEBUG } 06:05:27 DEBUG } 06:05:27 INFO [http_cmd]: http status code OK 06:05:27 DEBUG --- status code --- 06:05:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:27 DEBUG --- http response --- 06:05:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:05:27 DEBUG ----- output ----- 06:05:27 DEBUG { 06:05:27 DEBUG "status": "success", 06:05:27 DEBUG "data": { 06:05:27 DEBUG "resultType": "vector", 06:05:27 DEBUG "result": [ 06:05:27 DEBUG { 06:05:27 DEBUG "metric": { 06:05:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:05:27 DEBUG }, 06:05:27 DEBUG "value": [ 06:05:27 INFO [http_cmd]: http status code OK 06:05:27 DEBUG 06:05:27 DEBUG --- status code --- 06:05:27 DEBUG 06:05:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:27 DEBUG 06:05:27 DEBUG --- http response --- 06:05:27 DEBUG [...] 06:05:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:05:27 DEBUG 06:05:27 DEBUG ----- output ----- 06:05:27 DEBUG 06:05:27 DEBUG { 06:05:27 DEBUG 06:05:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:05:27 DEBUG "status": "success", 06:05:27 DEBUG }, 06:05:27 DEBUG "value": [ 06:05:27 DEBUG 1712725527, 06:05:27 DEBUG "259810" 06:05:27 DEBUG ] 06:05:27 DEBUG } 06:05:27 DEBUG "data": { 06:05:27 DEBUG "resultType": "vector", 06:05:27 DEBUG "result": [ 06:05:27 DEBUG { 06:05:27 DEBUG "metric": { 06:05:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:05:27 DEBUG }, 06:05:27 DEBUG "value": [ 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG [...] 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:05:27 DEBUG }, 06:05:27 DEBUG "value": [ 06:05:27 DEBUG 1712725527, 06:05:27 DEBUG "1430.9333333333334" 06:05:27 DEBUG ] 06:05:27 DEBUG } 06:05:27 DEBUG ] 06:05:27 DEBUG } 06:05:27 DEBUG } 06:05:27 DEBUG ] 06:05:27 INFO [http_cmd]: http status code OK 06:05:27 DEBUG } 06:05:27 INFO [http_cmd]: http status code OK 06:05:27 DEBUG --- status code --- 06:05:27 DEBUG } 06:05:27 DEBUG --- status code --- 06:05:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:27 DEBUG --- http response --- 06:05:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:05:27 DEBUG ----- output ----- 06:05:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:27 DEBUG { 06:05:27 INFO [http_cmd]: http status code OK 06:05:27 INFO [http_cmd]: http status code OK 06:05:27 DEBUG --- http response --- 06:05:27 DEBUG "status": "success", 06:05:27 DEBUG --- status code --- 06:05:27 DEBUG --- status code --- 06:05:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:05:27 DEBUG "data": { 06:05:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:27 DEBUG --- http response --- 06:05:27 DEBUG ----- output ----- 06:05:27 DEBUG "resultType": "vector", 06:05:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:05:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:27 DEBUG { 06:05:27 DEBUG "status": "success", 06:05:27 DEBUG "data": { 06:05:27 DEBUG --- http response --- 06:05:27 DEBUG ----- output ----- 06:05:27 DEBUG "resultType": "vector", 06:05:27 DEBUG "result": [ 06:05:27 DEBUG { 06:05:27 DEBUG { 06:05:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 06:05:27 DEBUG "result": [ 06:05:27 DEBUG "metric": { 06:05:27 DEBUG "status": "success", 06:05:27 DEBUG ----- output ----- 06:05:27 DEBUG { 06:05:27 DEBUG "instance": "10.142.0.25:9100" 06:05:27 DEBUG "data": { 06:05:27 DEBUG { 06:05:27 DEBUG "metric": { 06:05:27 DEBUG }, 06:05:27 DEBUG "resultType": "vector", 06:05:27 DEBUG "status": "success", 06:05:27 DEBUG "instance": "10.142.0.25:9100" 06:05:27 DEBUG "value": [ 06:05:27 DEBUG "result": [ 06:05:27 DEBUG "data": { 06:05:27 DEBUG }, 06:05:27 DEBUG "value": [ 06:05:27 DEBUG { 06:05:27 DEBUG "resultType": "vector", 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG "metric": { 06:05:27 DEBUG "pod": "ds-idrepo-0" 06:05:27 DEBUG }, 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG "result": [ 06:05:27 DEBUG { 06:05:27 DEBUG "metric": { 06:05:27 DEBUG [...] 06:05:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:05:27 DEBUG }, 06:05:27 DEBUG "value": [ 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG [...] 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:05:27 DEBUG }, 06:05:27 DEBUG 06:05:27 DEBUG "value": [ 06:05:27 DEBUG 1712725527, 06:05:27 DEBUG "0" 06:05:27 DEBUG ] 06:05:27 DEBUG } 06:05:27 DEBUG 06:05:27 DEBUG [...] 06:05:27 DEBUG 06:05:27 DEBUG ] 06:05:27 DEBUG } 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG } 06:05:27 DEBUG "value": [ 06:05:27 DEBUG "instance": "10.142.0.77:9100" 06:05:27 DEBUG }, 06:05:27 DEBUG 06:05:27 DEBUG "value": [ 06:05:27 DEBUG 1712725527, 06:05:27 DEBUG "350641286144" 06:05:27 DEBUG ] 06:05:27 DEBUG } 06:05:27 DEBUG ] 06:05:27 DEBUG } 06:05:27 DEBUG } 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG [...] 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG "pod": "ds-cts-2" 06:05:27 DEBUG }, 06:05:27 DEBUG "value": [ 06:05:27 DEBUG 1712725527, 06:05:27 DEBUG "0" 06:05:27 DEBUG ] 06:05:27 DEBUG } 06:05:27 DEBUG ] 06:05:27 DEBUG } 06:05:27 DEBUG } 06:05:27 DEBUG "instance": "10.142.0.77:9100" 06:05:27 DEBUG }, 06:05:27 DEBUG "value": [ 06:05:27 DEBUG 1712725527, 06:05:27 DEBUG "42531429.63333333" 06:05:27 DEBUG ] 06:05:27 DEBUG } 06:05:27 DEBUG ] 06:05:27 DEBUG } 06:05:27 DEBUG } 06:05:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725527" 06:05:27 INFO [http_cmd]: http status code OK 06:05:27 DEBUG --- status code --- 06:05:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:27 DEBUG --- http response --- 06:05:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:05:27 DEBUG ----- output ----- 06:05:27 DEBUG { 06:05:27 DEBUG "status": "success", 06:05:27 DEBUG "data": { 06:05:27 DEBUG "resultType": "vector", 06:05:27 DEBUG "result": [ 06:05:27 DEBUG { 06:05:27 DEBUG "metric": { 06:05:27 DEBUG "pod": "ds-idrepo-0" 06:05:27 DEBUG }, 06:05:27 DEBUG "value": [ 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG [...] 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG "pod": "ds-idrepo-1" 06:05:27 DEBUG }, 06:05:27 DEBUG "value": [ 06:05:27 DEBUG 1712725527, 06:05:27 DEBUG "0" 06:05:27 DEBUG ] 06:05:27 DEBUG } 06:05:27 DEBUG ] 06:05:27 DEBUG } 06:05:27 DEBUG } 06:05:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725527" 06:05:27 INFO [http_cmd]: http status code OK 06:05:27 DEBUG --- status code --- 06:05:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:27 DEBUG --- http response --- 06:05:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:05:27 DEBUG ----- output ----- 06:05:27 DEBUG { 06:05:27 DEBUG "status": "success", 06:05:27 DEBUG "data": { 06:05:27 DEBUG "resultType": "vector", 06:05:27 DEBUG "result": [ 06:05:27 DEBUG { 06:05:27 DEBUG "metric": { 06:05:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:05:27 DEBUG }, 06:05:27 DEBUG "value": [ 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG [...] 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:05:27 DEBUG }, 06:05:27 DEBUG "value": [ 06:05:27 DEBUG 1712725527, 06:05:27 DEBUG "1440.4666666666667" 06:05:27 DEBUG ] 06:05:27 DEBUG } 06:05:27 DEBUG ] 06:05:27 DEBUG } 06:05:27 DEBUG } 06:05:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725527" 06:05:27 INFO [http_cmd]: http status code OK 06:05:27 DEBUG --- status code --- 06:05:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:27 DEBUG --- http response --- 06:05:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:05:27 DEBUG ----- output ----- 06:05:27 DEBUG { 06:05:27 DEBUG "status": "success", 06:05:27 DEBUG "data": { 06:05:27 DEBUG "resultType": "vector", 06:05:27 DEBUG "result": [ 06:05:27 DEBUG { 06:05:27 DEBUG "metric": { 06:05:27 DEBUG "instance": "10.142.0.25:9100" 06:05:27 DEBUG }, 06:05:27 DEBUG "value": [ 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG [...] 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG 06:05:27 DEBUG "instance": "10.142.0.77:9100" 06:05:27 DEBUG }, 06:05:27 DEBUG "value": [ 06:05:27 DEBUG 1712725527, 06:05:27 DEBUG "0.15670000000000214" 06:05:27 DEBUG ] 06:05:27 DEBUG } 06:05:27 DEBUG ] 06:05:27 DEBUG } 06:05:27 DEBUG } 06:05:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725528" 06:05:28 INFO [http_cmd]: http status code OK 06:05:28 DEBUG --- status code --- 06:05:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:28 DEBUG --- http response --- 06:05:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:05:28 DEBUG ----- output ----- 06:05:28 DEBUG { 06:05:28 DEBUG "status": "success", 06:05:28 DEBUG "data": { 06:05:28 DEBUG "resultType": "vector", 06:05:28 DEBUG "result": [ 06:05:28 DEBUG { 06:05:28 DEBUG "metric": { 06:05:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:05:28 DEBUG }, 06:05:28 DEBUG "value": [ 06:05:28 DEBUG 06:05:28 DEBUG 06:05:28 DEBUG 06:05:28 DEBUG [...] 06:05:28 DEBUG 06:05:28 DEBUG 06:05:28 DEBUG 06:05:28 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:05:28 DEBUG }, 06:05:28 DEBUG "value": [ 06:05:28 DEBUG 1712725528, 06:05:28 DEBUG "0" 06:05:28 DEBUG ] 06:05:28 DEBUG } 06:05:28 DEBUG ] 06:05:28 DEBUG } 06:05:28 DEBUG } 06:05:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712725528" 06:05:28 INFO [http_cmd]: http status code OK 06:05:28 DEBUG --- status code --- 06:05:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:28 DEBUG --- http response --- 06:05:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:05:28 DEBUG ----- output ----- 06:05:28 DEBUG { 06:05:28 DEBUG "status": "success", 06:05:28 DEBUG "data": { 06:05:28 DEBUG "resultType": "vector", 06:05:28 DEBUG "result": [ 06:05:28 DEBUG { 06:05:28 DEBUG "metric": { 06:05:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:05:28 DEBUG }, 06:05:28 DEBUG "value": [ 06:05:28 DEBUG 06:05:28 DEBUG 06:05:28 DEBUG 06:05:28 DEBUG [...] 06:05:28 DEBUG 06:05:28 DEBUG 06:05:28 DEBUG 06:05:28 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:05:28 DEBUG }, 06:05:28 DEBUG "value": [ 06:05:28 DEBUG 1712725528, 06:05:28 DEBUG "4.157349699676461" 06:05:28 DEBUG ] 06:05:28 DEBUG } 06:05:28 DEBUG ] 06:05:28 DEBUG } 06:05:28 DEBUG } 06:05:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712725528" 06:05:28 INFO [http_cmd]: http status code OK 06:05:28 DEBUG --- status code --- 06:05:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:28 DEBUG --- http response --- 06:05:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:05:28 DEBUG ----- output ----- 06:05:28 DEBUG { 06:05:28 DEBUG "status": "success", 06:05:28 DEBUG "data": { 06:05:28 DEBUG "resultType": "vector", 06:05:28 DEBUG "result": [ 06:05:28 DEBUG { 06:05:28 DEBUG "metric": { 06:05:28 DEBUG "pod": "ds-idrepo-0" 06:05:28 DEBUG }, 06:05:28 DEBUG "value": [ 06:05:28 DEBUG 06:05:28 DEBUG 06:05:28 DEBUG 06:05:28 DEBUG [...] 06:05:28 DEBUG 06:05:28 DEBUG 06:05:28 DEBUG 06:05:28 DEBUG "pod": "ds-cts-2" 06:05:28 DEBUG }, 06:05:28 DEBUG "value": [ 06:05:28 DEBUG 1712725528, 06:05:28 DEBUG "0" 06:05:28 DEBUG ] 06:05:28 DEBUG } 06:05:28 DEBUG ] 06:05:28 DEBUG } 06:05:28 DEBUG } 06:05:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725529" 06:05:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725529" 06:05:29 INFO [http_cmd]: http status code OK 06:05:29 INFO [http_cmd]: http status code OK 06:05:29 DEBUG --- status code --- 06:05:29 DEBUG --- status code --- 06:05:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:29 DEBUG --- http response --- 06:05:29 DEBUG --- http response --- 06:05:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:05:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:05:29 DEBUG ----- output ----- 06:05:29 DEBUG ----- output ----- 06:05:29 DEBUG { 06:05:29 DEBUG "status": "success", 06:05:29 DEBUG { 06:05:29 DEBUG "data": { 06:05:29 DEBUG "status": "success", 06:05:29 DEBUG "resultType": "vector", 06:05:29 DEBUG "data": { 06:05:29 DEBUG "result": [ 06:05:29 DEBUG "resultType": "vector", 06:05:29 DEBUG { 06:05:29 DEBUG "result": [ 06:05:29 DEBUG { 06:05:29 DEBUG "metric": { 06:05:29 DEBUG "metric": { 06:05:29 DEBUG "pod": "ds-idrepo-2" 06:05:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:05:29 DEBUG }, 06:05:29 DEBUG "value": [ 06:05:29 DEBUG 06:05:29 DEBUG 06:05:29 DEBUG 06:05:29 DEBUG [...] 06:05:29 DEBUG 06:05:29 DEBUG 06:05:29 DEBUG 06:05:29 DEBUG "pod": "login-ui-5f964fb894-5zpvq" 06:05:29 DEBUG }, 06:05:29 DEBUG "value": [ 06:05:29 DEBUG 1712725529, 06:05:29 DEBUG "0.000029034948141469752" 06:05:29 DEBUG ] 06:05:29 DEBUG } 06:05:29 DEBUG ] 06:05:29 DEBUG } 06:05:29 DEBUG } 06:05:29 DEBUG }, 06:05:29 DEBUG "value": [ 06:05:29 DEBUG 06:05:29 DEBUG 06:05:29 DEBUG 06:05:29 DEBUG [...] 06:05:29 DEBUG 06:05:29 DEBUG 06:05:29 DEBUG 06:05:29 DEBUG "pod": "ds-cts-2" 06:05:29 DEBUG }, 06:05:29 DEBUG "value": [ 06:05:29 DEBUG 1712725529, 06:05:29 DEBUG "0.00046666666666666666" 06:05:29 DEBUG ] 06:05:29 DEBUG } 06:05:29 DEBUG ] 06:05:29 DEBUG } 06:05:29 DEBUG } 06:05:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725529" 06:05:29 INFO [http_cmd]: http status code OK 06:05:29 DEBUG --- status code --- 06:05:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:29 DEBUG --- http response --- 06:05:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:05:29 DEBUG ----- output ----- 06:05:29 DEBUG { 06:05:29 DEBUG "status": "success", 06:05:29 DEBUG "data": { 06:05:29 DEBUG "resultType": "vector", 06:05:29 DEBUG "result": [ 06:05:29 DEBUG { 06:05:29 DEBUG "metric": { 06:05:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:05:29 DEBUG }, 06:05:29 DEBUG "value": [ 06:05:29 DEBUG 06:05:29 DEBUG 06:05:29 DEBUG 06:05:29 DEBUG [...] 06:05:29 DEBUG 06:05:29 DEBUG 06:05:29 DEBUG 06:05:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:05:29 DEBUG }, 06:05:29 DEBUG "value": [ 06:05:29 DEBUG 1712725529, 06:05:29 DEBUG "1440.1" 06:05:29 DEBUG ] 06:05:29 DEBUG } 06:05:29 DEBUG ] 06:05:29 DEBUG } 06:05:29 DEBUG } 06:05:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712725529" 06:05:29 INFO [http_cmd]: http status code OK 06:05:29 DEBUG --- status code --- 06:05:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:29 DEBUG --- http response --- 06:05:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:05:29 DEBUG ----- output ----- 06:05:29 DEBUG { 06:05:29 DEBUG "status": "success", 06:05:29 DEBUG "data": { 06:05:29 DEBUG "resultType": "vector", 06:05:29 DEBUG "result": [ 06:05:29 DEBUG { 06:05:29 DEBUG "metric": { 06:05:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:05:29 DEBUG }, 06:05:29 DEBUG "value": [ 06:05:29 DEBUG 06:05:29 DEBUG 06:05:29 DEBUG 06:05:29 DEBUG [...] 06:05:29 DEBUG 06:05:29 DEBUG 06:05:29 DEBUG 06:05:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:05:29 DEBUG }, 06:05:29 DEBUG "value": [ 06:05:29 DEBUG 1712725529, 06:05:29 DEBUG "0.048" 06:05:29 DEBUG ] 06:05:29 DEBUG } 06:05:29 DEBUG ] 06:05:29 DEBUG } 06:05:29 DEBUG } 06:05:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725529" 06:05:29 INFO [http_cmd]: http status code OK 06:05:29 DEBUG --- status code --- 06:05:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:29 DEBUG --- http response --- 06:05:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:05:29 DEBUG ----- output ----- 06:05:29 DEBUG { 06:05:29 DEBUG "status": "success", 06:05:29 DEBUG "data": { 06:05:29 DEBUG "resultType": "vector", 06:05:29 DEBUG "result": [ 06:05:29 DEBUG { 06:05:29 DEBUG "metric": { 06:05:29 DEBUG "pod": "ds-idrepo-0" 06:05:29 DEBUG }, 06:05:29 DEBUG "value": [ 06:05:29 DEBUG 06:05:29 DEBUG 06:05:29 DEBUG 06:05:29 DEBUG [...] 06:05:29 DEBUG 06:05:29 DEBUG 06:05:29 DEBUG 06:05:29 DEBUG "pod": "ds-idrepo-1" 06:05:29 DEBUG }, 06:05:29 DEBUG "value": [ 06:05:29 DEBUG 1712725529, 06:05:29 DEBUG "0" 06:05:29 DEBUG ] 06:05:29 DEBUG } 06:05:29 DEBUG ] 06:05:29 DEBUG } 06:05:29 DEBUG } 06:05:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725529" 06:05:29 INFO [http_cmd]: http status code OK 06:05:29 DEBUG --- status code --- 06:05:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:29 DEBUG --- http response --- 06:05:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:05:29 DEBUG ----- output ----- 06:05:29 DEBUG { 06:05:29 DEBUG "status": "success", 06:05:29 DEBUG "data": { 06:05:29 DEBUG "resultType": "vector", 06:05:29 DEBUG "result": [ 06:05:29 DEBUG { 06:05:29 DEBUG "metric": { 06:05:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:05:29 DEBUG }, 06:05:29 DEBUG "value": [ 06:05:29 DEBUG 06:05:29 DEBUG 06:05:29 DEBUG 06:05:29 DEBUG [...] 06:05:29 DEBUG 06:05:29 DEBUG 06:05:29 DEBUG 06:05:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:05:29 DEBUG }, 06:05:29 DEBUG "value": [ 06:05:29 DEBUG 1712725529, 06:05:29 DEBUG "1427.9333333333334" 06:05:29 DEBUG ] 06:05:29 DEBUG } 06:05:29 DEBUG ] 06:05:29 DEBUG } 06:05:29 DEBUG } 06:05:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725530" 06:05:30 INFO [http_cmd]: http status code OK 06:05:30 DEBUG --- status code --- 06:05:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:30 DEBUG --- http response --- 06:05:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:05:30 DEBUG ----- output ----- 06:05:30 DEBUG { 06:05:30 DEBUG "status": "success", 06:05:30 DEBUG "data": { 06:05:30 DEBUG "resultType": "vector", 06:05:30 DEBUG "result": [ 06:05:30 DEBUG { 06:05:30 DEBUG "metric": { 06:05:30 DEBUG "pod": "ds-idrepo-0" 06:05:30 DEBUG }, 06:05:30 DEBUG "value": [ 06:05:30 DEBUG 06:05:30 DEBUG 06:05:30 DEBUG 06:05:30 DEBUG [...] 06:05:30 DEBUG 06:05:30 DEBUG 06:05:30 DEBUG 06:05:30 DEBUG "pod": "ds-idrepo-1" 06:05:30 DEBUG }, 06:05:30 DEBUG "value": [ 06:05:30 DEBUG 1712725530, 06:05:30 DEBUG "0" 06:05:30 DEBUG ] 06:05:30 DEBUG } 06:05:30 DEBUG ] 06:05:30 DEBUG } 06:05:30 DEBUG } 06:05:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725530" 06:05:30 INFO [http_cmd]: http status code OK 06:05:30 DEBUG --- status code --- 06:05:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:30 DEBUG --- http response --- 06:05:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:05:30 DEBUG ----- output ----- 06:05:30 DEBUG { 06:05:30 DEBUG "status": "success", 06:05:30 DEBUG "data": { 06:05:30 DEBUG "resultType": "vector", 06:05:30 DEBUG "result": [ 06:05:30 DEBUG { 06:05:30 DEBUG "metric": { 06:05:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:05:30 DEBUG }, 06:05:30 DEBUG "value": [ 06:05:30 DEBUG 06:05:30 DEBUG 06:05:30 DEBUG 06:05:30 DEBUG [...] 06:05:30 DEBUG 06:05:30 DEBUG 06:05:30 DEBUG 06:05:30 DEBUG "pod": "login-ui-5f964fb894-5zpvq" 06:05:30 DEBUG }, 06:05:30 DEBUG "value": [ 06:05:30 DEBUG 1712725530, 06:05:30 DEBUG "0" 06:05:30 DEBUG ] 06:05:30 DEBUG } 06:05:30 DEBUG ] 06:05:30 DEBUG } 06:05:30 DEBUG } 06:05:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712725530" 06:05:30 INFO [http_cmd]: http status code OK 06:05:30 DEBUG --- status code --- 06:05:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:30 DEBUG --- http response --- 06:05:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:05:30 DEBUG ----- output ----- 06:05:30 DEBUG { 06:05:30 DEBUG "status": "success", 06:05:30 DEBUG "data": { 06:05:30 DEBUG "resultType": "vector", 06:05:30 DEBUG "result": [ 06:05:30 DEBUG { 06:05:30 DEBUG "metric": { 06:05:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:05:30 DEBUG }, 06:05:30 DEBUG "value": [ 06:05:30 DEBUG 06:05:30 DEBUG 06:05:30 DEBUG 06:05:30 DEBUG [...] 06:05:30 DEBUG 06:05:30 DEBUG 06:05:30 DEBUG 06:05:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:05:30 DEBUG }, 06:05:30 DEBUG "value": [ 06:05:30 DEBUG 1712725530, 06:05:30 DEBUG "2487197696" 06:05:30 DEBUG ] 06:05:30 DEBUG } 06:05:30 DEBUG ] 06:05:30 DEBUG } 06:05:30 DEBUG } 06:05:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725530" 06:05:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725530" 06:05:30 INFO [http_cmd]: http status code OK 06:05:30 DEBUG --- status code --- 06:05:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:30 DEBUG --- http response --- 06:05:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:05:30 DEBUG ----- output ----- 06:05:30 DEBUG { 06:05:30 DEBUG "status": "success", 06:05:30 DEBUG "data": { 06:05:30 DEBUG "resultType": "vector", 06:05:30 DEBUG "result": [ 06:05:30 DEBUG { 06:05:30 DEBUG "metric": { 06:05:30 DEBUG "instance": "10.142.0.25:9100" 06:05:30 DEBUG }, 06:05:30 DEBUG "value": [ 06:05:30 DEBUG 06:05:30 DEBUG 06:05:30 DEBUG 06:05:30 DEBUG [...] 06:05:30 DEBUG 06:05:30 DEBUG 06:05:30 DEBUG 06:05:30 DEBUG "instance": "10.142.0.77:9100" 06:05:30 DEBUG }, 06:05:30 DEBUG "value": [ 06:05:30 DEBUG 1712725530, 06:05:30 DEBUG "42533300.2" 06:05:30 DEBUG ] 06:05:30 DEBUG } 06:05:30 DEBUG ] 06:05:30 DEBUG } 06:05:30 DEBUG } 06:05:30 INFO [http_cmd]: http status code OK 06:05:30 DEBUG --- status code --- 06:05:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:30 DEBUG --- http response --- 06:05:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:05:30 DEBUG ----- output ----- 06:05:30 DEBUG { 06:05:30 DEBUG "status": "success", 06:05:30 DEBUG "data": { 06:05:30 DEBUG "resultType": "vector", 06:05:30 DEBUG "result": [ 06:05:30 DEBUG { 06:05:30 DEBUG "metric": { 06:05:30 DEBUG "instance": "10.142.0.25:9100" 06:05:30 DEBUG }, 06:05:30 DEBUG "value": [ 06:05:30 DEBUG 06:05:30 DEBUG 06:05:30 DEBUG 06:05:30 DEBUG [...] 06:05:30 DEBUG 06:05:30 DEBUG 06:05:30 DEBUG 06:05:30 DEBUG "instance": "10.142.0.77:9100" 06:05:30 DEBUG }, 06:05:30 DEBUG "value": [ 06:05:30 DEBUG 1712725530, 06:05:30 DEBUG "0.04600000000000014" 06:05:30 DEBUG ] 06:05:30 DEBUG } 06:05:30 DEBUG ] 06:05:30 DEBUG } 06:05:30 DEBUG } 06:05:47 INFO 06:05:47 INFO [loop_until]: kubectl --namespace=xlou top pods 06:05:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:05:47 INFO [loop_until]: OK (rc = 0) 06:05:47 DEBUG --- stdout --- 06:05:47 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10334m 8964Mi am-654c4fcbcf-9scrj 9891m 8445Mi am-654c4fcbcf-v5qb5 8361m 8444Mi ds-cts-0 8344m 12440Mi ds-cts-1 8510m 12451Mi ds-cts-2 7864m 12612Mi ds-idrepo-0 10932m 5622Mi ds-idrepo-1 411m 5672Mi ds-idrepo-2 389m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 2m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4142m 2236Mi 06:05:47 DEBUG --- stderr --- 06:05:47 DEBUG 06:05:55 INFO 06:05:55 INFO [loop_until]: kubectl --namespace=xlou top node 06:05:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:05:55 INFO [loop_until]: OK (rc = 0) 06:05:55 DEBUG --- stdout --- 06:05:55 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 98m 0% 2305Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9246m 58% 9313Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 83m 0% 3253Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13166m 82% 6515Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10924m 68% 9327Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11359m 71% 9789Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 78m 0% 859Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10849m 68% 6297Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8541m 53% 13144Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 7961m 50% 13276Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 449m 2% 6384Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8550m 53% 14062Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 422m 2% 6347Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4259m 26% 5883Mi 10% 06:05:55 DEBUG --- stderr --- 06:05:55 DEBUG 06:06:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725586" 06:06:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725586" 06:06:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712725586" 06:06:26 INFO [http_cmd]: http status code OK 06:06:26 INFO [http_cmd]: http status code OK 06:06:26 DEBUG --- status code --- 06:06:26 DEBUG --- status code --- 06:06:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:26 ERROR [http_cmd]: ERROR 06:06:26 ERROR --- status code --- 06:06:26 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:06:26 ERROR --- http response --- 06:06:26 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:06:26 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:06:26 DEBUG --- http response --- 06:06:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:06:26 DEBUG ----- output ----- 06:06:26 DEBUG { 06:06:26 DEBUG "status": "success", 06:06:26 DEBUG "data": { 06:06:26 DEBUG "resultType": "vector", 06:06:26 DEBUG "result": [ 06:06:26 DEBUG { 06:06:26 DEBUG "metric": { 06:06:26 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:06:26 DEBUG }, 06:06:26 DEBUG "value": [ 06:06:26 DEBUG 06:06:26 DEBUG 06:06:26 DEBUG 06:06:26 DEBUG [...] 06:06:26 DEBUG 06:06:26 DEBUG 06:06:26 DEBUG 06:06:26 DEBUG "pod": "am-654c4fcbcf-487cv" 06:06:26 DEBUG }, 06:06:26 DEBUG "value": [ 06:06:26 DEBUG 1712725586, 06:06:26 DEBUG "0" 06:06:26 DEBUG ] 06:06:26 DEBUG } 06:06:26 DEBUG ] 06:06:26 DEBUG } 06:06:26 DEBUG } 06:06:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:26 DEBUG --- http response --- 06:06:26 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:06:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725587" 06:06:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725587" 06:06:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725587" 06:06:27 INFO [http_cmd]: http status code OK 06:06:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725587" 06:06:27 DEBUG --- status code --- 06:06:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725587" 06:06:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725587" 06:06:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725587" 06:06:27 DEBUG --- http response --- 06:06:27 INFO [http_cmd]: http status code OK 06:06:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:06:27 DEBUG --- status code --- 06:06:27 DEBUG ----- output ----- 06:06:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:27 DEBUG { 06:06:27 DEBUG --- http response --- 06:06:27 DEBUG "status": "success", 06:06:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:06:27 DEBUG "data": { 06:06:27 DEBUG ----- output ----- 06:06:27 DEBUG "resultType": "vector", 06:06:27 DEBUG { 06:06:27 DEBUG "result": [ 06:06:27 DEBUG { 06:06:27 DEBUG "metric": { 06:06:27 DEBUG "pod": "ds-idrepo-0" 06:06:27 DEBUG }, 06:06:27 DEBUG "value": [ 06:06:27 DEBUG 06:06:27 DEBUG 06:06:27 DEBUG 06:06:27 DEBUG [...] 06:06:27 DEBUG 06:06:27 DEBUG 06:06:27 DEBUG 06:06:27 DEBUG "pod": "ds-cts-2" 06:06:27 DEBUG }, 06:06:27 DEBUG "value": [ 06:06:27 DEBUG 1712725587, 06:06:27 DEBUG "29.966666666666665" 06:06:27 DEBUG "status": "success", 06:06:27 DEBUG ] 06:06:27 INFO [http_cmd]: http status code OK 06:06:27 INFO [http_cmd]: http status code OK 06:06:27 INFO [http_cmd]: http status code OK 06:06:27 DEBUG "data": { 06:06:27 DEBUG "resultType": "vector", 06:06:27 DEBUG --- status code --- 06:06:27 DEBUG --- status code --- 06:06:27 DEBUG --- status code --- 06:06:27 DEBUG "result": [ 06:06:27 DEBUG } 06:06:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:27 INFO [http_cmd]: http status code OK 06:06:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:27 DEBUG { 06:06:27 DEBUG ] 06:06:27 DEBUG --- http response --- 06:06:27 DEBUG --- http response --- 06:06:27 DEBUG --- status code --- 06:06:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:27 DEBUG --- http response --- 06:06:27 DEBUG "metric": { 06:06:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:06:27 DEBUG ----- output ----- 06:06:27 DEBUG { 06:06:27 DEBUG "status": "success", 06:06:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:06:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:06:27 DEBUG --- http response --- 06:06:27 DEBUG "data": { 06:06:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:06:27 DEBUG ----- output ----- 06:06:27 DEBUG }, 06:06:27 INFO [http_cmd]: http status code OK 06:06:27 DEBUG ----- output ----- 06:06:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:06:27 DEBUG "resultType": "vector", 06:06:27 DEBUG "result": [ 06:06:27 DEBUG } 06:06:27 DEBUG } 06:06:27 DEBUG --- status code --- 06:06:27 DEBUG { 06:06:27 DEBUG ----- output ----- 06:06:27 DEBUG { 06:06:27 DEBUG { 06:06:27 DEBUG "metric": { 06:06:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:27 DEBUG "status": "success", 06:06:27 DEBUG { 06:06:27 DEBUG "status": "success", 06:06:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:06:27 DEBUG }, 06:06:27 DEBUG --- http response --- 06:06:27 DEBUG "data": { 06:06:27 DEBUG "status": "success", 06:06:27 DEBUG "data": { 06:06:27 DEBUG "resultType": "vector", 06:06:27 DEBUG "result": [ 06:06:27 DEBUG "value": [ 06:06:27 DEBUG 06:06:27 DEBUG 06:06:27 DEBUG 06:06:27 DEBUG [...] 06:06:27 DEBUG 06:06:27 DEBUG 06:06:27 DEBUG { 06:06:27 DEBUG "data": { 06:06:27 DEBUG 06:06:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:06:27 DEBUG "value": [ 06:06:27 DEBUG 06:06:27 DEBUG "metric": { 06:06:27 DEBUG "resultType": "vector", 06:06:27 DEBUG "result": [ 06:06:27 DEBUG ----- output ----- 06:06:27 DEBUG 06:06:27 DEBUG "resultType": "vector", 06:06:27 DEBUG "instance": "10.142.0.25:9100" 06:06:27 DEBUG { 06:06:27 DEBUG "metric": { 06:06:27 DEBUG { 06:06:27 DEBUG 06:06:27 DEBUG "result": [ 06:06:27 DEBUG }, 06:06:27 DEBUG "pod": "ds-idrepo-0" 06:06:27 DEBUG }, 06:06:27 DEBUG "status": "success", 06:06:27 DEBUG [...] 06:06:27 DEBUG 06:06:27 DEBUG "value": [ 06:06:27 DEBUG "value": [ 06:06:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:06:27 DEBUG "data": { 06:06:27 DEBUG 06:06:27 DEBUG 06:06:27 DEBUG "pod": "ds-idrepo-2" 06:06:27 DEBUG 06:06:27 DEBUG }, 06:06:27 DEBUG "resultType": "vector", 06:06:27 DEBUG { 06:06:27 DEBUG }, 06:06:27 DEBUG "value": [ 06:06:27 DEBUG 06:06:27 DEBUG "value": [ 06:06:27 DEBUG "result": [ 06:06:27 DEBUG "metric": { 06:06:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:06:27 DEBUG }, 06:06:27 DEBUG 06:06:27 DEBUG 1712725587, 06:06:27 DEBUG { 06:06:27 DEBUG "value": [ 06:06:27 DEBUG 1712725587, 06:06:27 DEBUG "0" 06:06:27 DEBUG [...] 06:06:27 DEBUG "259676" 06:06:27 DEBUG "metric": { 06:06:27 DEBUG "instance": "10.142.0.25:9100" 06:06:27 DEBUG ] 06:06:27 DEBUG } 06:06:27 DEBUG 06:06:27 DEBUG ] 06:06:27 DEBUG }, 06:06:27 DEBUG 06:06:27 DEBUG ] 06:06:27 DEBUG } 06:06:27 DEBUG 06:06:27 DEBUG } 06:06:27 DEBUG "value": [ 06:06:27 DEBUG 06:06:27 DEBUG } 06:06:27 DEBUG 06:06:27 DEBUG 06:06:27 DEBUG ] 06:06:27 DEBUG 06:06:27 DEBUG 06:06:27 DEBUG 06:06:27 DEBUG "instance": "10.142.0.77:9100" 06:06:27 DEBUG } 06:06:27 DEBUG 06:06:27 DEBUG [...] 06:06:27 DEBUG 06:06:27 DEBUG }, 06:06:27 DEBUG } 06:06:27 DEBUG 06:06:27 DEBUG 06:06:27 DEBUG [...] 06:06:27 DEBUG 06:06:27 DEBUG 06:06:27 DEBUG 06:06:27 DEBUG 06:06:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:06:27 DEBUG 06:06:27 DEBUG "value": [ 06:06:27 DEBUG 1712725587, 06:06:27 DEBUG }, 06:06:27 DEBUG "pod": "ds-cts-2" 06:06:27 DEBUG [...] 06:06:27 DEBUG "352670647296" 06:06:27 DEBUG "value": [ 06:06:27 DEBUG }, 06:06:27 DEBUG 06:06:27 DEBUG 06:06:27 DEBUG 06:06:27 DEBUG 1712725587, 06:06:27 DEBUG "value": [ 06:06:27 DEBUG 1712725587, 06:06:27 DEBUG "0" 06:06:27 DEBUG "1441.9333333333334" 06:06:27 DEBUG ] 06:06:27 DEBUG } 06:06:27 DEBUG ] 06:06:27 DEBUG ] 06:06:27 DEBUG } 06:06:27 DEBUG } 06:06:27 DEBUG } 06:06:27 DEBUG "instance": "10.142.0.77:9100" 06:06:27 DEBUG ] 06:06:27 DEBUG ] 06:06:27 DEBUG } 06:06:27 DEBUG } 06:06:27 DEBUG }, 06:06:27 DEBUG ] 06:06:27 DEBUG } 06:06:27 DEBUG "value": [ 06:06:27 DEBUG 1712725587, 06:06:27 DEBUG "42698667.4" 06:06:27 DEBUG } 06:06:27 DEBUG ] 06:06:27 DEBUG } 06:06:27 DEBUG } 06:06:27 DEBUG ] 06:06:27 DEBUG } 06:06:27 DEBUG } 06:06:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725587" 06:06:27 INFO [http_cmd]: http status code OK 06:06:27 DEBUG --- status code --- 06:06:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:27 DEBUG --- http response --- 06:06:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:06:27 DEBUG ----- output ----- 06:06:27 DEBUG { 06:06:27 DEBUG "status": "success", 06:06:27 DEBUG "data": { 06:06:27 DEBUG "resultType": "vector", 06:06:27 DEBUG "result": [ 06:06:27 DEBUG { 06:06:27 DEBUG "metric": { 06:06:27 DEBUG "pod": "ds-idrepo-0" 06:06:27 DEBUG }, 06:06:27 DEBUG "value": [ 06:06:27 DEBUG 06:06:27 DEBUG 06:06:27 DEBUG 06:06:27 DEBUG [...] 06:06:27 DEBUG 06:06:27 DEBUG 06:06:27 DEBUG 06:06:27 DEBUG "pod": "ds-idrepo-1" 06:06:27 DEBUG }, 06:06:27 DEBUG "value": [ 06:06:27 DEBUG 1712725587, 06:06:27 DEBUG "0" 06:06:27 DEBUG ] 06:06:27 DEBUG } 06:06:27 DEBUG ] 06:06:27 DEBUG } 06:06:27 DEBUG } 06:06:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725587" 06:06:27 INFO [http_cmd]: http status code OK 06:06:27 DEBUG --- status code --- 06:06:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:27 DEBUG --- http response --- 06:06:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:06:27 DEBUG ----- output ----- 06:06:27 DEBUG { 06:06:27 DEBUG "status": "success", 06:06:27 DEBUG "data": { 06:06:27 DEBUG "resultType": "vector", 06:06:27 DEBUG "result": [ 06:06:27 DEBUG { 06:06:27 DEBUG "metric": { 06:06:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:06:27 DEBUG }, 06:06:27 DEBUG "value": [ 06:06:27 DEBUG 06:06:27 DEBUG 06:06:27 DEBUG 06:06:27 DEBUG [...] 06:06:27 DEBUG 06:06:27 DEBUG 06:06:27 DEBUG 06:06:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:06:27 DEBUG }, 06:06:27 DEBUG "value": [ 06:06:27 DEBUG 1712725587, 06:06:27 DEBUG "1446" 06:06:27 DEBUG ] 06:06:27 DEBUG } 06:06:27 DEBUG ] 06:06:27 DEBUG } 06:06:27 DEBUG } 06:06:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725587" 06:06:27 INFO [http_cmd]: http status code OK 06:06:27 DEBUG --- status code --- 06:06:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:27 DEBUG --- http response --- 06:06:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:06:27 DEBUG ----- output ----- 06:06:27 DEBUG { 06:06:27 DEBUG "status": "success", 06:06:27 DEBUG "data": { 06:06:27 DEBUG "resultType": "vector", 06:06:27 DEBUG "result": [ 06:06:27 DEBUG { 06:06:27 DEBUG "metric": { 06:06:27 DEBUG "instance": "10.142.0.25:9100" 06:06:27 DEBUG }, 06:06:27 DEBUG "value": [ 06:06:27 DEBUG 06:06:27 DEBUG 06:06:27 DEBUG 06:06:27 DEBUG [...] 06:06:27 DEBUG 06:06:27 DEBUG 06:06:27 DEBUG 06:06:27 DEBUG "instance": "10.142.0.77:9100" 06:06:27 DEBUG }, 06:06:27 DEBUG "value": [ 06:06:27 DEBUG 1712725587, 06:06:27 DEBUG "0.03723333333333763" 06:06:27 DEBUG ] 06:06:27 DEBUG } 06:06:27 DEBUG ] 06:06:27 DEBUG } 06:06:27 DEBUG } 06:06:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725588" 06:06:28 INFO [http_cmd]: http status code OK 06:06:28 DEBUG --- status code --- 06:06:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:28 DEBUG --- http response --- 06:06:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:06:28 DEBUG ----- output ----- 06:06:28 DEBUG { 06:06:28 DEBUG "status": "success", 06:06:28 DEBUG "data": { 06:06:28 DEBUG "resultType": "vector", 06:06:28 DEBUG "result": [ 06:06:28 DEBUG { 06:06:28 DEBUG "metric": { 06:06:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:06:28 DEBUG }, 06:06:28 DEBUG "value": [ 06:06:28 DEBUG 06:06:28 DEBUG 06:06:28 DEBUG 06:06:28 DEBUG [...] 06:06:28 DEBUG 06:06:28 DEBUG 06:06:28 DEBUG 06:06:28 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:06:28 DEBUG }, 06:06:28 DEBUG "value": [ 06:06:28 DEBUG 1712725588, 06:06:28 DEBUG "25821.709459252685" 06:06:28 DEBUG ] 06:06:28 DEBUG } 06:06:28 DEBUG ] 06:06:28 DEBUG } 06:06:28 DEBUG } 06:06:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712725588" 06:06:28 INFO [http_cmd]: http status code OK 06:06:28 DEBUG --- status code --- 06:06:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:28 DEBUG --- http response --- 06:06:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:06:28 DEBUG ----- output ----- 06:06:28 DEBUG { 06:06:28 DEBUG "status": "success", 06:06:28 DEBUG "data": { 06:06:28 DEBUG "resultType": "vector", 06:06:28 DEBUG "result": [ 06:06:28 DEBUG { 06:06:28 DEBUG "metric": { 06:06:28 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:06:28 DEBUG }, 06:06:28 DEBUG "value": [ 06:06:28 DEBUG 06:06:28 DEBUG 06:06:28 DEBUG 06:06:28 DEBUG [...] 06:06:28 DEBUG 06:06:28 DEBUG 06:06:28 DEBUG 06:06:28 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:06:28 DEBUG }, 06:06:28 DEBUG "value": [ 06:06:28 DEBUG 1712725588, 06:06:28 DEBUG "4.130351085613863" 06:06:28 DEBUG ] 06:06:28 DEBUG } 06:06:28 DEBUG ] 06:06:28 DEBUG } 06:06:28 DEBUG } 06:06:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712725589" 06:06:29 INFO [http_cmd]: http status code OK 06:06:29 DEBUG --- status code --- 06:06:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:29 DEBUG --- http response --- 06:06:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:06:29 DEBUG ----- output ----- 06:06:29 DEBUG { 06:06:29 DEBUG "status": "success", 06:06:29 DEBUG "data": { 06:06:29 DEBUG "resultType": "vector", 06:06:29 DEBUG "result": [ 06:06:29 DEBUG { 06:06:29 DEBUG "metric": { 06:06:29 DEBUG "pod": "ds-idrepo-0" 06:06:29 DEBUG }, 06:06:29 DEBUG "value": [ 06:06:29 DEBUG 06:06:29 DEBUG 06:06:29 DEBUG 06:06:29 DEBUG [...] 06:06:29 DEBUG 06:06:29 DEBUG 06:06:29 DEBUG 06:06:29 DEBUG "pod": "ds-cts-2" 06:06:29 DEBUG }, 06:06:29 DEBUG "value": [ 06:06:29 DEBUG 1712725589, 06:06:29 DEBUG "0" 06:06:29 DEBUG ] 06:06:29 DEBUG } 06:06:29 DEBUG ] 06:06:29 DEBUG } 06:06:29 DEBUG } 06:06:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725589" 06:06:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725589" 06:06:29 INFO [http_cmd]: http status code OK 06:06:29 DEBUG --- status code --- 06:06:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:29 DEBUG --- http response --- 06:06:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:06:29 DEBUG ----- output ----- 06:06:29 DEBUG { 06:06:29 DEBUG "status": "success", 06:06:29 INFO [http_cmd]: http status code OK 06:06:29 DEBUG "data": { 06:06:29 DEBUG --- status code --- 06:06:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:29 DEBUG --- http response --- 06:06:29 DEBUG "resultType": "vector", 06:06:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:06:29 DEBUG ----- output ----- 06:06:29 DEBUG "result": [ 06:06:29 DEBUG { 06:06:29 DEBUG { 06:06:29 DEBUG "status": "success", 06:06:29 DEBUG "data": { 06:06:29 DEBUG "resultType": "vector", 06:06:29 DEBUG "result": [ 06:06:29 DEBUG { 06:06:29 DEBUG "metric": { 06:06:29 DEBUG "pod": "ds-idrepo-2" 06:06:29 DEBUG }, 06:06:29 DEBUG "value": [ 06:06:29 DEBUG 06:06:29 DEBUG 06:06:29 DEBUG 06:06:29 DEBUG [...] 06:06:29 DEBUG 06:06:29 DEBUG 06:06:29 DEBUG 06:06:29 DEBUG "pod": "ds-cts-2" 06:06:29 DEBUG }, 06:06:29 DEBUG "value": [ 06:06:29 DEBUG 1712725589, 06:06:29 DEBUG "0.00019530666666666669" 06:06:29 DEBUG ] 06:06:29 DEBUG } 06:06:29 DEBUG ] 06:06:29 DEBUG } 06:06:29 DEBUG } 06:06:29 DEBUG "metric": { 06:06:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:06:29 DEBUG }, 06:06:29 DEBUG "value": [ 06:06:29 DEBUG 06:06:29 DEBUG 06:06:29 DEBUG 06:06:29 DEBUG [...] 06:06:29 DEBUG 06:06:29 DEBUG 06:06:29 DEBUG 06:06:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:06:29 DEBUG }, 06:06:29 DEBUG "value": [ 06:06:29 DEBUG 1712725589, 06:06:29 DEBUG "8.424445937448276" 06:06:29 DEBUG ] 06:06:29 DEBUG } 06:06:29 DEBUG ] 06:06:29 DEBUG } 06:06:29 DEBUG } 06:06:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725589" 06:06:29 INFO [http_cmd]: http status code OK 06:06:29 DEBUG --- status code --- 06:06:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:29 DEBUG --- http response --- 06:06:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:06:29 DEBUG ----- output ----- 06:06:29 DEBUG { 06:06:29 DEBUG "status": "success", 06:06:29 DEBUG "data": { 06:06:29 DEBUG "resultType": "vector", 06:06:29 DEBUG "result": [ 06:06:29 DEBUG { 06:06:29 DEBUG "metric": { 06:06:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:06:29 DEBUG }, 06:06:29 DEBUG "value": [ 06:06:29 DEBUG 06:06:29 DEBUG 06:06:29 DEBUG 06:06:29 DEBUG [...] 06:06:29 DEBUG 06:06:29 DEBUG 06:06:29 DEBUG 06:06:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:06:29 DEBUG }, 06:06:29 DEBUG "value": [ 06:06:29 DEBUG 1712725589, 06:06:29 DEBUG "1446.0666666666666" 06:06:29 DEBUG ] 06:06:29 DEBUG } 06:06:29 DEBUG ] 06:06:29 DEBUG } 06:06:29 DEBUG } 06:06:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712725589" 06:06:29 INFO [http_cmd]: http status code OK 06:06:29 DEBUG --- status code --- 06:06:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:29 DEBUG --- http response --- 06:06:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:06:29 DEBUG ----- output ----- 06:06:29 DEBUG { 06:06:29 DEBUG "status": "success", 06:06:29 DEBUG "data": { 06:06:29 DEBUG "resultType": "vector", 06:06:29 DEBUG "result": [ 06:06:29 DEBUG { 06:06:29 DEBUG "metric": { 06:06:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:06:29 DEBUG }, 06:06:29 DEBUG "value": [ 06:06:29 DEBUG 06:06:29 DEBUG 06:06:29 DEBUG 06:06:29 DEBUG [...] 06:06:29 DEBUG 06:06:29 DEBUG 06:06:29 DEBUG 06:06:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:06:29 DEBUG }, 06:06:29 DEBUG "value": [ 06:06:29 DEBUG 1712725589, 06:06:29 DEBUG "0.048" 06:06:29 DEBUG ] 06:06:29 DEBUG } 06:06:29 DEBUG ] 06:06:29 DEBUG } 06:06:29 DEBUG } 06:06:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725589" 06:06:29 INFO [http_cmd]: http status code OK 06:06:29 DEBUG --- status code --- 06:06:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:29 DEBUG --- http response --- 06:06:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:06:29 DEBUG ----- output ----- 06:06:29 DEBUG { 06:06:29 DEBUG "status": "success", 06:06:29 DEBUG "data": { 06:06:29 DEBUG "resultType": "vector", 06:06:29 DEBUG "result": [ 06:06:29 DEBUG { 06:06:29 DEBUG "metric": { 06:06:29 DEBUG "pod": "ds-idrepo-0" 06:06:29 DEBUG }, 06:06:29 DEBUG "value": [ 06:06:29 DEBUG 06:06:29 DEBUG 06:06:29 DEBUG 06:06:29 DEBUG [...] 06:06:29 DEBUG 06:06:29 DEBUG 06:06:29 DEBUG 06:06:29 DEBUG "pod": "ds-idrepo-1" 06:06:29 DEBUG }, 06:06:29 DEBUG "value": [ 06:06:29 DEBUG 1712725589, 06:06:29 DEBUG "0" 06:06:29 DEBUG ] 06:06:29 DEBUG } 06:06:29 DEBUG ] 06:06:29 DEBUG } 06:06:29 DEBUG } 06:06:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725590" 06:06:30 INFO [http_cmd]: http status code OK 06:06:30 DEBUG --- status code --- 06:06:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:30 DEBUG --- http response --- 06:06:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:06:30 DEBUG ----- output ----- 06:06:30 DEBUG { 06:06:30 DEBUG "status": "success", 06:06:30 DEBUG "data": { 06:06:30 DEBUG "resultType": "vector", 06:06:30 DEBUG "result": [ 06:06:30 DEBUG { 06:06:30 DEBUG "metric": { 06:06:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:06:30 DEBUG }, 06:06:30 DEBUG "value": [ 06:06:30 DEBUG 06:06:30 DEBUG 06:06:30 DEBUG 06:06:30 DEBUG [...] 06:06:30 DEBUG 06:06:30 DEBUG 06:06:30 DEBUG 06:06:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:06:30 DEBUG }, 06:06:30 DEBUG "value": [ 06:06:30 DEBUG 1712725590, 06:06:30 DEBUG "1455.1333333333332" 06:06:30 DEBUG ] 06:06:30 DEBUG } 06:06:30 DEBUG ] 06:06:30 DEBUG } 06:06:30 DEBUG } 06:06:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725590" 06:06:30 INFO [http_cmd]: http status code OK 06:06:30 DEBUG --- status code --- 06:06:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:30 DEBUG --- http response --- 06:06:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:06:30 DEBUG ----- output ----- 06:06:30 DEBUG { 06:06:30 DEBUG "status": "success", 06:06:30 DEBUG "data": { 06:06:30 DEBUG "resultType": "vector", 06:06:30 DEBUG "result": [ 06:06:30 DEBUG { 06:06:30 DEBUG "metric": { 06:06:30 DEBUG "pod": "ds-idrepo-0" 06:06:30 DEBUG }, 06:06:30 DEBUG "value": [ 06:06:30 DEBUG 06:06:30 DEBUG 06:06:30 DEBUG 06:06:30 DEBUG [...] 06:06:30 DEBUG 06:06:30 DEBUG 06:06:30 DEBUG 06:06:30 DEBUG "pod": "ds-idrepo-1" 06:06:30 DEBUG }, 06:06:30 DEBUG "value": [ 06:06:30 DEBUG 1712725590, 06:06:30 DEBUG "0" 06:06:30 DEBUG ] 06:06:30 DEBUG } 06:06:30 DEBUG ] 06:06:30 DEBUG } 06:06:30 DEBUG } 06:06:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725590" 06:06:30 INFO [http_cmd]: http status code OK 06:06:30 DEBUG --- status code --- 06:06:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:30 DEBUG --- http response --- 06:06:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:06:30 DEBUG ----- output ----- 06:06:30 DEBUG { 06:06:30 DEBUG "status": "success", 06:06:30 DEBUG "data": { 06:06:30 DEBUG "resultType": "vector", 06:06:30 DEBUG "result": [ 06:06:30 DEBUG { 06:06:30 DEBUG "metric": { 06:06:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:06:30 DEBUG }, 06:06:30 DEBUG "value": [ 06:06:30 DEBUG 06:06:30 DEBUG 06:06:30 DEBUG 06:06:30 DEBUG [...] 06:06:30 DEBUG 06:06:30 DEBUG 06:06:30 DEBUG 06:06:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:06:30 DEBUG }, 06:06:30 DEBUG "value": [ 06:06:30 DEBUG 1712725590, 06:06:30 DEBUG "0" 06:06:30 DEBUG ] 06:06:30 DEBUG } 06:06:30 DEBUG ] 06:06:30 DEBUG } 06:06:30 DEBUG } 06:06:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712725590" 06:06:30 INFO [http_cmd]: http status code OK 06:06:30 DEBUG --- status code --- 06:06:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:30 DEBUG --- http response --- 06:06:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:06:30 DEBUG ----- output ----- 06:06:30 DEBUG { 06:06:30 DEBUG "status": "success", 06:06:30 DEBUG "data": { 06:06:30 DEBUG "resultType": "vector", 06:06:30 DEBUG "result": [ 06:06:30 DEBUG { 06:06:30 DEBUG "metric": { 06:06:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:06:30 DEBUG }, 06:06:30 DEBUG "value": [ 06:06:30 DEBUG 06:06:30 DEBUG 06:06:30 DEBUG 06:06:30 DEBUG [...] 06:06:30 DEBUG 06:06:30 DEBUG 06:06:30 DEBUG 06:06:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:06:30 DEBUG }, 06:06:30 DEBUG "value": [ 06:06:30 DEBUG 1712725590, 06:06:30 DEBUG "2483449856" 06:06:30 DEBUG ] 06:06:30 DEBUG } 06:06:30 DEBUG ] 06:06:30 DEBUG } 06:06:30 DEBUG } 06:06:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725590" 06:06:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725590" 06:06:30 INFO [http_cmd]: http status code OK 06:06:30 DEBUG --- status code --- 06:06:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:30 DEBUG --- http response --- 06:06:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:06:30 DEBUG ----- output ----- 06:06:30 DEBUG { 06:06:30 DEBUG "status": "success", 06:06:30 DEBUG "data": { 06:06:30 DEBUG "resultType": "vector", 06:06:30 DEBUG "result": [ 06:06:30 DEBUG { 06:06:30 DEBUG "metric": { 06:06:30 DEBUG "instance": "10.142.0.25:9100" 06:06:30 DEBUG }, 06:06:30 DEBUG "value": [ 06:06:30 DEBUG 06:06:30 DEBUG 06:06:30 DEBUG 06:06:30 DEBUG [...] 06:06:30 DEBUG 06:06:30 DEBUG 06:06:30 DEBUG 06:06:30 DEBUG "instance": "10.142.0.77:9100" 06:06:30 DEBUG }, 06:06:30 DEBUG "value": [ 06:06:30 DEBUG 1712725590, 06:06:30 DEBUG "42700320" 06:06:30 DEBUG ] 06:06:30 DEBUG } 06:06:30 DEBUG ] 06:06:30 DEBUG } 06:06:30 DEBUG } 06:06:30 INFO [http_cmd]: http status code OK 06:06:30 DEBUG --- status code --- 06:06:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:30 DEBUG --- http response --- 06:06:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:06:30 DEBUG ----- output ----- 06:06:30 DEBUG { 06:06:30 DEBUG "status": "success", 06:06:30 DEBUG "data": { 06:06:30 DEBUG "resultType": "vector", 06:06:30 DEBUG "result": [ 06:06:30 DEBUG { 06:06:30 DEBUG "metric": { 06:06:30 DEBUG "instance": "10.142.0.25:9100" 06:06:30 DEBUG }, 06:06:30 DEBUG "value": [ 06:06:30 DEBUG 06:06:30 DEBUG 06:06:30 DEBUG 06:06:30 DEBUG [...] 06:06:30 DEBUG 06:06:30 DEBUG 06:06:30 DEBUG 06:06:30 DEBUG "instance": "10.142.0.77:9100" 06:06:30 DEBUG }, 06:06:30 DEBUG "value": [ 06:06:30 DEBUG 1712725590, 06:06:30 DEBUG "0.01533333333333271" 06:06:30 DEBUG ] 06:06:30 DEBUG } 06:06:30 DEBUG ] 06:06:30 DEBUG } 06:06:30 DEBUG } 06:06:47 INFO 06:06:47 INFO [loop_until]: kubectl --namespace=xlou top pods 06:06:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:06:47 INFO [loop_until]: OK (rc = 0) 06:06:47 DEBUG --- stdout --- 06:06:47 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10150m 8964Mi am-654c4fcbcf-9scrj 9945m 8444Mi am-654c4fcbcf-v5qb5 8569m 8441Mi ds-cts-0 8488m 13108Mi ds-cts-1 8335m 13087Mi ds-cts-2 8130m 13327Mi ds-idrepo-0 10684m 5622Mi ds-idrepo-1 354m 5671Mi ds-idrepo-2 428m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 2m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4130m 2238Mi 06:06:47 DEBUG --- stderr --- 06:06:47 DEBUG 06:06:55 INFO 06:06:55 INFO [loop_until]: kubectl --namespace=xlou top node 06:06:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:06:55 INFO [loop_until]: OK (rc = 0) 06:06:55 DEBUG --- stdout --- 06:06:55 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 95m 0% 2307Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9293m 58% 9311Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 80m 0% 3254Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 12682m 79% 6500Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10879m 68% 9328Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11005m 69% 9793Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 77m 0% 861Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10794m 67% 6295Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8624m 54% 13758Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 8312m 52% 14053Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 480m 3% 6386Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8178m 51% 13672Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 400m 2% 6345Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4099m 25% 5886Mi 10% 06:06:55 DEBUG --- stderr --- 06:06:55 DEBUG 06:07:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712725646" 06:07:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725647" 06:07:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725647" 06:07:27 INFO [http_cmd]: http status code OK 06:07:27 ERROR [http_cmd]: ERROR 06:07:27 DEBUG --- status code --- 06:07:27 INFO [http_cmd]: http status code OK 06:07:27 ERROR --- status code --- 06:07:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:27 DEBUG --- status code --- 06:07:27 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:07:27 ERROR --- http response --- 06:07:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:27 DEBUG --- http response --- 06:07:27 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:07:27 DEBUG --- http response --- 06:07:27 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:07:27 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:07:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:07:27 DEBUG ----- output ----- 06:07:27 DEBUG { 06:07:27 DEBUG "status": "success", 06:07:27 DEBUG "data": { 06:07:27 DEBUG "resultType": "vector", 06:07:27 DEBUG "result": [ 06:07:27 DEBUG { 06:07:27 DEBUG "metric": { 06:07:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:07:27 DEBUG }, 06:07:27 DEBUG "value": [ 06:07:27 DEBUG 06:07:27 DEBUG 06:07:27 DEBUG 06:07:27 DEBUG [...] 06:07:27 DEBUG 06:07:27 DEBUG 06:07:27 DEBUG 06:07:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:07:27 DEBUG }, 06:07:27 DEBUG "value": [ 06:07:27 DEBUG 1712725647, 06:07:27 DEBUG "0" 06:07:27 DEBUG ] 06:07:27 DEBUG } 06:07:27 DEBUG ] 06:07:27 DEBUG } 06:07:27 DEBUG } 06:07:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725647" 06:07:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725647" 06:07:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725647" 06:07:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725647" 06:07:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725647" 06:07:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725647" 06:07:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725647" 06:07:27 INFO [http_cmd]: http status code OK 06:07:27 DEBUG --- status code --- 06:07:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:27 DEBUG --- http response --- 06:07:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:07:27 INFO [http_cmd]: http status code OK 06:07:27 DEBUG ----- output ----- 06:07:27 INFO [http_cmd]: http status code OK 06:07:27 DEBUG --- status code --- 06:07:27 INFO [http_cmd]: http status code OK 06:07:27 DEBUG --- status code --- 06:07:27 DEBUG { 06:07:27 DEBUG --- status code --- 06:07:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:27 DEBUG --- http response --- 06:07:27 INFO [http_cmd]: http status code OK 06:07:27 DEBUG "status": "success", 06:07:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:27 DEBUG --- http response --- 06:07:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:07:27 DEBUG --- status code --- 06:07:27 DEBUG "data": { 06:07:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 06:07:27 DEBUG ----- output ----- 06:07:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:27 DEBUG "resultType": "vector", 06:07:27 DEBUG --- http response --- 06:07:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:07:27 DEBUG ----- output ----- 06:07:27 DEBUG { 06:07:27 DEBUG "result": [ 06:07:27 DEBUG { 06:07:27 DEBUG "status": "success", 06:07:27 DEBUG --- http response --- 06:07:27 DEBUG ----- output ----- 06:07:27 DEBUG { 06:07:27 DEBUG "status": "success", 06:07:27 INFO [http_cmd]: http status code OK 06:07:27 DEBUG "data": { 06:07:27 DEBUG { 06:07:27 INFO [http_cmd]: http status code OK 06:07:27 DEBUG --- status code --- 06:07:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:27 DEBUG "data": { 06:07:27 DEBUG --- status code --- 06:07:27 DEBUG "resultType": "vector", 06:07:27 DEBUG "metric": { 06:07:27 DEBUG "status": "success", 06:07:27 DEBUG --- http response --- 06:07:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:07:27 DEBUG "resultType": "vector", 06:07:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:27 DEBUG "result": [ 06:07:27 DEBUG "pod": "ds-idrepo-0" 06:07:27 DEBUG "data": { 06:07:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:07:27 DEBUG ----- output ----- 06:07:27 DEBUG "result": [ 06:07:27 DEBUG --- http response --- 06:07:27 DEBUG { 06:07:27 DEBUG }, 06:07:27 DEBUG "value": [ 06:07:27 DEBUG 06:07:27 DEBUG { 06:07:27 DEBUG { 06:07:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:07:27 DEBUG "metric": { 06:07:27 DEBUG "resultType": "vector", 06:07:27 DEBUG ----- output ----- 06:07:27 DEBUG 06:07:27 DEBUG "status": "success", 06:07:27 DEBUG "metric": { 06:07:27 DEBUG ----- output ----- 06:07:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:07:27 DEBUG "result": [ 06:07:27 DEBUG { 06:07:27 DEBUG 06:07:27 DEBUG "data": { 06:07:27 DEBUG "pod": "ds-idrepo-0" 06:07:27 DEBUG { 06:07:27 DEBUG }, 06:07:27 DEBUG { 06:07:27 DEBUG "status": "success", 06:07:27 DEBUG [...] 06:07:27 DEBUG "resultType": "vector", 06:07:27 DEBUG }, 06:07:27 DEBUG "status": "success", 06:07:27 DEBUG "value": [ 06:07:27 DEBUG "metric": { 06:07:27 DEBUG "data": { 06:07:27 DEBUG "resultType": "vector", 06:07:27 DEBUG "result": [ 06:07:27 DEBUG "value": [ 06:07:27 DEBUG "data": { 06:07:27 DEBUG 06:07:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:07:27 DEBUG "result": [ 06:07:27 DEBUG 06:07:27 DEBUG { 06:07:27 DEBUG 06:07:27 DEBUG "resultType": "vector", 06:07:27 DEBUG 06:07:27 DEBUG }, 06:07:27 DEBUG { 06:07:27 DEBUG "metric": { 06:07:27 DEBUG "metric": { 06:07:27 DEBUG 06:07:27 DEBUG "result": [ 06:07:27 DEBUG 06:07:27 DEBUG "value": [ 06:07:27 DEBUG 06:07:27 DEBUG "instance": "10.142.0.25:9100" 06:07:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:07:27 DEBUG 06:07:27 DEBUG { 06:07:27 DEBUG [...] 06:07:27 DEBUG 06:07:27 DEBUG 06:07:27 DEBUG }, 06:07:27 DEBUG }, 06:07:27 DEBUG [...] 06:07:27 DEBUG "metric": { 06:07:27 DEBUG 06:07:27 DEBUG [...] 06:07:27 DEBUG 06:07:27 DEBUG "value": [ 06:07:27 DEBUG "value": [ 06:07:27 DEBUG 06:07:27 DEBUG "instance": "10.142.0.25:9100" 06:07:27 DEBUG 06:07:27 DEBUG 06:07:27 DEBUG 06:07:27 DEBUG 06:07:27 DEBUG 06:07:27 DEBUG 06:07:27 DEBUG }, 06:07:27 DEBUG 06:07:27 DEBUG 06:07:27 DEBUG "pod": "ds-cts-2" 06:07:27 DEBUG 06:07:27 DEBUG 06:07:27 DEBUG 06:07:27 DEBUG "value": [ 06:07:27 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:07:27 DEBUG 06:07:27 DEBUG }, 06:07:27 DEBUG 06:07:27 DEBUG 06:07:27 DEBUG "pod": "ds-cts-2" 06:07:27 DEBUG 06:07:27 DEBUG }, 06:07:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:07:27 DEBUG "value": [ 06:07:27 DEBUG [...] 06:07:27 DEBUG [...] 06:07:27 DEBUG 06:07:27 DEBUG 06:07:27 DEBUG "value": [ 06:07:27 DEBUG }, 06:07:27 DEBUG 1712725647, 06:07:27 DEBUG 06:07:27 DEBUG }, 06:07:27 DEBUG 06:07:27 DEBUG 06:07:27 DEBUG 1712725647, 06:07:27 DEBUG "value": [ 06:07:27 DEBUG "29.966666666666665" 06:07:27 DEBUG 06:07:27 DEBUG "value": [ 06:07:27 DEBUG 06:07:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:07:27 DEBUG "0" 06:07:27 DEBUG 1712725647, 06:07:27 DEBUG ] 06:07:27 DEBUG 06:07:27 DEBUG 1712725647, 06:07:27 DEBUG [...] 06:07:27 DEBUG }, 06:07:27 DEBUG ] 06:07:27 DEBUG "1439.1666666666667" 06:07:27 DEBUG } 06:07:27 DEBUG "instance": "10.142.0.77:9100" 06:07:27 DEBUG "0" 06:07:27 DEBUG 06:07:27 DEBUG "value": [ 06:07:27 DEBUG } 06:07:27 DEBUG ] 06:07:27 DEBUG ] 06:07:27 DEBUG }, 06:07:27 DEBUG ] 06:07:27 DEBUG 06:07:27 DEBUG 1712725647, 06:07:27 DEBUG ] 06:07:27 DEBUG } 06:07:27 DEBUG } 06:07:27 DEBUG "value": [ 06:07:27 DEBUG } 06:07:27 DEBUG 06:07:27 DEBUG "258882.5" 06:07:27 DEBUG ] 06:07:27 DEBUG ] 06:07:27 DEBUG } 06:07:27 DEBUG 1712725647, 06:07:27 DEBUG "354820561920" 06:07:27 DEBUG ] 06:07:27 DEBUG } 06:07:27 DEBUG } 06:07:27 DEBUG } 06:07:27 DEBUG } 06:07:27 DEBUG "instance": "10.142.0.77:9100" 06:07:27 DEBUG } 06:07:27 DEBUG ] 06:07:27 DEBUG } 06:07:27 DEBUG }, 06:07:27 DEBUG ] 06:07:27 DEBUG ] 06:07:27 DEBUG } 06:07:27 DEBUG "value": [ 06:07:27 DEBUG } 06:07:27 DEBUG } 06:07:27 DEBUG } 06:07:27 DEBUG 1712725647, 06:07:27 DEBUG } 06:07:27 DEBUG } 06:07:27 DEBUG "42641569.733333334" 06:07:27 DEBUG ] 06:07:27 DEBUG } 06:07:27 DEBUG ] 06:07:27 DEBUG } 06:07:27 DEBUG } 06:07:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725647" 06:07:27 INFO [http_cmd]: http status code OK 06:07:27 DEBUG --- status code --- 06:07:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:27 DEBUG --- http response --- 06:07:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:07:27 DEBUG ----- output ----- 06:07:27 DEBUG { 06:07:27 DEBUG "status": "success", 06:07:27 DEBUG "data": { 06:07:27 DEBUG "resultType": "vector", 06:07:27 DEBUG "result": [ 06:07:27 DEBUG { 06:07:27 DEBUG "metric": { 06:07:27 DEBUG "pod": "ds-idrepo-0" 06:07:27 DEBUG }, 06:07:27 DEBUG "value": [ 06:07:27 DEBUG 06:07:27 DEBUG 06:07:27 DEBUG 06:07:27 DEBUG [...] 06:07:27 DEBUG 06:07:27 DEBUG 06:07:27 DEBUG 06:07:27 DEBUG "pod": "ds-idrepo-1" 06:07:27 DEBUG }, 06:07:27 DEBUG "value": [ 06:07:27 DEBUG 1712725647, 06:07:27 DEBUG "0" 06:07:27 DEBUG ] 06:07:27 DEBUG } 06:07:27 DEBUG ] 06:07:27 DEBUG } 06:07:27 DEBUG } 06:07:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725647" 06:07:27 INFO [http_cmd]: http status code OK 06:07:27 DEBUG --- status code --- 06:07:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:27 DEBUG --- http response --- 06:07:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:07:27 DEBUG ----- output ----- 06:07:27 DEBUG { 06:07:27 DEBUG "status": "success", 06:07:27 DEBUG "data": { 06:07:27 DEBUG "resultType": "vector", 06:07:27 DEBUG "result": [ 06:07:27 DEBUG { 06:07:27 DEBUG "metric": { 06:07:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:07:27 DEBUG }, 06:07:27 DEBUG "value": [ 06:07:27 DEBUG 06:07:27 DEBUG 06:07:27 DEBUG 06:07:27 DEBUG [...] 06:07:27 DEBUG 06:07:27 DEBUG 06:07:27 DEBUG 06:07:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:07:27 DEBUG }, 06:07:27 DEBUG "value": [ 06:07:27 DEBUG 1712725647, 06:07:27 DEBUG "1422.3666666666666" 06:07:27 DEBUG ] 06:07:27 DEBUG } 06:07:27 DEBUG ] 06:07:27 DEBUG } 06:07:27 DEBUG } 06:07:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725648" 06:07:28 INFO [http_cmd]: http status code OK 06:07:28 DEBUG --- status code --- 06:07:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:28 DEBUG --- http response --- 06:07:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:07:28 DEBUG ----- output ----- 06:07:28 DEBUG { 06:07:28 DEBUG "status": "success", 06:07:28 DEBUG "data": { 06:07:28 DEBUG "resultType": "vector", 06:07:28 DEBUG "result": [ 06:07:28 DEBUG { 06:07:28 DEBUG "metric": { 06:07:28 DEBUG "instance": "10.142.0.25:9100" 06:07:28 DEBUG }, 06:07:28 DEBUG "value": [ 06:07:28 DEBUG 06:07:28 DEBUG 06:07:28 DEBUG 06:07:28 DEBUG [...] 06:07:28 DEBUG 06:07:28 DEBUG 06:07:28 DEBUG 06:07:28 DEBUG "instance": "10.142.0.77:9100" 06:07:28 DEBUG }, 06:07:28 DEBUG "value": [ 06:07:28 DEBUG 1712725648, 06:07:28 DEBUG "0.1507999999999985" 06:07:28 DEBUG ] 06:07:28 DEBUG } 06:07:28 DEBUG ] 06:07:28 DEBUG } 06:07:28 DEBUG } 06:07:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725648" 06:07:28 INFO [http_cmd]: http status code OK 06:07:28 DEBUG --- status code --- 06:07:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:28 DEBUG --- http response --- 06:07:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:07:28 DEBUG ----- output ----- 06:07:28 DEBUG { 06:07:28 DEBUG "status": "success", 06:07:28 DEBUG "data": { 06:07:28 DEBUG "resultType": "vector", 06:07:28 DEBUG "result": [ 06:07:28 DEBUG { 06:07:28 DEBUG "metric": { 06:07:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:07:28 DEBUG }, 06:07:28 DEBUG "value": [ 06:07:28 DEBUG 06:07:28 DEBUG 06:07:28 DEBUG 06:07:28 DEBUG [...] 06:07:28 DEBUG 06:07:28 DEBUG 06:07:28 DEBUG 06:07:28 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:07:28 DEBUG }, 06:07:28 DEBUG "value": [ 06:07:28 DEBUG 1712725648, 06:07:28 DEBUG "2117.5581708381687" 06:07:28 DEBUG ] 06:07:28 DEBUG } 06:07:28 DEBUG ] 06:07:28 DEBUG } 06:07:28 DEBUG } 06:07:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712725649" 06:07:29 INFO [http_cmd]: http status code OK 06:07:29 DEBUG --- status code --- 06:07:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:29 DEBUG --- http response --- 06:07:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:07:29 DEBUG ----- output ----- 06:07:29 DEBUG { 06:07:29 DEBUG "status": "success", 06:07:29 DEBUG "data": { 06:07:29 DEBUG "resultType": "vector", 06:07:29 DEBUG "result": [ 06:07:29 DEBUG { 06:07:29 DEBUG "metric": { 06:07:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:07:29 DEBUG }, 06:07:29 DEBUG "value": [ 06:07:29 DEBUG 06:07:29 DEBUG 06:07:29 DEBUG 06:07:29 DEBUG [...] 06:07:29 DEBUG 06:07:29 DEBUG 06:07:29 DEBUG 06:07:29 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:07:29 DEBUG }, 06:07:29 DEBUG "value": [ 06:07:29 DEBUG 1712725649, 06:07:29 DEBUG "4.125302065706083" 06:07:29 DEBUG ] 06:07:29 DEBUG } 06:07:29 DEBUG ] 06:07:29 DEBUG } 06:07:29 DEBUG } 06:07:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712725649" 06:07:29 INFO [http_cmd]: http status code OK 06:07:29 DEBUG --- status code --- 06:07:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:29 DEBUG --- http response --- 06:07:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:07:29 DEBUG ----- output ----- 06:07:29 DEBUG { 06:07:29 DEBUG "status": "success", 06:07:29 DEBUG "data": { 06:07:29 DEBUG "resultType": "vector", 06:07:29 DEBUG "result": [ 06:07:29 DEBUG { 06:07:29 DEBUG "metric": { 06:07:29 DEBUG "pod": "ds-idrepo-0" 06:07:29 DEBUG }, 06:07:29 DEBUG "value": [ 06:07:29 DEBUG 06:07:29 DEBUG 06:07:29 DEBUG 06:07:29 DEBUG [...] 06:07:29 DEBUG 06:07:29 DEBUG 06:07:29 DEBUG 06:07:29 DEBUG "pod": "ds-cts-2" 06:07:29 DEBUG }, 06:07:29 DEBUG "value": [ 06:07:29 DEBUG 1712725649, 06:07:29 DEBUG "0" 06:07:29 DEBUG ] 06:07:29 DEBUG } 06:07:29 DEBUG ] 06:07:29 DEBUG } 06:07:29 DEBUG } 06:07:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725649" 06:07:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725649" 06:07:29 INFO [http_cmd]: http status code OK 06:07:29 INFO [http_cmd]: http status code OK 06:07:29 DEBUG --- status code --- 06:07:29 DEBUG --- status code --- 06:07:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:29 DEBUG --- http response --- 06:07:29 DEBUG --- http response --- 06:07:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:07:29 DEBUG ----- output ----- 06:07:29 DEBUG { 06:07:29 DEBUG "status": "success", 06:07:29 DEBUG "data": { 06:07:29 DEBUG "resultType": "vector", 06:07:29 DEBUG "result": [ 06:07:29 DEBUG { 06:07:29 DEBUG "metric": { 06:07:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:07:29 DEBUG }, 06:07:29 DEBUG "value": [ 06:07:29 DEBUG 06:07:29 DEBUG 06:07:29 DEBUG 06:07:29 DEBUG [...] 06:07:29 DEBUG 06:07:29 DEBUG 06:07:29 DEBUG 06:07:29 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:07:29 DEBUG }, 06:07:29 DEBUG "value": [ 06:07:29 DEBUG 1712725649, 06:07:29 DEBUG "0.005847313728480793" 06:07:29 DEBUG ] 06:07:29 DEBUG } 06:07:29 DEBUG ] 06:07:29 DEBUG } 06:07:29 DEBUG } 06:07:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:07:29 DEBUG ----- output ----- 06:07:29 DEBUG { 06:07:29 DEBUG "status": "success", 06:07:29 DEBUG "data": { 06:07:29 DEBUG "resultType": "vector", 06:07:29 DEBUG "result": [ 06:07:29 DEBUG { 06:07:29 DEBUG "metric": { 06:07:29 DEBUG "pod": "ds-idrepo-2" 06:07:29 DEBUG }, 06:07:29 DEBUG "value": [ 06:07:29 DEBUG 06:07:29 DEBUG 06:07:29 DEBUG 06:07:29 DEBUG [...] 06:07:29 DEBUG 06:07:29 DEBUG 06:07:29 DEBUG 06:07:29 DEBUG "pod": "ds-cts-2" 06:07:29 DEBUG }, 06:07:29 DEBUG "value": [ 06:07:29 DEBUG 1712725649, 06:07:29 DEBUG "0.00013333333333333334" 06:07:29 DEBUG ] 06:07:29 DEBUG } 06:07:29 DEBUG ] 06:07:29 DEBUG } 06:07:29 DEBUG } 06:07:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725649" 06:07:29 INFO [http_cmd]: http status code OK 06:07:29 DEBUG --- status code --- 06:07:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:29 DEBUG --- http response --- 06:07:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:07:29 DEBUG ----- output ----- 06:07:29 DEBUG { 06:07:29 DEBUG "status": "success", 06:07:29 DEBUG "data": { 06:07:29 DEBUG "resultType": "vector", 06:07:29 DEBUG "result": [ 06:07:29 DEBUG { 06:07:29 DEBUG "metric": { 06:07:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:07:29 DEBUG }, 06:07:29 DEBUG "value": [ 06:07:29 DEBUG 06:07:29 DEBUG 06:07:29 DEBUG 06:07:29 DEBUG [...] 06:07:29 DEBUG 06:07:29 DEBUG 06:07:29 DEBUG 06:07:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:07:29 DEBUG }, 06:07:29 DEBUG "value": [ 06:07:29 DEBUG 1712725649, 06:07:29 DEBUG "1421.7666666666667" 06:07:29 DEBUG ] 06:07:29 DEBUG } 06:07:29 DEBUG ] 06:07:29 DEBUG } 06:07:29 DEBUG } 06:07:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712725649" 06:07:29 INFO [http_cmd]: http status code OK 06:07:29 DEBUG --- status code --- 06:07:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:29 DEBUG --- http response --- 06:07:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:07:29 DEBUG ----- output ----- 06:07:29 DEBUG { 06:07:29 DEBUG "status": "success", 06:07:29 DEBUG "data": { 06:07:29 DEBUG "resultType": "vector", 06:07:29 DEBUG "result": [ 06:07:29 DEBUG { 06:07:29 DEBUG "metric": { 06:07:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:07:29 DEBUG }, 06:07:29 DEBUG "value": [ 06:07:29 DEBUG 06:07:29 DEBUG 06:07:29 DEBUG 06:07:29 DEBUG [...] 06:07:29 DEBUG 06:07:29 DEBUG 06:07:29 DEBUG 06:07:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:07:29 DEBUG }, 06:07:29 DEBUG "value": [ 06:07:29 DEBUG 1712725649, 06:07:29 DEBUG "0.048" 06:07:29 DEBUG ] 06:07:29 DEBUG } 06:07:29 DEBUG ] 06:07:29 DEBUG } 06:07:29 DEBUG } 06:07:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725649" 06:07:29 INFO [http_cmd]: http status code OK 06:07:29 DEBUG --- status code --- 06:07:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:29 DEBUG --- http response --- 06:07:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:07:29 DEBUG ----- output ----- 06:07:29 DEBUG { 06:07:29 DEBUG "status": "success", 06:07:29 DEBUG "data": { 06:07:29 DEBUG "resultType": "vector", 06:07:29 DEBUG "result": [ 06:07:29 DEBUG { 06:07:29 DEBUG "metric": { 06:07:29 DEBUG "pod": "ds-idrepo-0" 06:07:29 DEBUG }, 06:07:29 DEBUG "value": [ 06:07:29 DEBUG 06:07:29 DEBUG 06:07:29 DEBUG 06:07:29 DEBUG [...] 06:07:29 DEBUG 06:07:29 DEBUG 06:07:29 DEBUG 06:07:29 DEBUG "pod": "ds-idrepo-1" 06:07:29 DEBUG }, 06:07:29 DEBUG "value": [ 06:07:29 DEBUG 1712725649, 06:07:29 DEBUG "0" 06:07:29 DEBUG ] 06:07:29 DEBUG } 06:07:29 DEBUG ] 06:07:29 DEBUG } 06:07:29 DEBUG } 06:07:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725650" 06:07:30 INFO [http_cmd]: http status code OK 06:07:30 DEBUG --- status code --- 06:07:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:30 DEBUG --- http response --- 06:07:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:07:30 DEBUG ----- output ----- 06:07:30 DEBUG { 06:07:30 DEBUG "status": "success", 06:07:30 DEBUG "data": { 06:07:30 DEBUG "resultType": "vector", 06:07:30 DEBUG "result": [ 06:07:30 DEBUG { 06:07:30 DEBUG "metric": { 06:07:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:07:30 DEBUG }, 06:07:30 DEBUG "value": [ 06:07:30 DEBUG 06:07:30 DEBUG 06:07:30 DEBUG 06:07:30 DEBUG [...] 06:07:30 DEBUG 06:07:30 DEBUG 06:07:30 DEBUG 06:07:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:07:30 DEBUG }, 06:07:30 DEBUG "value": [ 06:07:30 DEBUG 1712725650, 06:07:30 DEBUG "1448.2666666666667" 06:07:30 DEBUG ] 06:07:30 DEBUG } 06:07:30 DEBUG ] 06:07:30 DEBUG } 06:07:30 DEBUG } 06:07:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725650" 06:07:30 INFO [http_cmd]: http status code OK 06:07:30 DEBUG --- status code --- 06:07:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:30 DEBUG --- http response --- 06:07:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:07:30 DEBUG ----- output ----- 06:07:30 DEBUG { 06:07:30 DEBUG "status": "success", 06:07:30 DEBUG "data": { 06:07:30 DEBUG "resultType": "vector", 06:07:30 DEBUG "result": [ 06:07:30 DEBUG { 06:07:30 DEBUG "metric": { 06:07:30 DEBUG "pod": "ds-idrepo-0" 06:07:30 DEBUG }, 06:07:30 DEBUG "value": [ 06:07:30 DEBUG 06:07:30 DEBUG 06:07:30 DEBUG 06:07:30 DEBUG [...] 06:07:30 DEBUG 06:07:30 DEBUG 06:07:30 DEBUG 06:07:30 DEBUG "pod": "ds-idrepo-1" 06:07:30 DEBUG }, 06:07:30 DEBUG "value": [ 06:07:30 DEBUG 1712725650, 06:07:30 DEBUG "0" 06:07:30 DEBUG ] 06:07:30 DEBUG } 06:07:30 DEBUG ] 06:07:30 DEBUG } 06:07:30 DEBUG } 06:07:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725650" 06:07:30 INFO [http_cmd]: http status code OK 06:07:30 DEBUG --- status code --- 06:07:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:30 DEBUG --- http response --- 06:07:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:07:30 DEBUG ----- output ----- 06:07:30 DEBUG { 06:07:30 DEBUG "status": "success", 06:07:30 DEBUG "data": { 06:07:30 DEBUG "resultType": "vector", 06:07:30 DEBUG "result": [ 06:07:30 DEBUG { 06:07:30 DEBUG "metric": { 06:07:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:07:30 DEBUG }, 06:07:30 DEBUG "value": [ 06:07:30 DEBUG 06:07:30 DEBUG 06:07:30 DEBUG 06:07:30 DEBUG [...] 06:07:30 DEBUG 06:07:30 DEBUG 06:07:30 DEBUG 06:07:30 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:07:30 DEBUG }, 06:07:30 DEBUG "value": [ 06:07:30 DEBUG 1712725650, 06:07:30 DEBUG "0" 06:07:30 DEBUG ] 06:07:30 DEBUG } 06:07:30 DEBUG ] 06:07:30 DEBUG } 06:07:30 DEBUG } 06:07:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712725650" 06:07:30 INFO [http_cmd]: http status code OK 06:07:30 DEBUG --- status code --- 06:07:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:30 DEBUG --- http response --- 06:07:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:07:30 DEBUG ----- output ----- 06:07:30 DEBUG { 06:07:30 DEBUG "status": "success", 06:07:30 DEBUG "data": { 06:07:30 DEBUG "resultType": "vector", 06:07:30 DEBUG "result": [ 06:07:30 DEBUG { 06:07:30 DEBUG "metric": { 06:07:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:07:30 DEBUG }, 06:07:30 DEBUG "value": [ 06:07:30 DEBUG 06:07:30 DEBUG 06:07:30 DEBUG 06:07:30 DEBUG [...] 06:07:30 DEBUG 06:07:30 DEBUG 06:07:30 DEBUG 06:07:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:07:30 DEBUG }, 06:07:30 DEBUG "value": [ 06:07:30 DEBUG 1712725650, 06:07:30 DEBUG "2485039104" 06:07:30 DEBUG ] 06:07:30 DEBUG } 06:07:30 DEBUG ] 06:07:30 DEBUG } 06:07:30 DEBUG } 06:07:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725650" 06:07:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725650" 06:07:30 INFO [http_cmd]: http status code OK 06:07:30 DEBUG --- status code --- 06:07:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:30 DEBUG --- http response --- 06:07:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:07:30 DEBUG ----- output ----- 06:07:30 DEBUG { 06:07:30 DEBUG "status": "success", 06:07:30 DEBUG "data": { 06:07:30 DEBUG "resultType": "vector", 06:07:30 DEBUG "result": [ 06:07:30 DEBUG { 06:07:30 DEBUG "metric": { 06:07:30 DEBUG "instance": "10.142.0.25:9100" 06:07:30 DEBUG }, 06:07:30 DEBUG "value": [ 06:07:30 DEBUG 06:07:30 DEBUG 06:07:30 DEBUG 06:07:30 DEBUG [...] 06:07:30 DEBUG 06:07:30 DEBUG 06:07:30 DEBUG 06:07:30 DEBUG "instance": "10.142.0.77:9100" 06:07:30 DEBUG }, 06:07:30 DEBUG "value": [ 06:07:30 DEBUG 1712725650, 06:07:30 DEBUG "42643384.46666667" 06:07:30 DEBUG ] 06:07:30 DEBUG } 06:07:30 DEBUG ] 06:07:30 DEBUG } 06:07:30 DEBUG } 06:07:30 INFO [http_cmd]: http status code OK 06:07:30 DEBUG --- status code --- 06:07:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:30 DEBUG --- http response --- 06:07:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:07:30 DEBUG ----- output ----- 06:07:30 DEBUG { 06:07:30 DEBUG "status": "success", 06:07:30 DEBUG "data": { 06:07:30 DEBUG "resultType": "vector", 06:07:30 DEBUG "result": [ 06:07:30 DEBUG { 06:07:30 DEBUG "metric": { 06:07:30 DEBUG "instance": "10.142.0.25:9100" 06:07:30 DEBUG }, 06:07:30 DEBUG "value": [ 06:07:30 DEBUG 06:07:30 DEBUG 06:07:30 DEBUG 06:07:30 DEBUG [...] 06:07:30 DEBUG 06:07:30 DEBUG 06:07:30 DEBUG 06:07:30 DEBUG "instance": "10.142.0.77:9100" 06:07:30 DEBUG }, 06:07:30 DEBUG "value": [ 06:07:30 DEBUG 1712725650, 06:07:30 DEBUG "0.04800000000000058" 06:07:30 DEBUG ] 06:07:30 DEBUG } 06:07:30 DEBUG ] 06:07:30 DEBUG } 06:07:30 DEBUG } 06:07:47 INFO 06:07:47 INFO [loop_until]: kubectl --namespace=xlou top pods 06:07:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:07:47 INFO [loop_until]: OK (rc = 0) 06:07:47 DEBUG --- stdout --- 06:07:47 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10263m 8972Mi am-654c4fcbcf-9scrj 9933m 8442Mi am-654c4fcbcf-v5qb5 8297m 8442Mi ds-cts-0 8373m 13382Mi ds-cts-1 8522m 12576Mi ds-cts-2 8302m 12691Mi ds-idrepo-0 10759m 5622Mi ds-idrepo-1 353m 5672Mi ds-idrepo-2 381m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 8m 71Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4108m 2238Mi 06:07:47 DEBUG --- stderr --- 06:07:47 DEBUG 06:07:55 INFO 06:07:55 INFO [loop_until]: kubectl --namespace=xlou top node 06:07:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:07:55 INFO [loop_until]: OK (rc = 0) 06:07:55 DEBUG --- stdout --- 06:07:55 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 91m 0% 2311Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8988m 56% 9313Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 78m 0% 3258Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 12996m 81% 6511Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10822m 68% 9329Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11436m 71% 9792Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 76m 0% 856Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10818m 68% 6296Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8202m 51% 14041Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 8013m 50% 13466Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 444m 2% 6386Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8659m 54% 13244Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 415m 2% 6349Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4230m 26% 5878Mi 10% 06:07:55 DEBUG --- stderr --- 06:07:55 DEBUG 06:08:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712725707" 06:08:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725707" 06:08:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725707" 06:08:27 INFO [http_cmd]: http status code OK 06:08:27 ERROR [http_cmd]: ERROR 06:08:27 DEBUG --- status code --- 06:08:27 ERROR --- status code --- 06:08:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:27 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:08:27 DEBUG --- http response --- 06:08:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:08:27 INFO [http_cmd]: http status code OK 06:08:27 ERROR --- http response --- 06:08:27 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:08:27 DEBUG --- status code --- 06:08:27 DEBUG ----- output ----- 06:08:27 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:08:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:27 DEBUG { 06:08:27 DEBUG --- http response --- 06:08:27 DEBUG "status": "success", 06:08:27 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:08:27 DEBUG "data": { 06:08:27 DEBUG "resultType": "vector", 06:08:27 DEBUG "result": [ 06:08:27 DEBUG { 06:08:27 DEBUG "metric": { 06:08:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:08:27 DEBUG }, 06:08:27 DEBUG "value": [ 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG [...] 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:08:27 DEBUG }, 06:08:27 DEBUG "value": [ 06:08:27 DEBUG 1712725707, 06:08:27 DEBUG "0" 06:08:27 DEBUG ] 06:08:27 DEBUG } 06:08:27 DEBUG ] 06:08:27 DEBUG } 06:08:27 DEBUG } 06:08:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725707" 06:08:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725707" 06:08:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725707" 06:08:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725707" 06:08:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725707" 06:08:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725707" 06:08:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725707" 06:08:27 INFO [http_cmd]: http status code OK 06:08:27 INFO [http_cmd]: http status code OK 06:08:27 DEBUG --- status code --- 06:08:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:27 DEBUG --- http response --- 06:08:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 06:08:27 DEBUG ----- output ----- 06:08:27 DEBUG { 06:08:27 DEBUG "status": "success", 06:08:27 DEBUG "data": { 06:08:27 DEBUG "resultType": "vector", 06:08:27 DEBUG "result": [ 06:08:27 DEBUG { 06:08:27 DEBUG "metric": { 06:08:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:08:27 DEBUG }, 06:08:27 DEBUG "value": [ 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG [...] 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:08:27 DEBUG }, 06:08:27 DEBUG "value": [ 06:08:27 DEBUG 1712725707, 06:08:27 DEBUG "0" 06:08:27 DEBUG ] 06:08:27 DEBUG } 06:08:27 DEBUG ] 06:08:27 DEBUG } 06:08:27 DEBUG } 06:08:27 DEBUG --- status code --- 06:08:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:27 DEBUG --- http response --- 06:08:27 INFO [http_cmd]: http status code OK 06:08:27 DEBUG --- status code --- 06:08:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:08:27 DEBUG ----- output ----- 06:08:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:27 DEBUG { 06:08:27 INFO [http_cmd]: http status code OK 06:08:27 INFO [http_cmd]: http status code OK 06:08:27 DEBUG --- http response --- 06:08:27 INFO [http_cmd]: http status code OK 06:08:27 INFO [http_cmd]: http status code OK 06:08:27 DEBUG --- status code --- 06:08:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:27 DEBUG --- status code --- 06:08:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:08:27 DEBUG --- status code --- 06:08:27 DEBUG "status": "success", 06:08:27 DEBUG --- http response --- 06:08:27 DEBUG --- status code --- 06:08:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:27 DEBUG ----- output ----- 06:08:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:27 DEBUG "data": { 06:08:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:08:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:27 DEBUG --- http response --- 06:08:27 DEBUG { 06:08:27 DEBUG --- http response --- 06:08:27 DEBUG "resultType": "vector", 06:08:27 DEBUG ----- output ----- 06:08:27 DEBUG --- http response --- 06:08:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:08:27 DEBUG ----- output ----- 06:08:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:08:27 DEBUG "result": [ 06:08:27 DEBUG { 06:08:27 DEBUG "status": "success", 06:08:27 DEBUG "data": { 06:08:27 DEBUG { 06:08:27 DEBUG ----- output ----- 06:08:27 DEBUG { 06:08:27 DEBUG "status": "success", 06:08:27 DEBUG "resultType": "vector", 06:08:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:08:27 DEBUG "status": "success", 06:08:27 DEBUG { 06:08:27 DEBUG "metric": { 06:08:27 DEBUG "data": { 06:08:27 DEBUG "result": [ 06:08:27 DEBUG ----- output ----- 06:08:27 DEBUG "data": { 06:08:27 DEBUG "status": "success", 06:08:27 DEBUG "pod": "ds-idrepo-0" 06:08:27 DEBUG "resultType": "vector", 06:08:27 DEBUG { 06:08:27 DEBUG { 06:08:27 DEBUG "resultType": "vector", 06:08:27 DEBUG "data": { 06:08:27 DEBUG }, 06:08:27 DEBUG "result": [ 06:08:27 DEBUG "metric": { 06:08:27 DEBUG "status": "success", 06:08:27 DEBUG "result": [ 06:08:27 DEBUG "resultType": "vector", 06:08:27 DEBUG "value": [ 06:08:27 DEBUG { 06:08:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:08:27 DEBUG "data": { 06:08:27 DEBUG { 06:08:27 DEBUG "result": [ 06:08:27 DEBUG 06:08:27 DEBUG "metric": { 06:08:27 DEBUG }, 06:08:27 DEBUG "resultType": "vector", 06:08:27 DEBUG "metric": { 06:08:27 DEBUG { 06:08:27 DEBUG 06:08:27 DEBUG "instance": "10.142.0.25:9100" 06:08:27 DEBUG "value": [ 06:08:27 DEBUG "result": [ 06:08:27 DEBUG { 06:08:27 DEBUG "metric": { 06:08:27 DEBUG 06:08:27 DEBUG }, 06:08:27 DEBUG 06:08:27 DEBUG "metric": { 06:08:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:08:27 DEBUG "pod": "ds-idrepo-0" 06:08:27 DEBUG [...] 06:08:27 DEBUG "value": [ 06:08:27 DEBUG 06:08:27 DEBUG "instance": "10.142.0.25:9100" 06:08:27 DEBUG }, 06:08:27 DEBUG }, 06:08:27 DEBUG "value": [ 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG }, 06:08:27 DEBUG "value": [ 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG [...] 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG "pod": "ds-cts-2" 06:08:27 DEBUG }, 06:08:27 DEBUG "value": [ 06:08:27 DEBUG 1712725707, 06:08:27 DEBUG [...] 06:08:27 DEBUG "0" 06:08:27 DEBUG ] 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG } 06:08:27 DEBUG "value": [ 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG [...] 06:08:27 DEBUG 06:08:27 DEBUG ] 06:08:27 DEBUG 06:08:27 DEBUG [...] 06:08:27 DEBUG 06:08:27 DEBUG "pod": "ds-cts-2" 06:08:27 DEBUG 06:08:27 DEBUG } 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG }, 06:08:27 DEBUG "instance": "10.142.0.77:9100" 06:08:27 DEBUG } 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG "value": [ 06:08:27 DEBUG }, 06:08:27 DEBUG [...] 06:08:27 DEBUG 06:08:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:08:27 DEBUG 1712725707, 06:08:27 DEBUG "value": [ 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG }, 06:08:27 DEBUG "29.733333333333334" 06:08:27 DEBUG 1712725707, 06:08:27 DEBUG 06:08:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:08:27 DEBUG "value": [ 06:08:27 DEBUG ] 06:08:27 DEBUG "356976548864" 06:08:27 DEBUG "instance": "10.142.0.77:9100" 06:08:27 DEBUG }, 06:08:27 DEBUG 1712725707, 06:08:27 DEBUG "258688.5" 06:08:27 DEBUG ] 06:08:27 DEBUG "value": [ 06:08:27 DEBUG }, 06:08:27 DEBUG ] 06:08:27 DEBUG } 06:08:27 DEBUG ] 06:08:27 DEBUG 1712725707, 06:08:27 DEBUG "value": [ 06:08:27 DEBUG 1712725707, 06:08:27 DEBUG } 06:08:27 DEBUG } 06:08:27 DEBUG "42880643.8" 06:08:27 DEBUG "1444.6" 06:08:27 DEBUG } 06:08:27 DEBUG } 06:08:27 DEBUG ] 06:08:27 DEBUG ] 06:08:27 DEBUG ] 06:08:27 DEBUG ] 06:08:27 DEBUG } 06:08:27 DEBUG } 06:08:27 DEBUG ] 06:08:27 DEBUG } 06:08:27 DEBUG } 06:08:27 DEBUG } 06:08:27 DEBUG } 06:08:27 DEBUG } 06:08:27 DEBUG ] 06:08:27 DEBUG } 06:08:27 DEBUG } 06:08:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725707" 06:08:27 DEBUG } 06:08:27 INFO [http_cmd]: http status code OK 06:08:27 DEBUG --- status code --- 06:08:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:27 DEBUG --- http response --- 06:08:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:08:27 DEBUG ----- output ----- 06:08:27 DEBUG { 06:08:27 DEBUG "status": "success", 06:08:27 DEBUG "data": { 06:08:27 DEBUG "resultType": "vector", 06:08:27 DEBUG "result": [ 06:08:27 DEBUG { 06:08:27 DEBUG "metric": { 06:08:27 DEBUG "pod": "ds-idrepo-0" 06:08:27 DEBUG }, 06:08:27 DEBUG "value": [ 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG [...] 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG "pod": "ds-idrepo-1" 06:08:27 DEBUG }, 06:08:27 DEBUG "value": [ 06:08:27 DEBUG 1712725707, 06:08:27 DEBUG "0" 06:08:27 DEBUG ] 06:08:27 DEBUG } 06:08:27 DEBUG ] 06:08:27 DEBUG } 06:08:27 DEBUG } 06:08:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725707" 06:08:27 INFO [http_cmd]: http status code OK 06:08:27 DEBUG --- status code --- 06:08:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:27 DEBUG --- http response --- 06:08:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:08:27 DEBUG ----- output ----- 06:08:27 DEBUG { 06:08:27 DEBUG "status": "success", 06:08:27 DEBUG "data": { 06:08:27 DEBUG "resultType": "vector", 06:08:27 DEBUG "result": [ 06:08:27 DEBUG { 06:08:27 DEBUG "metric": { 06:08:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:08:27 DEBUG }, 06:08:27 DEBUG "value": [ 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG [...] 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG 06:08:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:08:27 DEBUG }, 06:08:27 DEBUG "value": [ 06:08:27 DEBUG 1712725707, 06:08:27 DEBUG "1449.2666666666667" 06:08:27 DEBUG ] 06:08:27 DEBUG } 06:08:27 DEBUG ] 06:08:27 DEBUG } 06:08:27 DEBUG } 06:08:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725708" 06:08:28 INFO [http_cmd]: http status code OK 06:08:28 DEBUG --- status code --- 06:08:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:28 DEBUG --- http response --- 06:08:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:08:28 DEBUG ----- output ----- 06:08:28 DEBUG { 06:08:28 DEBUG "status": "success", 06:08:28 DEBUG "data": { 06:08:28 DEBUG "resultType": "vector", 06:08:28 DEBUG "result": [ 06:08:28 DEBUG { 06:08:28 DEBUG "metric": { 06:08:28 DEBUG "instance": "10.142.0.25:9100" 06:08:28 DEBUG }, 06:08:28 DEBUG "value": [ 06:08:28 DEBUG 06:08:28 DEBUG 06:08:28 DEBUG 06:08:28 DEBUG [...] 06:08:28 DEBUG 06:08:28 DEBUG 06:08:28 DEBUG 06:08:28 DEBUG "instance": "10.142.0.77:9100" 06:08:28 DEBUG }, 06:08:28 DEBUG "value": [ 06:08:28 DEBUG 1712725708, 06:08:28 DEBUG "0.08006666666666754" 06:08:28 DEBUG ] 06:08:28 DEBUG } 06:08:28 DEBUG ] 06:08:28 DEBUG } 06:08:28 DEBUG } 06:08:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725709" 06:08:29 INFO [http_cmd]: http status code OK 06:08:29 DEBUG --- status code --- 06:08:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:29 DEBUG --- http response --- 06:08:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:08:29 DEBUG ----- output ----- 06:08:29 DEBUG { 06:08:29 DEBUG "status": "success", 06:08:29 DEBUG "data": { 06:08:29 DEBUG "resultType": "vector", 06:08:29 DEBUG "result": [ 06:08:29 DEBUG { 06:08:29 DEBUG "metric": { 06:08:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:08:29 DEBUG }, 06:08:29 DEBUG "value": [ 06:08:29 DEBUG 06:08:29 DEBUG 06:08:29 DEBUG 06:08:29 DEBUG [...] 06:08:29 DEBUG 06:08:29 DEBUG 06:08:29 DEBUG 06:08:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:08:29 DEBUG }, 06:08:29 DEBUG "value": [ 06:08:29 DEBUG 1712725709, 06:08:29 DEBUG "0" 06:08:29 DEBUG ] 06:08:29 DEBUG } 06:08:29 DEBUG ] 06:08:29 DEBUG } 06:08:29 DEBUG } 06:08:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712725709" 06:08:29 INFO [http_cmd]: http status code OK 06:08:29 DEBUG --- status code --- 06:08:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:29 DEBUG --- http response --- 06:08:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:08:29 DEBUG ----- output ----- 06:08:29 DEBUG { 06:08:29 DEBUG "status": "success", 06:08:29 DEBUG "data": { 06:08:29 DEBUG "resultType": "vector", 06:08:29 DEBUG "result": [ 06:08:29 DEBUG { 06:08:29 DEBUG "metric": { 06:08:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:08:29 DEBUG }, 06:08:29 DEBUG "value": [ 06:08:29 DEBUG 06:08:29 DEBUG 06:08:29 DEBUG 06:08:29 DEBUG [...] 06:08:29 DEBUG 06:08:29 DEBUG 06:08:29 DEBUG 06:08:29 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:08:29 DEBUG }, 06:08:29 DEBUG "value": [ 06:08:29 DEBUG 1712725709, 06:08:29 DEBUG "4.153935180037361" 06:08:29 DEBUG ] 06:08:29 DEBUG } 06:08:29 DEBUG ] 06:08:29 DEBUG } 06:08:29 DEBUG } 06:08:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712725709" 06:08:29 INFO [http_cmd]: http status code OK 06:08:29 DEBUG --- status code --- 06:08:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:29 DEBUG --- http response --- 06:08:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:08:29 DEBUG ----- output ----- 06:08:29 DEBUG { 06:08:29 DEBUG "status": "success", 06:08:29 DEBUG "data": { 06:08:29 DEBUG "resultType": "vector", 06:08:29 DEBUG "result": [ 06:08:29 DEBUG { 06:08:29 DEBUG "metric": { 06:08:29 DEBUG "pod": "ds-idrepo-0" 06:08:29 DEBUG }, 06:08:29 DEBUG "value": [ 06:08:29 DEBUG 06:08:29 DEBUG 06:08:29 DEBUG 06:08:29 DEBUG [...] 06:08:29 DEBUG 06:08:29 DEBUG 06:08:29 DEBUG 06:08:29 DEBUG "pod": "ds-cts-2" 06:08:29 DEBUG }, 06:08:29 DEBUG "value": [ 06:08:29 DEBUG 1712725709, 06:08:29 DEBUG "0" 06:08:29 DEBUG ] 06:08:29 DEBUG } 06:08:29 DEBUG ] 06:08:29 DEBUG } 06:08:29 DEBUG } 06:08:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725709" 06:08:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725709" 06:08:29 INFO [http_cmd]: http status code OK 06:08:29 DEBUG --- status code --- 06:08:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:29 DEBUG --- http response --- 06:08:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:08:29 DEBUG ----- output ----- 06:08:29 DEBUG { 06:08:29 DEBUG "status": "success", 06:08:29 DEBUG "data": { 06:08:29 DEBUG "resultType": "vector", 06:08:29 DEBUG "result": [ 06:08:29 DEBUG { 06:08:29 DEBUG "metric": { 06:08:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:08:29 DEBUG }, 06:08:29 DEBUG "value": [ 06:08:29 DEBUG 06:08:29 DEBUG 06:08:29 DEBUG 06:08:29 DEBUG [...] 06:08:29 DEBUG 06:08:29 DEBUG 06:08:29 DEBUG 06:08:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:08:29 DEBUG }, 06:08:29 DEBUG "value": [ 06:08:29 DEBUG 1712725709, 06:08:29 DEBUG "0" 06:08:29 DEBUG ] 06:08:29 DEBUG } 06:08:29 DEBUG ] 06:08:29 DEBUG } 06:08:29 DEBUG } 06:08:29 INFO [http_cmd]: http status code OK 06:08:29 DEBUG --- status code --- 06:08:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:29 DEBUG --- http response --- 06:08:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:08:29 DEBUG ----- output ----- 06:08:29 DEBUG { 06:08:29 DEBUG "status": "success", 06:08:29 DEBUG "data": { 06:08:29 DEBUG "resultType": "vector", 06:08:29 DEBUG "result": [ 06:08:29 DEBUG { 06:08:29 DEBUG "metric": { 06:08:29 DEBUG "pod": "ds-idrepo-2" 06:08:29 DEBUG }, 06:08:29 DEBUG "value": [ 06:08:29 DEBUG 06:08:29 DEBUG 06:08:29 DEBUG 06:08:29 DEBUG [...] 06:08:29 DEBUG 06:08:29 DEBUG 06:08:29 DEBUG 06:08:29 DEBUG "pod": "ds-cts-2" 06:08:29 DEBUG }, 06:08:29 DEBUG "value": [ 06:08:29 DEBUG 1712725709, 06:08:29 DEBUG "0.000033333333333333335" 06:08:29 DEBUG ] 06:08:29 DEBUG } 06:08:29 DEBUG ] 06:08:29 DEBUG } 06:08:29 DEBUG } 06:08:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725709" 06:08:29 INFO [http_cmd]: http status code OK 06:08:29 DEBUG --- status code --- 06:08:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:29 DEBUG --- http response --- 06:08:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:08:29 DEBUG ----- output ----- 06:08:29 DEBUG { 06:08:29 DEBUG "status": "success", 06:08:29 DEBUG "data": { 06:08:29 DEBUG "resultType": "vector", 06:08:29 DEBUG "result": [ 06:08:29 DEBUG { 06:08:29 DEBUG "metric": { 06:08:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:08:29 DEBUG }, 06:08:29 DEBUG "value": [ 06:08:29 DEBUG 06:08:29 DEBUG 06:08:29 DEBUG 06:08:29 DEBUG [...] 06:08:29 DEBUG 06:08:29 DEBUG 06:08:29 DEBUG 06:08:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:08:29 DEBUG }, 06:08:29 DEBUG "value": [ 06:08:29 DEBUG 1712725709, 06:08:29 DEBUG "1449.8666666666666" 06:08:29 DEBUG ] 06:08:29 DEBUG } 06:08:29 DEBUG ] 06:08:29 DEBUG } 06:08:29 DEBUG } 06:08:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712725709" 06:08:29 INFO [http_cmd]: http status code OK 06:08:29 DEBUG --- status code --- 06:08:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:29 DEBUG --- http response --- 06:08:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:08:29 DEBUG ----- output ----- 06:08:29 DEBUG { 06:08:29 DEBUG "status": "success", 06:08:29 DEBUG "data": { 06:08:29 DEBUG "resultType": "vector", 06:08:29 DEBUG "result": [ 06:08:29 DEBUG { 06:08:29 DEBUG "metric": { 06:08:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:08:29 DEBUG }, 06:08:29 DEBUG "value": [ 06:08:29 DEBUG 06:08:29 DEBUG 06:08:29 DEBUG 06:08:29 DEBUG [...] 06:08:29 DEBUG 06:08:29 DEBUG 06:08:29 DEBUG 06:08:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:08:29 DEBUG }, 06:08:29 DEBUG "value": [ 06:08:29 DEBUG 1712725709, 06:08:29 DEBUG "0.048" 06:08:29 DEBUG ] 06:08:29 DEBUG } 06:08:29 DEBUG ] 06:08:29 DEBUG } 06:08:29 DEBUG } 06:08:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725710" 06:08:30 INFO [http_cmd]: http status code OK 06:08:30 DEBUG --- status code --- 06:08:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:30 DEBUG --- http response --- 06:08:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:08:30 DEBUG ----- output ----- 06:08:30 DEBUG { 06:08:30 DEBUG "status": "success", 06:08:30 DEBUG "data": { 06:08:30 DEBUG "resultType": "vector", 06:08:30 DEBUG "result": [ 06:08:30 DEBUG { 06:08:30 DEBUG "metric": { 06:08:30 DEBUG "pod": "ds-idrepo-0" 06:08:30 DEBUG }, 06:08:30 DEBUG "value": [ 06:08:30 DEBUG 06:08:30 DEBUG 06:08:30 DEBUG 06:08:30 DEBUG [...] 06:08:30 DEBUG 06:08:30 DEBUG 06:08:30 DEBUG 06:08:30 DEBUG "pod": "ds-idrepo-1" 06:08:30 DEBUG }, 06:08:30 DEBUG "value": [ 06:08:30 DEBUG 1712725710, 06:08:30 DEBUG "0" 06:08:30 DEBUG ] 06:08:30 DEBUG } 06:08:30 DEBUG ] 06:08:30 DEBUG } 06:08:30 DEBUG } 06:08:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725710" 06:08:30 INFO [http_cmd]: http status code OK 06:08:30 DEBUG --- status code --- 06:08:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:30 DEBUG --- http response --- 06:08:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:08:30 DEBUG ----- output ----- 06:08:30 DEBUG { 06:08:30 DEBUG "status": "success", 06:08:30 DEBUG "data": { 06:08:30 DEBUG "resultType": "vector", 06:08:30 DEBUG "result": [ 06:08:30 DEBUG { 06:08:30 DEBUG "metric": { 06:08:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:08:30 DEBUG }, 06:08:30 DEBUG "value": [ 06:08:30 DEBUG 06:08:30 DEBUG 06:08:30 DEBUG 06:08:30 DEBUG [...] 06:08:30 DEBUG 06:08:30 DEBUG 06:08:30 DEBUG 06:08:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:08:30 DEBUG }, 06:08:30 DEBUG "value": [ 06:08:30 DEBUG 1712725710, 06:08:30 DEBUG "1437.9333333333334" 06:08:30 DEBUG ] 06:08:30 DEBUG } 06:08:30 DEBUG ] 06:08:30 DEBUG } 06:08:30 DEBUG } 06:08:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725710" 06:08:30 INFO [http_cmd]: http status code OK 06:08:30 DEBUG --- status code --- 06:08:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:30 DEBUG --- http response --- 06:08:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:08:30 DEBUG ----- output ----- 06:08:30 DEBUG { 06:08:30 DEBUG "status": "success", 06:08:30 DEBUG "data": { 06:08:30 DEBUG "resultType": "vector", 06:08:30 DEBUG "result": [ 06:08:30 DEBUG { 06:08:30 DEBUG "metric": { 06:08:30 DEBUG "pod": "ds-idrepo-0" 06:08:30 DEBUG }, 06:08:30 DEBUG "value": [ 06:08:30 DEBUG 06:08:30 DEBUG 06:08:30 DEBUG 06:08:30 DEBUG [...] 06:08:30 DEBUG 06:08:30 DEBUG 06:08:30 DEBUG 06:08:30 DEBUG "pod": "ds-idrepo-1" 06:08:30 DEBUG }, 06:08:30 DEBUG "value": [ 06:08:30 DEBUG 1712725710, 06:08:30 DEBUG "0" 06:08:30 DEBUG ] 06:08:30 DEBUG } 06:08:30 DEBUG ] 06:08:30 DEBUG } 06:08:30 DEBUG } 06:08:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725710" 06:08:30 INFO [http_cmd]: http status code OK 06:08:30 DEBUG --- status code --- 06:08:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:30 DEBUG --- http response --- 06:08:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:08:30 DEBUG ----- output ----- 06:08:30 DEBUG { 06:08:30 DEBUG "status": "success", 06:08:30 DEBUG "data": { 06:08:30 DEBUG "resultType": "vector", 06:08:30 DEBUG "result": [ 06:08:30 DEBUG { 06:08:30 DEBUG "metric": { 06:08:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:08:30 DEBUG }, 06:08:30 DEBUG "value": [ 06:08:30 DEBUG 06:08:30 DEBUG 06:08:30 DEBUG 06:08:30 DEBUG [...] 06:08:30 DEBUG 06:08:30 DEBUG 06:08:30 DEBUG 06:08:30 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:08:30 DEBUG }, 06:08:30 DEBUG "value": [ 06:08:30 DEBUG 1712725710, 06:08:30 DEBUG "0" 06:08:30 DEBUG ] 06:08:30 DEBUG } 06:08:30 DEBUG ] 06:08:30 DEBUG } 06:08:30 DEBUG } 06:08:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712725710" 06:08:30 INFO [http_cmd]: http status code OK 06:08:30 DEBUG --- status code --- 06:08:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:30 DEBUG --- http response --- 06:08:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:08:30 DEBUG ----- output ----- 06:08:30 DEBUG { 06:08:30 DEBUG "status": "success", 06:08:30 DEBUG "data": { 06:08:30 DEBUG "resultType": "vector", 06:08:30 DEBUG "result": [ 06:08:30 DEBUG { 06:08:30 DEBUG "metric": { 06:08:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:08:30 DEBUG }, 06:08:30 DEBUG "value": [ 06:08:30 DEBUG 06:08:30 DEBUG 06:08:30 DEBUG 06:08:30 DEBUG [...] 06:08:30 DEBUG 06:08:30 DEBUG 06:08:30 DEBUG 06:08:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:08:30 DEBUG }, 06:08:30 DEBUG "value": [ 06:08:30 DEBUG 1712725710, 06:08:30 DEBUG "2487681024" 06:08:30 DEBUG ] 06:08:30 DEBUG } 06:08:30 DEBUG ] 06:08:30 DEBUG } 06:08:30 DEBUG } 06:08:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725710" 06:08:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725710" 06:08:30 INFO [http_cmd]: http status code OK 06:08:30 DEBUG --- status code --- 06:08:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:30 DEBUG --- http response --- 06:08:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:08:30 DEBUG ----- output ----- 06:08:30 DEBUG { 06:08:30 DEBUG "status": "success", 06:08:30 DEBUG "data": { 06:08:30 DEBUG "resultType": "vector", 06:08:30 DEBUG "result": [ 06:08:30 DEBUG { 06:08:30 DEBUG "metric": { 06:08:30 DEBUG "instance": "10.142.0.25:9100" 06:08:30 DEBUG }, 06:08:30 DEBUG "value": [ 06:08:30 DEBUG 06:08:30 DEBUG 06:08:30 DEBUG 06:08:30 DEBUG [...] 06:08:30 DEBUG 06:08:30 DEBUG 06:08:30 DEBUG 06:08:30 DEBUG "instance": "10.142.0.77:9100" 06:08:30 DEBUG }, 06:08:30 DEBUG "value": [ 06:08:30 DEBUG 1712725710, 06:08:30 DEBUG "42882533.83333333" 06:08:30 DEBUG ] 06:08:30 DEBUG } 06:08:30 DEBUG ] 06:08:30 DEBUG } 06:08:30 DEBUG } 06:08:30 INFO [http_cmd]: http status code OK 06:08:30 DEBUG --- status code --- 06:08:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:30 DEBUG --- http response --- 06:08:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:08:30 DEBUG ----- output ----- 06:08:30 DEBUG { 06:08:30 DEBUG "status": "success", 06:08:30 DEBUG "data": { 06:08:30 DEBUG "resultType": "vector", 06:08:30 DEBUG "result": [ 06:08:30 DEBUG { 06:08:30 DEBUG "metric": { 06:08:30 DEBUG "instance": "10.142.0.25:9100" 06:08:30 DEBUG }, 06:08:30 DEBUG "value": [ 06:08:30 DEBUG 06:08:30 DEBUG 06:08:30 DEBUG 06:08:30 DEBUG [...] 06:08:30 DEBUG 06:08:30 DEBUG 06:08:30 DEBUG 06:08:30 DEBUG "instance": "10.142.0.77:9100" 06:08:30 DEBUG }, 06:08:30 DEBUG "value": [ 06:08:30 DEBUG 1712725710, 06:08:30 DEBUG "0.02299999999999975" 06:08:30 DEBUG ] 06:08:30 DEBUG } 06:08:30 DEBUG ] 06:08:30 DEBUG } 06:08:30 DEBUG } 06:08:47 INFO 06:08:47 INFO [loop_until]: kubectl --namespace=xlou top pods 06:08:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:08:48 INFO [loop_until]: OK (rc = 0) 06:08:48 DEBUG --- stdout --- 06:08:48 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10298m 8968Mi am-654c4fcbcf-9scrj 10184m 8445Mi am-654c4fcbcf-v5qb5 8346m 8441Mi ds-cts-0 8626m 12658Mi ds-cts-1 8460m 13272Mi ds-cts-2 8173m 13394Mi ds-idrepo-0 10628m 5622Mi ds-idrepo-1 354m 5672Mi ds-idrepo-2 373m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 9m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4191m 2238Mi 06:08:48 DEBUG --- stderr --- 06:08:48 DEBUG 06:08:55 INFO 06:08:55 INFO [loop_until]: kubectl --namespace=xlou top node 06:08:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:08:55 INFO [loop_until]: OK (rc = 0) 06:08:55 DEBUG --- stdout --- 06:08:55 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 94m 0% 2295Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9308m 58% 9313Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 80m 0% 3260Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13137m 82% 6506Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10936m 68% 9327Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11303m 71% 9791Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 75m 0% 856Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10853m 68% 6292Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8618m 54% 13275Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 8088m 50% 14055Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 428m 2% 6383Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8837m 55% 14037Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 471m 2% 6346Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4243m 26% 5880Mi 10% 06:08:55 DEBUG --- stderr --- 06:08:55 DEBUG 06:09:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712725767" 06:09:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725767" 06:09:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725767" 06:09:27 INFO [http_cmd]: http status code OK 06:09:27 INFO [http_cmd]: http status code OK 06:09:27 DEBUG --- status code --- 06:09:27 DEBUG --- status code --- 06:09:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:27 DEBUG --- http response --- 06:09:27 DEBUG --- http response --- 06:09:27 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:09:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:09:27 DEBUG ----- output ----- 06:09:27 DEBUG { 06:09:27 DEBUG "status": "success", 06:09:27 DEBUG "data": { 06:09:27 DEBUG "resultType": "vector", 06:09:27 DEBUG "result": [ 06:09:27 DEBUG { 06:09:27 DEBUG "metric": { 06:09:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:09:27 DEBUG }, 06:09:27 DEBUG "value": [ 06:09:27 DEBUG 06:09:27 DEBUG 06:09:27 DEBUG 06:09:27 DEBUG [...] 06:09:27 DEBUG 06:09:27 DEBUG 06:09:27 DEBUG 06:09:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:09:27 DEBUG }, 06:09:27 DEBUG "value": [ 06:09:27 DEBUG 1712725767, 06:09:27 DEBUG "0" 06:09:27 DEBUG ] 06:09:27 DEBUG } 06:09:27 DEBUG ] 06:09:27 DEBUG } 06:09:27 DEBUG } 06:09:27 ERROR [http_cmd]: ERROR 06:09:27 ERROR --- status code --- 06:09:27 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:09:27 ERROR --- http response --- 06:09:27 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:09:27 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:09:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725767" 06:09:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725767" 06:09:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725767" 06:09:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725767" 06:09:27 INFO [http_cmd]: http status code OK 06:09:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725767" 06:09:27 DEBUG --- status code --- 06:09:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725767" 06:09:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725767" 06:09:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:27 DEBUG --- http response --- 06:09:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:09:27 DEBUG ----- output ----- 06:09:27 DEBUG { 06:09:27 DEBUG "status": "success", 06:09:27 DEBUG "data": { 06:09:27 DEBUG "resultType": "vector", 06:09:27 DEBUG "result": [ 06:09:27 DEBUG { 06:09:27 DEBUG "metric": { 06:09:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:09:27 DEBUG }, 06:09:27 DEBUG "value": [ 06:09:27 DEBUG 06:09:27 DEBUG 06:09:27 DEBUG 06:09:27 DEBUG [...] 06:09:27 DEBUG 06:09:27 DEBUG 06:09:27 DEBUG 06:09:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:09:27 DEBUG }, 06:09:27 DEBUG "value": [ 06:09:27 INFO [http_cmd]: http status code OK 06:09:27 DEBUG 1712725767, 06:09:27 INFO [http_cmd]: http status code OK 06:09:27 DEBUG --- status code --- 06:09:27 DEBUG "0" 06:09:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:27 DEBUG --- status code --- 06:09:27 DEBUG ] 06:09:27 DEBUG --- http response --- 06:09:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:27 DEBUG } 06:09:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:09:27 DEBUG --- http response --- 06:09:27 DEBUG ] 06:09:27 DEBUG ----- output ----- 06:09:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:09:27 INFO [http_cmd]: http status code OK 06:09:27 INFO [http_cmd]: http status code OK 06:09:27 DEBUG } 06:09:27 DEBUG } 06:09:27 INFO [http_cmd]: http status code OK 06:09:27 DEBUG { 06:09:27 DEBUG ----- output ----- 06:09:27 DEBUG --- status code --- 06:09:27 DEBUG --- status code --- 06:09:27 INFO [http_cmd]: http status code OK 06:09:27 DEBUG --- status code --- 06:09:27 DEBUG "status": "success", 06:09:27 DEBUG { 06:09:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:27 DEBUG --- status code --- 06:09:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:27 DEBUG "data": { 06:09:27 DEBUG "status": "success", 06:09:27 DEBUG --- http response --- 06:09:27 DEBUG --- http response --- 06:09:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:27 DEBUG --- http response --- 06:09:27 DEBUG "resultType": "vector", 06:09:27 DEBUG "data": { 06:09:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:09:27 DEBUG ----- output ----- 06:09:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725767" 06:09:27 DEBUG --- http response --- 06:09:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:09:27 DEBUG "result": [ 06:09:27 DEBUG "resultType": "vector", 06:09:27 DEBUG "result": [ 06:09:27 DEBUG { 06:09:27 DEBUG "metric": { 06:09:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:09:27 DEBUG { 06:09:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:09:27 DEBUG }, 06:09:27 DEBUG { 06:09:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:09:27 DEBUG "metric": { 06:09:27 DEBUG "pod": "ds-idrepo-0" 06:09:27 DEBUG ----- output ----- 06:09:27 DEBUG { 06:09:27 DEBUG "status": "success", 06:09:27 DEBUG ----- output ----- 06:09:27 DEBUG ----- output ----- 06:09:27 DEBUG }, 06:09:27 DEBUG "value": [ 06:09:27 DEBUG "status": "success", 06:09:27 DEBUG "data": { 06:09:27 DEBUG { 06:09:27 DEBUG { 06:09:27 DEBUG "value": [ 06:09:27 DEBUG 06:09:27 DEBUG "resultType": "vector", 06:09:27 DEBUG "result": [ 06:09:27 DEBUG "data": { 06:09:27 DEBUG "status": "success", 06:09:27 DEBUG 06:09:27 DEBUG 06:09:27 DEBUG { 06:09:27 DEBUG "status": "success", 06:09:27 DEBUG "data": { 06:09:27 DEBUG "data": { 06:09:27 DEBUG 06:09:27 DEBUG 06:09:27 DEBUG "metric": { 06:09:27 DEBUG "pod": "ds-idrepo-0" 06:09:27 DEBUG }, 06:09:27 DEBUG "resultType": "vector", 06:09:27 DEBUG 06:09:27 DEBUG [...] 06:09:27 DEBUG "resultType": "vector", 06:09:27 DEBUG "result": [ 06:09:27 DEBUG "value": [ 06:09:27 DEBUG "result": [ 06:09:27 DEBUG [...] 06:09:27 DEBUG 06:09:27 DEBUG "resultType": "vector", 06:09:27 DEBUG { 06:09:27 DEBUG 06:09:27 DEBUG { 06:09:27 DEBUG 06:09:27 DEBUG 06:09:27 DEBUG "result": [ 06:09:27 DEBUG "metric": { 06:09:27 DEBUG 06:09:27 DEBUG "metric": { 06:09:27 DEBUG 06:09:27 DEBUG 06:09:27 DEBUG { 06:09:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:09:27 DEBUG 06:09:27 DEBUG "instance": "10.142.0.25:9100" 06:09:27 DEBUG 06:09:27 DEBUG "pod": "ds-cts-2" 06:09:27 DEBUG "metric": { 06:09:27 DEBUG }, 06:09:27 DEBUG [...] 06:09:27 DEBUG }, 06:09:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:09:27 DEBUG }, 06:09:27 DEBUG "instance": "10.142.0.25:9100" 06:09:27 DEBUG "value": [ 06:09:27 DEBUG 06:09:27 DEBUG "value": [ 06:09:27 DEBUG }, 06:09:27 DEBUG "value": [ 06:09:27 DEBUG }, 06:09:27 DEBUG 06:09:27 DEBUG 06:09:27 DEBUG 06:09:27 DEBUG "value": [ 06:09:27 DEBUG 1712725767, 06:09:27 DEBUG "value": [ 06:09:27 DEBUG [...] 06:09:27 DEBUG 06:09:27 DEBUG 06:09:27 DEBUG 1712725767, 06:09:27 DEBUG "0" 06:09:27 DEBUG 06:09:27 DEBUG 06:09:27 DEBUG 06:09:27 DEBUG 06:09:27 DEBUG [...] 06:09:27 DEBUG 06:09:27 DEBUG 06:09:27 DEBUG 06:09:27 DEBUG 06:09:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:09:27 DEBUG 06:09:27 DEBUG 06:09:27 DEBUG ] 06:09:27 DEBUG }, 06:09:27 DEBUG "value": [ 06:09:27 DEBUG 1712725767, 06:09:27 DEBUG 06:09:27 DEBUG "instance": "10.142.0.77:9100" 06:09:27 DEBUG } 06:09:27 DEBUG "1434.8" 06:09:27 DEBUG 06:09:27 DEBUG "258906.5" 06:09:27 DEBUG [...] 06:09:27 DEBUG }, 06:09:27 DEBUG ] 06:09:27 DEBUG ] 06:09:27 DEBUG "pod": "ds-cts-2" 06:09:27 DEBUG ] 06:09:27 DEBUG 06:09:27 DEBUG "value": [ 06:09:27 DEBUG } 06:09:27 DEBUG } 06:09:27 DEBUG }, 06:09:27 DEBUG } 06:09:27 DEBUG ] 06:09:27 DEBUG 1712725767, 06:09:27 DEBUG "359040539648" 06:09:27 DEBUG ] 06:09:27 DEBUG "value": [ 06:09:27 DEBUG } 06:09:27 DEBUG ] 06:09:27 DEBUG } 06:09:27 DEBUG } 06:09:27 DEBUG 06:09:27 INFO [http_cmd]: http status code OK 06:09:27 DEBUG 1712725767, 06:09:27 DEBUG "29.89402119576085" 06:09:27 DEBUG ] 06:09:27 DEBUG ] 06:09:27 DEBUG 06:09:27 DEBUG --- status code --- 06:09:27 DEBUG } 06:09:27 DEBUG } 06:09:27 DEBUG } 06:09:27 DEBUG } 06:09:27 DEBUG "instance": "10.142.0.77:9100" 06:09:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:27 DEBUG } 06:09:27 DEBUG ] 06:09:27 DEBUG } 06:09:27 DEBUG }, 06:09:27 DEBUG --- http response --- 06:09:27 DEBUG } 06:09:27 DEBUG } 06:09:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:09:27 DEBUG ----- output ----- 06:09:27 DEBUG { 06:09:27 DEBUG "status": "success", 06:09:27 DEBUG "data": { 06:09:27 DEBUG "resultType": "vector", 06:09:27 DEBUG "result": [ 06:09:27 DEBUG { 06:09:27 DEBUG "metric": { 06:09:27 DEBUG "pod": "ds-idrepo-0" 06:09:27 DEBUG }, 06:09:27 DEBUG "value": [ 06:09:27 DEBUG 06:09:27 DEBUG 06:09:27 DEBUG 06:09:27 DEBUG [...] 06:09:27 DEBUG 06:09:27 DEBUG 06:09:27 DEBUG 06:09:27 DEBUG "pod": "ds-idrepo-1" 06:09:27 DEBUG }, 06:09:27 DEBUG "value": [ 06:09:27 DEBUG 1712725767, 06:09:27 DEBUG "0" 06:09:27 DEBUG ] 06:09:27 DEBUG } 06:09:27 DEBUG ] 06:09:27 DEBUG } 06:09:27 DEBUG } 06:09:27 DEBUG "value": [ 06:09:27 DEBUG 1712725767, 06:09:27 DEBUG "42529272.1" 06:09:27 DEBUG ] 06:09:27 DEBUG } 06:09:27 DEBUG ] 06:09:27 DEBUG } 06:09:27 DEBUG } 06:09:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725767" 06:09:27 INFO [http_cmd]: http status code OK 06:09:27 DEBUG --- status code --- 06:09:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:27 DEBUG --- http response --- 06:09:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:09:27 DEBUG ----- output ----- 06:09:27 DEBUG { 06:09:27 DEBUG "status": "success", 06:09:27 DEBUG "data": { 06:09:27 DEBUG "resultType": "vector", 06:09:27 DEBUG "result": [ 06:09:27 DEBUG { 06:09:27 DEBUG "metric": { 06:09:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:09:27 DEBUG }, 06:09:27 DEBUG "value": [ 06:09:27 DEBUG 06:09:27 DEBUG 06:09:27 DEBUG 06:09:27 DEBUG [...] 06:09:27 DEBUG 06:09:27 DEBUG 06:09:27 DEBUG 06:09:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:09:27 DEBUG }, 06:09:27 DEBUG "value": [ 06:09:27 DEBUG 1712725767, 06:09:27 DEBUG "1435.8666666666666" 06:09:27 DEBUG ] 06:09:27 DEBUG } 06:09:27 DEBUG ] 06:09:27 DEBUG } 06:09:27 DEBUG } 06:09:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725768" 06:09:28 INFO [http_cmd]: http status code OK 06:09:28 DEBUG --- status code --- 06:09:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:28 DEBUG --- http response --- 06:09:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:09:28 DEBUG ----- output ----- 06:09:28 DEBUG { 06:09:28 DEBUG "status": "success", 06:09:28 DEBUG "data": { 06:09:28 DEBUG "resultType": "vector", 06:09:28 DEBUG "result": [ 06:09:28 DEBUG { 06:09:28 DEBUG "metric": { 06:09:28 DEBUG "instance": "10.142.0.25:9100" 06:09:28 DEBUG }, 06:09:28 DEBUG "value": [ 06:09:28 DEBUG 06:09:28 DEBUG 06:09:28 DEBUG 06:09:28 DEBUG [...] 06:09:28 DEBUG 06:09:28 DEBUG 06:09:28 DEBUG 06:09:28 DEBUG "instance": "10.142.0.77:9100" 06:09:28 DEBUG }, 06:09:28 DEBUG "value": [ 06:09:28 DEBUG 1712725768, 06:09:28 DEBUG "0.14933333333333193" 06:09:28 DEBUG ] 06:09:28 DEBUG } 06:09:28 DEBUG ] 06:09:28 DEBUG } 06:09:28 DEBUG } 06:09:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725769" 06:09:29 INFO [http_cmd]: http status code OK 06:09:29 DEBUG --- status code --- 06:09:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:29 DEBUG --- http response --- 06:09:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:09:29 DEBUG ----- output ----- 06:09:29 DEBUG { 06:09:29 DEBUG "status": "success", 06:09:29 DEBUG "data": { 06:09:29 DEBUG "resultType": "vector", 06:09:29 DEBUG "result": [ 06:09:29 DEBUG { 06:09:29 DEBUG "metric": { 06:09:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:09:29 DEBUG }, 06:09:29 DEBUG "value": [ 06:09:29 DEBUG 06:09:29 DEBUG 06:09:29 DEBUG 06:09:29 DEBUG [...] 06:09:29 DEBUG 06:09:29 DEBUG 06:09:29 DEBUG 06:09:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:09:29 DEBUG }, 06:09:29 DEBUG "value": [ 06:09:29 DEBUG 1712725769, 06:09:29 DEBUG "84395.4241035665" 06:09:29 DEBUG ] 06:09:29 DEBUG } 06:09:29 DEBUG ] 06:09:29 DEBUG } 06:09:29 DEBUG } 06:09:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712725769" 06:09:29 INFO [http_cmd]: http status code OK 06:09:29 DEBUG --- status code --- 06:09:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:29 DEBUG --- http response --- 06:09:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:09:29 DEBUG ----- output ----- 06:09:29 DEBUG { 06:09:29 DEBUG "status": "success", 06:09:29 DEBUG "data": { 06:09:29 DEBUG "resultType": "vector", 06:09:29 DEBUG "result": [ 06:09:29 DEBUG { 06:09:29 DEBUG "metric": { 06:09:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:09:29 DEBUG }, 06:09:29 DEBUG "value": [ 06:09:29 DEBUG 06:09:29 DEBUG 06:09:29 DEBUG 06:09:29 DEBUG [...] 06:09:29 DEBUG 06:09:29 DEBUG 06:09:29 DEBUG 06:09:29 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:09:29 DEBUG }, 06:09:29 DEBUG "value": [ 06:09:29 DEBUG 1712725769, 06:09:29 DEBUG "4.145730393627353" 06:09:29 DEBUG ] 06:09:29 DEBUG } 06:09:29 DEBUG ] 06:09:29 DEBUG } 06:09:29 DEBUG } 06:09:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712725769" 06:09:29 INFO [http_cmd]: http status code OK 06:09:29 DEBUG --- status code --- 06:09:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:29 DEBUG --- http response --- 06:09:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:09:29 DEBUG ----- output ----- 06:09:29 DEBUG { 06:09:29 DEBUG "status": "success", 06:09:29 DEBUG "data": { 06:09:29 DEBUG "resultType": "vector", 06:09:29 DEBUG "result": [ 06:09:29 DEBUG { 06:09:29 DEBUG "metric": { 06:09:29 DEBUG "pod": "ds-idrepo-0" 06:09:29 DEBUG }, 06:09:29 DEBUG "value": [ 06:09:29 DEBUG 06:09:29 DEBUG 06:09:29 DEBUG 06:09:29 DEBUG [...] 06:09:29 DEBUG 06:09:29 DEBUG 06:09:29 DEBUG 06:09:29 DEBUG "pod": "ds-cts-2" 06:09:29 DEBUG }, 06:09:29 DEBUG "value": [ 06:09:29 DEBUG 1712725769, 06:09:29 DEBUG "0" 06:09:29 DEBUG ] 06:09:29 DEBUG } 06:09:29 DEBUG ] 06:09:29 DEBUG } 06:09:29 DEBUG } 06:09:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725769" 06:09:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725769" 06:09:29 INFO [http_cmd]: http status code OK 06:09:29 DEBUG --- status code --- 06:09:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:29 DEBUG --- http response --- 06:09:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:09:29 DEBUG ----- output ----- 06:09:29 DEBUG { 06:09:29 DEBUG "status": "success", 06:09:29 DEBUG "data": { 06:09:29 DEBUG "resultType": "vector", 06:09:29 DEBUG "result": [ 06:09:29 DEBUG { 06:09:29 DEBUG "metric": { 06:09:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:09:29 DEBUG }, 06:09:29 DEBUG "value": [ 06:09:29 DEBUG 06:09:29 DEBUG 06:09:29 DEBUG 06:09:29 DEBUG [...] 06:09:29 DEBUG 06:09:29 DEBUG 06:09:29 DEBUG 06:09:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:09:29 DEBUG }, 06:09:29 DEBUG "value": [ 06:09:29 DEBUG 1712725769, 06:09:29 DEBUG "8.444574700676187" 06:09:29 DEBUG ] 06:09:29 DEBUG } 06:09:29 DEBUG ] 06:09:29 DEBUG } 06:09:29 DEBUG } 06:09:29 INFO [http_cmd]: http status code OK 06:09:29 DEBUG --- status code --- 06:09:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:29 DEBUG --- http response --- 06:09:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:09:29 DEBUG ----- output ----- 06:09:29 DEBUG { 06:09:29 DEBUG "status": "success", 06:09:29 DEBUG "data": { 06:09:29 DEBUG "resultType": "vector", 06:09:29 DEBUG "result": [ 06:09:29 DEBUG { 06:09:29 DEBUG "metric": { 06:09:29 DEBUG "pod": "ds-idrepo-2" 06:09:29 DEBUG }, 06:09:29 DEBUG "value": [ 06:09:29 DEBUG 06:09:29 DEBUG 06:09:29 DEBUG 06:09:29 DEBUG [...] 06:09:29 DEBUG 06:09:29 DEBUG 06:09:29 DEBUG 06:09:29 DEBUG "pod": "ds-cts-2" 06:09:29 DEBUG }, 06:09:29 DEBUG "value": [ 06:09:29 DEBUG 1712725769, 06:09:29 DEBUG "0.000033326667999733394" 06:09:29 DEBUG ] 06:09:29 DEBUG } 06:09:29 DEBUG ] 06:09:29 DEBUG } 06:09:29 DEBUG } 06:09:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725769" 06:09:29 INFO [http_cmd]: http status code OK 06:09:29 DEBUG --- status code --- 06:09:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:29 DEBUG --- http response --- 06:09:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:09:29 DEBUG ----- output ----- 06:09:29 DEBUG { 06:09:29 DEBUG "status": "success", 06:09:29 DEBUG "data": { 06:09:29 DEBUG "resultType": "vector", 06:09:29 DEBUG "result": [ 06:09:29 DEBUG { 06:09:29 DEBUG "metric": { 06:09:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:09:29 DEBUG }, 06:09:29 DEBUG "value": [ 06:09:29 DEBUG 06:09:29 DEBUG 06:09:29 DEBUG 06:09:29 DEBUG [...] 06:09:29 DEBUG 06:09:29 DEBUG 06:09:29 DEBUG 06:09:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:09:29 DEBUG }, 06:09:29 DEBUG "value": [ 06:09:29 DEBUG 1712725769, 06:09:29 DEBUG "1436.5" 06:09:29 DEBUG ] 06:09:29 DEBUG } 06:09:29 DEBUG ] 06:09:29 DEBUG } 06:09:29 DEBUG } 06:09:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712725769" 06:09:29 INFO [http_cmd]: http status code OK 06:09:29 DEBUG --- status code --- 06:09:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:29 DEBUG --- http response --- 06:09:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:09:29 DEBUG ----- output ----- 06:09:29 DEBUG { 06:09:29 DEBUG "status": "success", 06:09:29 DEBUG "data": { 06:09:29 DEBUG "resultType": "vector", 06:09:29 DEBUG "result": [ 06:09:29 DEBUG { 06:09:29 DEBUG "metric": { 06:09:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:09:29 DEBUG }, 06:09:29 DEBUG "value": [ 06:09:29 DEBUG 06:09:29 DEBUG 06:09:29 DEBUG 06:09:29 DEBUG [...] 06:09:29 DEBUG 06:09:29 DEBUG 06:09:29 DEBUG 06:09:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:09:29 DEBUG }, 06:09:29 DEBUG "value": [ 06:09:29 DEBUG 1712725769, 06:09:29 DEBUG "0.048" 06:09:29 DEBUG ] 06:09:29 DEBUG } 06:09:29 DEBUG ] 06:09:29 DEBUG } 06:09:29 DEBUG } 06:09:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725770" 06:09:30 INFO [http_cmd]: http status code OK 06:09:30 DEBUG --- status code --- 06:09:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:30 DEBUG --- http response --- 06:09:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:09:30 DEBUG ----- output ----- 06:09:30 DEBUG { 06:09:30 DEBUG "status": "success", 06:09:30 DEBUG "data": { 06:09:30 DEBUG "resultType": "vector", 06:09:30 DEBUG "result": [ 06:09:30 DEBUG { 06:09:30 DEBUG "metric": { 06:09:30 DEBUG "pod": "ds-idrepo-0" 06:09:30 DEBUG }, 06:09:30 DEBUG "value": [ 06:09:30 DEBUG 06:09:30 DEBUG 06:09:30 DEBUG 06:09:30 DEBUG [...] 06:09:30 DEBUG 06:09:30 DEBUG 06:09:30 DEBUG 06:09:30 DEBUG "pod": "ds-idrepo-1" 06:09:30 DEBUG }, 06:09:30 DEBUG "value": [ 06:09:30 DEBUG 1712725770, 06:09:30 DEBUG "0" 06:09:30 DEBUG ] 06:09:30 DEBUG } 06:09:30 DEBUG ] 06:09:30 DEBUG } 06:09:30 DEBUG } 06:09:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725770" 06:09:30 INFO [http_cmd]: http status code OK 06:09:30 DEBUG --- status code --- 06:09:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:30 DEBUG --- http response --- 06:09:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:09:30 DEBUG ----- output ----- 06:09:30 DEBUG { 06:09:30 DEBUG "status": "success", 06:09:30 DEBUG "data": { 06:09:30 DEBUG "resultType": "vector", 06:09:30 DEBUG "result": [ 06:09:30 DEBUG { 06:09:30 DEBUG "metric": { 06:09:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:09:30 DEBUG }, 06:09:30 DEBUG "value": [ 06:09:30 DEBUG 06:09:30 DEBUG 06:09:30 DEBUG 06:09:30 DEBUG [...] 06:09:30 DEBUG 06:09:30 DEBUG 06:09:30 DEBUG 06:09:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:09:30 DEBUG }, 06:09:30 DEBUG "value": [ 06:09:30 DEBUG 1712725770, 06:09:30 DEBUG "1448.0333333333333" 06:09:30 DEBUG ] 06:09:30 DEBUG } 06:09:30 DEBUG ] 06:09:30 DEBUG } 06:09:30 DEBUG } 06:09:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725770" 06:09:30 INFO [http_cmd]: http status code OK 06:09:30 DEBUG --- status code --- 06:09:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:30 DEBUG --- http response --- 06:09:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:09:30 DEBUG ----- output ----- 06:09:30 DEBUG { 06:09:30 DEBUG "status": "success", 06:09:30 DEBUG "data": { 06:09:30 DEBUG "resultType": "vector", 06:09:30 DEBUG "result": [ 06:09:30 DEBUG { 06:09:30 DEBUG "metric": { 06:09:30 DEBUG "pod": "ds-idrepo-0" 06:09:30 DEBUG }, 06:09:30 DEBUG "value": [ 06:09:30 DEBUG 06:09:30 DEBUG 06:09:30 DEBUG 06:09:30 DEBUG [...] 06:09:30 DEBUG 06:09:30 DEBUG 06:09:30 DEBUG 06:09:30 DEBUG "pod": "ds-idrepo-1" 06:09:30 DEBUG }, 06:09:30 DEBUG "value": [ 06:09:30 DEBUG 1712725770, 06:09:30 DEBUG "0" 06:09:30 DEBUG ] 06:09:30 DEBUG } 06:09:30 DEBUG ] 06:09:30 DEBUG } 06:09:30 DEBUG } 06:09:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725770" 06:09:30 INFO [http_cmd]: http status code OK 06:09:30 DEBUG --- status code --- 06:09:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:30 DEBUG --- http response --- 06:09:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:09:30 DEBUG ----- output ----- 06:09:30 DEBUG { 06:09:30 DEBUG "status": "success", 06:09:30 DEBUG "data": { 06:09:30 DEBUG "resultType": "vector", 06:09:30 DEBUG "result": [ 06:09:30 DEBUG { 06:09:30 DEBUG "metric": { 06:09:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:09:30 DEBUG }, 06:09:30 DEBUG "value": [ 06:09:30 DEBUG 06:09:30 DEBUG 06:09:30 DEBUG 06:09:30 DEBUG [...] 06:09:30 DEBUG 06:09:30 DEBUG 06:09:30 DEBUG 06:09:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:09:30 DEBUG }, 06:09:30 DEBUG "value": [ 06:09:30 DEBUG 1712725770, 06:09:30 DEBUG "0" 06:09:30 DEBUG ] 06:09:30 DEBUG } 06:09:30 DEBUG ] 06:09:30 DEBUG } 06:09:30 DEBUG } 06:09:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712725770" 06:09:30 INFO [http_cmd]: http status code OK 06:09:30 DEBUG --- status code --- 06:09:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:30 DEBUG --- http response --- 06:09:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:09:30 DEBUG ----- output ----- 06:09:30 DEBUG { 06:09:30 DEBUG "status": "success", 06:09:30 DEBUG "data": { 06:09:30 DEBUG "resultType": "vector", 06:09:30 DEBUG "result": [ 06:09:30 DEBUG { 06:09:30 DEBUG "metric": { 06:09:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:09:30 DEBUG }, 06:09:30 DEBUG "value": [ 06:09:30 DEBUG 06:09:30 DEBUG 06:09:30 DEBUG 06:09:30 DEBUG [...] 06:09:30 DEBUG 06:09:30 DEBUG 06:09:30 DEBUG 06:09:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:09:30 DEBUG }, 06:09:30 DEBUG "value": [ 06:09:30 DEBUG 1712725770, 06:09:30 DEBUG "2485104640" 06:09:30 DEBUG ] 06:09:30 DEBUG } 06:09:30 DEBUG ] 06:09:30 DEBUG } 06:09:30 DEBUG } 06:09:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725770" 06:09:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725770" 06:09:30 INFO [http_cmd]: http status code OK 06:09:30 DEBUG --- status code --- 06:09:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:30 DEBUG --- http response --- 06:09:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:09:30 DEBUG ----- output ----- 06:09:30 DEBUG { 06:09:30 DEBUG "status": "success", 06:09:30 DEBUG "data": { 06:09:30 DEBUG "resultType": "vector", 06:09:30 DEBUG "result": [ 06:09:30 DEBUG { 06:09:30 DEBUG "metric": { 06:09:30 DEBUG "instance": "10.142.0.25:9100" 06:09:30 DEBUG }, 06:09:30 DEBUG "value": [ 06:09:30 DEBUG 06:09:30 DEBUG 06:09:30 DEBUG 06:09:30 DEBUG [...] 06:09:30 DEBUG 06:09:30 DEBUG 06:09:30 DEBUG 06:09:30 DEBUG "instance": "10.142.0.77:9100" 06:09:30 DEBUG }, 06:09:30 DEBUG "value": [ 06:09:30 DEBUG 1712725770, 06:09:30 DEBUG "42533267.56666666" 06:09:30 DEBUG ] 06:09:30 DEBUG } 06:09:30 DEBUG ] 06:09:30 DEBUG } 06:09:30 DEBUG } 06:09:30 INFO [http_cmd]: http status code OK 06:09:30 DEBUG --- status code --- 06:09:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:30 DEBUG --- http response --- 06:09:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:09:30 DEBUG ----- output ----- 06:09:30 DEBUG { 06:09:30 DEBUG "status": "success", 06:09:30 DEBUG "data": { 06:09:30 DEBUG "resultType": "vector", 06:09:30 DEBUG "result": [ 06:09:30 DEBUG { 06:09:30 DEBUG "metric": { 06:09:30 DEBUG "instance": "10.142.0.25:9100" 06:09:30 DEBUG }, 06:09:30 DEBUG "value": [ 06:09:30 DEBUG 06:09:30 DEBUG 06:09:30 DEBUG 06:09:30 DEBUG [...] 06:09:30 DEBUG 06:09:30 DEBUG 06:09:30 DEBUG 06:09:30 DEBUG "instance": "10.142.0.77:9100" 06:09:30 DEBUG }, 06:09:30 DEBUG "value": [ 06:09:30 DEBUG 1712725770, 06:09:30 DEBUG "0.050666666666666464" 06:09:30 DEBUG ] 06:09:30 DEBUG } 06:09:30 DEBUG ] 06:09:30 DEBUG } 06:09:30 DEBUG } 06:09:48 INFO 06:09:48 INFO [loop_until]: kubectl --namespace=xlou top pods 06:09:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:09:48 INFO [loop_until]: OK (rc = 0) 06:09:48 DEBUG --- stdout --- 06:09:48 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10166m 8964Mi am-654c4fcbcf-9scrj 9722m 8444Mi am-654c4fcbcf-v5qb5 8584m 8448Mi ds-cts-0 8413m 13276Mi ds-cts-1 8565m 12737Mi ds-cts-2 8093m 12963Mi ds-idrepo-0 10495m 5622Mi ds-idrepo-1 342m 5672Mi ds-idrepo-2 378m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 2m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4041m 2240Mi 06:09:48 DEBUG --- stderr --- 06:09:48 DEBUG 06:09:55 INFO 06:09:55 INFO [loop_until]: kubectl --namespace=xlou top node 06:09:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:09:55 INFO [loop_until]: OK (rc = 0) 06:09:55 DEBUG --- stdout --- 06:09:55 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 96m 0% 2312Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9373m 58% 9319Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 83m 0% 3254Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13163m 82% 6538Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10980m 69% 9332Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11064m 69% 9791Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 79m 0% 852Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10806m 68% 6309Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8625m 54% 13890Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 8105m 51% 13648Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 437m 2% 6382Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8641m 54% 13430Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 429m 2% 6347Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4251m 26% 5880Mi 10% 06:09:55 DEBUG --- stderr --- 06:09:55 DEBUG 06:10:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725827" 06:10:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725827" 06:10:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712725827" 06:10:27 INFO [http_cmd]: http status code OK 06:10:27 INFO [http_cmd]: http status code OK 06:10:27 ERROR [http_cmd]: ERROR 06:10:27 DEBUG --- status code --- 06:10:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:27 ERROR --- status code --- 06:10:27 DEBUG --- http response --- 06:10:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:10:27 DEBUG ----- output ----- 06:10:27 DEBUG { 06:10:27 DEBUG "status": "success", 06:10:27 DEBUG "data": { 06:10:27 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:10:27 DEBUG "resultType": "vector", 06:10:27 DEBUG "result": [ 06:10:27 ERROR --- http response --- 06:10:27 DEBUG { 06:10:27 DEBUG "metric": { 06:10:27 DEBUG --- status code --- 06:10:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:27 DEBUG --- http response --- 06:10:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:10:27 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:10:27 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:10:27 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:10:27 DEBUG }, 06:10:27 DEBUG "value": [ 06:10:27 DEBUG 06:10:27 DEBUG 06:10:27 DEBUG 06:10:27 DEBUG [...] 06:10:27 DEBUG 06:10:27 DEBUG 06:10:27 DEBUG 06:10:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:10:27 DEBUG }, 06:10:27 DEBUG "value": [ 06:10:27 DEBUG 1712725827, 06:10:27 DEBUG "0" 06:10:27 DEBUG ] 06:10:27 DEBUG } 06:10:27 DEBUG ] 06:10:27 DEBUG } 06:10:27 DEBUG } 06:10:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725827" 06:10:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725827" 06:10:27 INFO [http_cmd]: http status code OK 06:10:27 DEBUG --- status code --- 06:10:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725827" 06:10:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725827" 06:10:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725827" 06:10:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725827" 06:10:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725827" 06:10:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725827" 06:10:27 DEBUG --- http response --- 06:10:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:10:27 DEBUG ----- output ----- 06:10:27 DEBUG { 06:10:27 DEBUG "status": "success", 06:10:27 DEBUG "data": { 06:10:27 DEBUG "resultType": "vector", 06:10:27 DEBUG "result": [ 06:10:27 DEBUG { 06:10:27 INFO [http_cmd]: http status code OK 06:10:27 DEBUG --- status code --- 06:10:27 DEBUG "metric": { 06:10:27 INFO [http_cmd]: http status code OK 06:10:27 DEBUG --- status code --- 06:10:27 INFO [http_cmd]: http status code OK 06:10:27 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:10:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:27 DEBUG --- status code --- 06:10:27 DEBUG }, 06:10:27 DEBUG --- http response --- 06:10:27 DEBUG --- http response --- 06:10:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:27 DEBUG "value": [ 06:10:27 DEBUG 06:10:27 INFO [http_cmd]: http status code OK 06:10:27 DEBUG --- status code --- 06:10:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:27 DEBUG --- http response --- 06:10:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:10:27 DEBUG ----- output ----- 06:10:27 DEBUG { 06:10:27 DEBUG "status": "success", 06:10:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:10:27 DEBUG ----- output ----- 06:10:27 INFO [http_cmd]: http status code OK 06:10:27 DEBUG --- status code --- 06:10:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:27 DEBUG --- http response --- 06:10:27 DEBUG { 06:10:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:10:27 DEBUG --- http response --- 06:10:27 INFO [http_cmd]: http status code OK 06:10:27 DEBUG "data": { 06:10:27 DEBUG 06:10:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:10:27 DEBUG "status": "success", 06:10:27 INFO [http_cmd]: http status code OK 06:10:27 DEBUG ----- output ----- 06:10:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:10:27 DEBUG ----- output ----- 06:10:27 DEBUG "resultType": "vector", 06:10:27 DEBUG 06:10:27 DEBUG ----- output ----- 06:10:27 DEBUG "data": { 06:10:27 DEBUG --- status code --- 06:10:27 DEBUG { 06:10:27 DEBUG --- status code --- 06:10:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:27 DEBUG --- http response --- 06:10:27 DEBUG "result": [ 06:10:27 DEBUG { 06:10:27 DEBUG "resultType": "vector", 06:10:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:27 DEBUG "status": "success", 06:10:27 DEBUG "data": { 06:10:27 DEBUG [...] 06:10:27 DEBUG 06:10:27 DEBUG { 06:10:27 DEBUG "status": "success", 06:10:27 DEBUG "result": [ 06:10:27 DEBUG --- http response --- 06:10:27 DEBUG { 06:10:27 DEBUG "resultType": "vector", 06:10:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:10:27 DEBUG ----- output ----- 06:10:27 DEBUG "metric": { 06:10:27 DEBUG "data": { 06:10:27 DEBUG { 06:10:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:10:27 DEBUG "status": "success", 06:10:27 DEBUG "data": { 06:10:27 DEBUG "resultType": "vector", 06:10:27 DEBUG { 06:10:27 DEBUG "status": "success", 06:10:27 DEBUG "instance": "10.142.0.25:9100" 06:10:27 DEBUG "metric": { 06:10:27 DEBUG ----- output ----- 06:10:27 DEBUG "result": [ 06:10:27 DEBUG { 06:10:27 DEBUG "metric": { 06:10:27 DEBUG "pod": "ds-idrepo-0" 06:10:27 DEBUG }, 06:10:27 DEBUG "value": [ 06:10:27 DEBUG }, 06:10:27 DEBUG { 06:10:27 DEBUG 06:10:27 DEBUG "resultType": "vector", 06:10:27 DEBUG "data": { 06:10:27 DEBUG "result": [ 06:10:27 DEBUG { 06:10:27 DEBUG "metric": { 06:10:27 DEBUG "value": [ 06:10:27 DEBUG 06:10:27 DEBUG 06:10:27 DEBUG "result": [ 06:10:27 DEBUG { 06:10:27 DEBUG "resultType": "vector", 06:10:27 DEBUG "pod": "ds-idrepo-0" 06:10:27 DEBUG 06:10:27 DEBUG "status": "success", 06:10:27 DEBUG 06:10:27 DEBUG 06:10:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:10:27 DEBUG "metric": { 06:10:27 DEBUG "pod": "ds-idrepo-0" 06:10:27 DEBUG }, 06:10:27 DEBUG "value": [ 06:10:27 DEBUG 06:10:27 DEBUG 06:10:27 DEBUG [...] 06:10:27 DEBUG }, 06:10:27 DEBUG "result": [ 06:10:27 DEBUG }, 06:10:27 DEBUG "data": { 06:10:27 DEBUG 06:10:27 DEBUG 06:10:27 DEBUG [...] 06:10:27 DEBUG 06:10:27 DEBUG 06:10:27 DEBUG { 06:10:27 DEBUG "value": [ 06:10:27 DEBUG "resultType": "vector", 06:10:27 DEBUG 06:10:27 DEBUG [...] 06:10:27 DEBUG 06:10:27 DEBUG 06:10:27 DEBUG 06:10:27 DEBUG "metric": { 06:10:27 DEBUG 06:10:27 DEBUG "result": [ 06:10:27 DEBUG { 06:10:27 DEBUG "metric": { 06:10:27 DEBUG "instance": "10.142.0.25:9100" 06:10:27 DEBUG }, 06:10:27 DEBUG "value": [ 06:10:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:10:27 DEBUG 06:10:27 DEBUG 06:10:27 DEBUG [...] 06:10:27 DEBUG 06:10:27 DEBUG 06:10:27 DEBUG 06:10:27 DEBUG "pod": "ds-cts-2" 06:10:27 DEBUG }, 06:10:27 DEBUG "value": [ 06:10:27 DEBUG 1712725827, 06:10:27 DEBUG }, 06:10:27 DEBUG "value": [ 06:10:27 DEBUG 06:10:27 DEBUG "29.30097669922331" 06:10:27 DEBUG ] 06:10:27 DEBUG } 06:10:27 DEBUG ] 06:10:27 DEBUG 06:10:27 DEBUG 06:10:27 DEBUG } 06:10:27 DEBUG } 06:10:27 DEBUG 06:10:27 DEBUG 06:10:27 DEBUG "pod": "ds-cts-2" 06:10:27 DEBUG }, 06:10:27 DEBUG "value": [ 06:10:27 DEBUG 06:10:27 DEBUG "pod": "ds-idrepo-1" 06:10:27 DEBUG }, 06:10:27 DEBUG "value": [ 06:10:27 DEBUG 06:10:27 DEBUG 1712725827, 06:10:27 DEBUG "pod": "ds-idrepo-1" 06:10:27 DEBUG 06:10:27 DEBUG [...] 06:10:27 DEBUG 06:10:27 DEBUG 06:10:27 DEBUG 06:10:27 DEBUG "0" 06:10:27 DEBUG }, 06:10:27 DEBUG "value": [ 06:10:27 DEBUG [...] 06:10:27 DEBUG "instance": "10.142.0.77:9100" 06:10:27 DEBUG 1712725827, 06:10:27 DEBUG 06:10:27 DEBUG ] 06:10:27 DEBUG 1712725827, 06:10:27 DEBUG "value": [ 06:10:27 DEBUG 06:10:27 DEBUG 06:10:27 DEBUG "0" 06:10:27 DEBUG 06:10:27 DEBUG } 06:10:27 DEBUG "0" 06:10:27 DEBUG ] 06:10:27 DEBUG } 06:10:27 DEBUG 06:10:27 DEBUG ] 06:10:27 DEBUG "instance": "10.142.0.77:9100" 06:10:27 DEBUG ] 06:10:27 DEBUG }, 06:10:27 DEBUG "value": [ 06:10:27 DEBUG 1712725827, 06:10:27 DEBUG "42533461.7" 06:10:27 DEBUG } 06:10:27 DEBUG ] 06:10:27 DEBUG } 06:10:27 DEBUG } 06:10:27 DEBUG ] 06:10:27 DEBUG }, 06:10:27 DEBUG ] 06:10:27 DEBUG 06:10:27 DEBUG 06:10:27 DEBUG } 06:10:27 DEBUG ] 06:10:27 DEBUG "value": [ 06:10:27 DEBUG } 06:10:27 DEBUG 06:10:27 DEBUG } 06:10:27 DEBUG [...] 06:10:27 DEBUG } 06:10:27 DEBUG 1712725827, 06:10:27 DEBUG } 06:10:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:10:27 DEBUG }, 06:10:27 DEBUG 06:10:27 DEBUG } 06:10:27 DEBUG "360966876160" 06:10:27 DEBUG ] 06:10:27 DEBUG } 06:10:27 DEBUG 06:10:27 DEBUG "value": [ 06:10:27 DEBUG } 06:10:27 DEBUG ] 06:10:27 DEBUG } 06:10:27 DEBUG } 06:10:27 DEBUG 06:10:27 DEBUG 1712725827, 06:10:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:10:27 DEBUG "259148" 06:10:27 DEBUG }, 06:10:27 DEBUG "value": [ 06:10:27 DEBUG 1712725827, 06:10:27 DEBUG "1448.155184481552" 06:10:27 DEBUG ] 06:10:27 DEBUG } 06:10:27 DEBUG ] 06:10:27 DEBUG ] 06:10:27 DEBUG } 06:10:27 DEBUG } 06:10:27 DEBUG } 06:10:27 DEBUG ] 06:10:27 DEBUG } 06:10:27 DEBUG } 06:10:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725828" 06:10:28 INFO [http_cmd]: http status code OK 06:10:28 DEBUG --- status code --- 06:10:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:28 DEBUG --- http response --- 06:10:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:10:28 DEBUG ----- output ----- 06:10:28 DEBUG { 06:10:28 DEBUG "status": "success", 06:10:28 DEBUG "data": { 06:10:28 DEBUG "resultType": "vector", 06:10:28 DEBUG "result": [ 06:10:28 DEBUG { 06:10:28 DEBUG "metric": { 06:10:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:10:28 DEBUG }, 06:10:28 DEBUG "value": [ 06:10:28 DEBUG 06:10:28 DEBUG 06:10:28 DEBUG 06:10:28 DEBUG [...] 06:10:28 DEBUG 06:10:28 DEBUG 06:10:28 DEBUG 06:10:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:10:28 DEBUG }, 06:10:28 DEBUG "value": [ 06:10:28 DEBUG 1712725828, 06:10:28 DEBUG "1452.9213745292138" 06:10:28 DEBUG ] 06:10:28 DEBUG } 06:10:28 DEBUG ] 06:10:28 DEBUG } 06:10:28 DEBUG } 06:10:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725828" 06:10:28 INFO [http_cmd]: http status code OK 06:10:28 DEBUG --- status code --- 06:10:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:28 DEBUG --- http response --- 06:10:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:10:28 DEBUG ----- output ----- 06:10:28 DEBUG { 06:10:28 DEBUG "status": "success", 06:10:28 DEBUG "data": { 06:10:28 DEBUG "resultType": "vector", 06:10:28 DEBUG "result": [ 06:10:28 DEBUG { 06:10:28 DEBUG "metric": { 06:10:28 DEBUG "instance": "10.142.0.25:9100" 06:10:28 DEBUG }, 06:10:28 DEBUG "value": [ 06:10:28 DEBUG 06:10:28 DEBUG 06:10:28 DEBUG 06:10:28 DEBUG [...] 06:10:28 DEBUG 06:10:28 DEBUG 06:10:28 DEBUG 06:10:28 DEBUG "instance": "10.142.0.77:9100" 06:10:28 DEBUG }, 06:10:28 DEBUG "value": [ 06:10:28 DEBUG 1712725828, 06:10:28 DEBUG "0.16376666666667" 06:10:28 DEBUG ] 06:10:28 DEBUG } 06:10:28 DEBUG ] 06:10:28 DEBUG } 06:10:28 DEBUG } 06:10:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725829" 06:10:29 INFO [http_cmd]: http status code OK 06:10:29 DEBUG --- status code --- 06:10:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:29 DEBUG --- http response --- 06:10:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:10:29 DEBUG ----- output ----- 06:10:29 DEBUG { 06:10:29 DEBUG "status": "success", 06:10:29 DEBUG "data": { 06:10:29 DEBUG "resultType": "vector", 06:10:29 DEBUG "result": [ 06:10:29 DEBUG { 06:10:29 DEBUG "metric": { 06:10:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:10:29 DEBUG }, 06:10:29 DEBUG "value": [ 06:10:29 DEBUG 06:10:29 DEBUG 06:10:29 DEBUG 06:10:29 DEBUG [...] 06:10:29 DEBUG 06:10:29 DEBUG 06:10:29 DEBUG 06:10:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:10:29 DEBUG }, 06:10:29 DEBUG "value": [ 06:10:29 DEBUG 1712725829, 06:10:29 DEBUG "338303755.5572531" 06:10:29 DEBUG ] 06:10:29 DEBUG } 06:10:29 DEBUG ] 06:10:29 DEBUG } 06:10:29 DEBUG } 06:10:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712725829" 06:10:29 INFO [http_cmd]: http status code OK 06:10:29 DEBUG --- status code --- 06:10:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:29 DEBUG --- http response --- 06:10:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:10:29 DEBUG ----- output ----- 06:10:29 DEBUG { 06:10:29 DEBUG "status": "success", 06:10:29 DEBUG "data": { 06:10:29 DEBUG "resultType": "vector", 06:10:29 DEBUG "result": [ 06:10:29 DEBUG { 06:10:29 DEBUG "metric": { 06:10:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:10:29 DEBUG }, 06:10:29 DEBUG "value": [ 06:10:29 DEBUG 06:10:29 DEBUG 06:10:29 DEBUG 06:10:29 DEBUG [...] 06:10:29 DEBUG 06:10:29 DEBUG 06:10:29 DEBUG 06:10:29 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:10:29 DEBUG }, 06:10:29 DEBUG "value": [ 06:10:29 DEBUG 1712725829, 06:10:29 DEBUG "4.146275275500209" 06:10:29 DEBUG ] 06:10:29 DEBUG } 06:10:29 DEBUG ] 06:10:29 DEBUG } 06:10:29 DEBUG } 06:10:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712725829" 06:10:29 INFO [http_cmd]: http status code OK 06:10:29 DEBUG --- status code --- 06:10:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:29 DEBUG --- http response --- 06:10:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:10:29 DEBUG ----- output ----- 06:10:29 DEBUG { 06:10:29 DEBUG "status": "success", 06:10:29 DEBUG "data": { 06:10:29 DEBUG "resultType": "vector", 06:10:29 DEBUG "result": [ 06:10:29 DEBUG { 06:10:29 DEBUG "metric": { 06:10:29 DEBUG "pod": "ds-idrepo-0" 06:10:29 DEBUG }, 06:10:29 DEBUG "value": [ 06:10:29 DEBUG 06:10:29 DEBUG 06:10:29 DEBUG 06:10:29 DEBUG [...] 06:10:29 DEBUG 06:10:29 DEBUG 06:10:29 DEBUG 06:10:29 DEBUG "pod": "ds-cts-2" 06:10:29 DEBUG }, 06:10:29 DEBUG "value": [ 06:10:29 DEBUG 1712725829, 06:10:29 DEBUG "0" 06:10:29 DEBUG ] 06:10:29 DEBUG } 06:10:29 DEBUG ] 06:10:29 DEBUG } 06:10:29 DEBUG } 06:10:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725829" 06:10:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725829" 06:10:29 INFO [http_cmd]: http status code OK 06:10:29 INFO [http_cmd]: http status code OK 06:10:29 DEBUG --- status code --- 06:10:29 DEBUG --- status code --- 06:10:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:29 DEBUG --- http response --- 06:10:29 DEBUG --- http response --- 06:10:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:10:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:10:29 DEBUG ----- output ----- 06:10:29 DEBUG ----- output ----- 06:10:29 DEBUG { 06:10:29 DEBUG { 06:10:29 DEBUG "status": "success", 06:10:29 DEBUG "status": "success", 06:10:29 DEBUG "data": { 06:10:29 DEBUG "data": { 06:10:29 DEBUG "resultType": "vector", 06:10:29 DEBUG "resultType": "vector", 06:10:29 DEBUG "result": [ 06:10:29 DEBUG "result": [ 06:10:29 DEBUG { 06:10:29 DEBUG { 06:10:29 DEBUG "metric": { 06:10:29 DEBUG "metric": { 06:10:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:10:29 DEBUG "pod": "ds-idrepo-2" 06:10:29 DEBUG }, 06:10:29 DEBUG }, 06:10:29 DEBUG "value": [ 06:10:29 DEBUG "value": [ 06:10:29 DEBUG 06:10:29 DEBUG 06:10:29 DEBUG 06:10:29 DEBUG 06:10:29 DEBUG 06:10:29 DEBUG 06:10:29 DEBUG [...] 06:10:29 DEBUG [...] 06:10:29 DEBUG 06:10:29 DEBUG 06:10:29 DEBUG 06:10:29 DEBUG 06:10:29 DEBUG 06:10:29 DEBUG 06:10:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:10:29 DEBUG "pod": "ds-cts-2" 06:10:29 DEBUG }, 06:10:29 DEBUG }, 06:10:29 DEBUG "value": [ 06:10:29 DEBUG 1712725829, 06:10:29 DEBUG "8.44648327347004" 06:10:29 DEBUG ] 06:10:29 DEBUG } 06:10:29 DEBUG ] 06:10:29 DEBUG } 06:10:29 DEBUG } 06:10:29 DEBUG "value": [ 06:10:29 DEBUG 1712725829, 06:10:29 DEBUG "0.0004849311643721458" 06:10:29 DEBUG ] 06:10:29 DEBUG } 06:10:29 DEBUG ] 06:10:29 DEBUG } 06:10:29 DEBUG } 06:10:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725829" 06:10:29 INFO [http_cmd]: http status code OK 06:10:29 DEBUG --- status code --- 06:10:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:29 DEBUG --- http response --- 06:10:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:10:29 DEBUG ----- output ----- 06:10:29 DEBUG { 06:10:29 DEBUG "status": "success", 06:10:29 DEBUG "data": { 06:10:29 DEBUG "resultType": "vector", 06:10:29 DEBUG "result": [ 06:10:29 DEBUG { 06:10:29 DEBUG "metric": { 06:10:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:10:29 DEBUG }, 06:10:29 DEBUG "value": [ 06:10:29 DEBUG 06:10:29 DEBUG 06:10:29 DEBUG 06:10:29 DEBUG [...] 06:10:29 DEBUG 06:10:29 DEBUG 06:10:29 DEBUG 06:10:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:10:29 DEBUG }, 06:10:29 DEBUG "value": [ 06:10:29 DEBUG 1712725829, 06:10:29 DEBUG "1452.7880545278806" 06:10:29 DEBUG ] 06:10:29 DEBUG } 06:10:29 DEBUG ] 06:10:29 DEBUG } 06:10:29 DEBUG } 06:10:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712725829" 06:10:29 INFO [http_cmd]: http status code OK 06:10:29 DEBUG --- status code --- 06:10:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:29 DEBUG --- http response --- 06:10:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:10:29 DEBUG ----- output ----- 06:10:29 DEBUG { 06:10:29 DEBUG "status": "success", 06:10:29 DEBUG "data": { 06:10:29 DEBUG "resultType": "vector", 06:10:29 DEBUG "result": [ 06:10:29 DEBUG { 06:10:29 DEBUG "metric": { 06:10:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:10:29 DEBUG }, 06:10:29 DEBUG "value": [ 06:10:29 DEBUG 06:10:29 DEBUG 06:10:29 DEBUG 06:10:29 DEBUG [...] 06:10:29 DEBUG 06:10:29 DEBUG 06:10:29 DEBUG 06:10:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:10:29 DEBUG }, 06:10:29 DEBUG "value": [ 06:10:29 DEBUG 1712725829, 06:10:29 DEBUG "0.049" 06:10:29 DEBUG ] 06:10:29 DEBUG } 06:10:29 DEBUG ] 06:10:29 DEBUG } 06:10:29 DEBUG } 06:10:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725830" 06:10:30 INFO [http_cmd]: http status code OK 06:10:30 DEBUG --- status code --- 06:10:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:30 DEBUG --- http response --- 06:10:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:10:30 DEBUG ----- output ----- 06:10:30 DEBUG { 06:10:30 DEBUG "status": "success", 06:10:30 DEBUG "data": { 06:10:30 DEBUG "resultType": "vector", 06:10:30 DEBUG "result": [ 06:10:30 DEBUG { 06:10:30 DEBUG "metric": { 06:10:30 DEBUG "pod": "ds-idrepo-0" 06:10:30 DEBUG }, 06:10:30 DEBUG "value": [ 06:10:30 DEBUG 06:10:30 DEBUG 06:10:30 DEBUG 06:10:30 DEBUG [...] 06:10:30 DEBUG 06:10:30 DEBUG 06:10:30 DEBUG 06:10:30 DEBUG "pod": "ds-idrepo-1" 06:10:30 DEBUG }, 06:10:30 DEBUG "value": [ 06:10:30 DEBUG 1712725830, 06:10:30 DEBUG "0" 06:10:30 DEBUG ] 06:10:30 DEBUG } 06:10:30 DEBUG ] 06:10:30 DEBUG } 06:10:30 DEBUG } 06:10:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725830" 06:10:30 INFO [http_cmd]: http status code OK 06:10:30 DEBUG --- status code --- 06:10:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:30 DEBUG --- http response --- 06:10:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:10:30 DEBUG ----- output ----- 06:10:30 DEBUG { 06:10:30 DEBUG "status": "success", 06:10:30 DEBUG "data": { 06:10:30 DEBUG "resultType": "vector", 06:10:30 DEBUG "result": [ 06:10:30 DEBUG { 06:10:30 DEBUG "metric": { 06:10:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:10:30 DEBUG }, 06:10:30 DEBUG "value": [ 06:10:30 DEBUG 06:10:30 DEBUG 06:10:30 DEBUG 06:10:30 DEBUG [...] 06:10:30 DEBUG 06:10:30 DEBUG 06:10:30 DEBUG 06:10:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:10:30 DEBUG }, 06:10:30 DEBUG "value": [ 06:10:30 DEBUG 1712725830, 06:10:30 DEBUG "1422.8910442289105" 06:10:30 DEBUG ] 06:10:30 DEBUG } 06:10:30 DEBUG ] 06:10:30 DEBUG } 06:10:30 DEBUG } 06:10:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725830" 06:10:30 INFO [http_cmd]: http status code OK 06:10:30 DEBUG --- status code --- 06:10:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:30 DEBUG --- http response --- 06:10:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:10:30 DEBUG ----- output ----- 06:10:30 DEBUG { 06:10:30 DEBUG "status": "success", 06:10:30 DEBUG "data": { 06:10:30 DEBUG "resultType": "vector", 06:10:30 DEBUG "result": [ 06:10:30 DEBUG { 06:10:30 DEBUG "metric": { 06:10:30 DEBUG "pod": "ds-idrepo-0" 06:10:30 DEBUG }, 06:10:30 DEBUG "value": [ 06:10:30 DEBUG 06:10:30 DEBUG 06:10:30 DEBUG 06:10:30 DEBUG [...] 06:10:30 DEBUG 06:10:30 DEBUG 06:10:30 DEBUG 06:10:30 DEBUG "pod": "ds-idrepo-1" 06:10:30 DEBUG }, 06:10:30 DEBUG "value": [ 06:10:30 DEBUG 1712725830, 06:10:30 DEBUG "0" 06:10:30 DEBUG ] 06:10:30 DEBUG } 06:10:30 DEBUG ] 06:10:30 DEBUG } 06:10:30 DEBUG } 06:10:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725830" 06:10:30 INFO [http_cmd]: http status code OK 06:10:30 DEBUG --- status code --- 06:10:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:30 DEBUG --- http response --- 06:10:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:10:30 DEBUG ----- output ----- 06:10:30 DEBUG { 06:10:30 DEBUG "status": "success", 06:10:30 DEBUG "data": { 06:10:30 DEBUG "resultType": "vector", 06:10:30 DEBUG "result": [ 06:10:30 DEBUG { 06:10:30 DEBUG "metric": { 06:10:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:10:30 DEBUG }, 06:10:30 DEBUG "value": [ 06:10:30 DEBUG 06:10:30 DEBUG 06:10:30 DEBUG 06:10:30 DEBUG [...] 06:10:30 DEBUG 06:10:30 DEBUG 06:10:30 DEBUG 06:10:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:10:30 DEBUG }, 06:10:30 DEBUG "value": [ 06:10:30 DEBUG 1712725830, 06:10:30 DEBUG "0" 06:10:30 DEBUG ] 06:10:30 DEBUG } 06:10:30 DEBUG ] 06:10:30 DEBUG } 06:10:30 DEBUG } 06:10:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712725830" 06:10:30 INFO [http_cmd]: http status code OK 06:10:30 DEBUG --- status code --- 06:10:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:30 DEBUG --- http response --- 06:10:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:10:30 DEBUG ----- output ----- 06:10:30 DEBUG { 06:10:30 DEBUG "status": "success", 06:10:30 DEBUG "data": { 06:10:30 DEBUG "resultType": "vector", 06:10:30 DEBUG "result": [ 06:10:30 DEBUG { 06:10:30 DEBUG "metric": { 06:10:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:10:30 DEBUG }, 06:10:30 DEBUG "value": [ 06:10:30 DEBUG 06:10:30 DEBUG 06:10:30 DEBUG 06:10:30 DEBUG [...] 06:10:30 DEBUG 06:10:30 DEBUG 06:10:30 DEBUG 06:10:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:10:30 DEBUG }, 06:10:30 DEBUG "value": [ 06:10:30 DEBUG 1712725830, 06:10:30 DEBUG "2484621312" 06:10:30 DEBUG ] 06:10:30 DEBUG } 06:10:30 DEBUG ] 06:10:30 DEBUG } 06:10:30 DEBUG } 06:10:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725830" 06:10:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725830" 06:10:30 INFO [http_cmd]: http status code OK 06:10:30 DEBUG --- status code --- 06:10:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:30 DEBUG --- http response --- 06:10:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:10:30 DEBUG ----- output ----- 06:10:30 DEBUG { 06:10:30 DEBUG "status": "success", 06:10:30 DEBUG "data": { 06:10:30 DEBUG "resultType": "vector", 06:10:30 DEBUG "result": [ 06:10:30 DEBUG { 06:10:30 DEBUG "metric": { 06:10:30 DEBUG "instance": "10.142.0.25:9100" 06:10:30 DEBUG }, 06:10:30 DEBUG "value": [ 06:10:30 DEBUG 06:10:30 DEBUG 06:10:30 DEBUG 06:10:30 DEBUG [...] 06:10:30 DEBUG 06:10:30 DEBUG 06:10:30 DEBUG 06:10:30 DEBUG "instance": "10.142.0.77:9100" 06:10:30 DEBUG }, 06:10:30 DEBUG "value": [ 06:10:30 DEBUG 1712725830, 06:10:30 DEBUG "42535611.166666664" 06:10:30 DEBUG ] 06:10:30 DEBUG } 06:10:30 DEBUG ] 06:10:30 DEBUG } 06:10:30 DEBUG } 06:10:30 INFO [http_cmd]: http status code OK 06:10:30 DEBUG --- status code --- 06:10:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:30 DEBUG --- http response --- 06:10:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:10:30 DEBUG ----- output ----- 06:10:30 DEBUG { 06:10:30 DEBUG "status": "success", 06:10:30 DEBUG "data": { 06:10:30 DEBUG "resultType": "vector", 06:10:30 DEBUG "result": [ 06:10:30 DEBUG { 06:10:30 DEBUG "metric": { 06:10:30 DEBUG "instance": "10.142.0.25:9100" 06:10:30 DEBUG }, 06:10:30 DEBUG "value": [ 06:10:30 DEBUG 06:10:30 DEBUG 06:10:30 DEBUG 06:10:30 DEBUG [...] 06:10:30 DEBUG 06:10:30 DEBUG 06:10:30 DEBUG 06:10:30 DEBUG "instance": "10.142.0.77:9100" 06:10:30 DEBUG }, 06:10:30 DEBUG "value": [ 06:10:30 DEBUG 1712725830, 06:10:30 DEBUG "0.05266666666666715" 06:10:30 DEBUG ] 06:10:30 DEBUG } 06:10:30 DEBUG ] 06:10:30 DEBUG } 06:10:30 DEBUG } 06:10:48 INFO 06:10:48 INFO [loop_until]: kubectl --namespace=xlou top pods 06:10:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:10:48 INFO [loop_until]: OK (rc = 0) 06:10:48 DEBUG --- stdout --- 06:10:48 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10506m 8991Mi am-654c4fcbcf-9scrj 9753m 8450Mi am-654c4fcbcf-v5qb5 8570m 8451Mi ds-cts-0 8242m 13386Mi ds-cts-1 8311m 13391Mi ds-cts-2 7897m 13394Mi ds-idrepo-0 10560m 5622Mi ds-idrepo-1 403m 5671Mi ds-idrepo-2 392m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 8m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4106m 2243Mi 06:10:48 DEBUG --- stderr --- 06:10:48 DEBUG 06:10:56 INFO 06:10:56 INFO [loop_until]: kubectl --namespace=xlou top node 06:10:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:10:56 INFO [loop_until]: OK (rc = 0) 06:10:56 DEBUG --- stdout --- 06:10:56 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 97m 0% 2308Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9261m 58% 9319Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 78m 0% 3257Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13133m 82% 6608Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10518m 66% 9330Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11547m 72% 9801Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 80m 0% 853Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10805m 67% 6295Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8521m 53% 14058Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 8131m 51% 13217Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 449m 2% 6385Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8682m 54% 14068Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 447m 2% 6348Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4149m 26% 5875Mi 10% 06:10:56 DEBUG --- stderr --- 06:10:56 DEBUG 06:11:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725887" 06:11:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712725887" 06:11:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725887" 06:11:27 INFO [http_cmd]: http status code OK 06:11:27 ERROR [http_cmd]: ERROR 06:11:27 INFO [http_cmd]: http status code OK 06:11:27 DEBUG --- status code --- 06:11:27 ERROR --- status code --- 06:11:27 DEBUG --- status code --- 06:11:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:27 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:11:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:27 DEBUG --- http response --- 06:11:27 ERROR --- http response --- 06:11:27 DEBUG --- http response --- 06:11:27 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:11:27 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:11:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:11:27 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:11:27 DEBUG ----- output ----- 06:11:27 DEBUG { 06:11:27 DEBUG "status": "success", 06:11:27 DEBUG "data": { 06:11:27 DEBUG "resultType": "vector", 06:11:27 DEBUG "result": [ 06:11:27 DEBUG { 06:11:27 DEBUG "metric": { 06:11:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:11:27 DEBUG }, 06:11:27 DEBUG "value": [ 06:11:27 DEBUG 06:11:27 DEBUG 06:11:27 DEBUG 06:11:27 DEBUG [...] 06:11:27 DEBUG 06:11:27 DEBUG 06:11:27 DEBUG 06:11:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:11:27 DEBUG }, 06:11:27 DEBUG "value": [ 06:11:27 DEBUG 1712725887, 06:11:27 DEBUG "0" 06:11:27 DEBUG ] 06:11:27 DEBUG } 06:11:27 DEBUG ] 06:11:27 DEBUG } 06:11:27 DEBUG } 06:11:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725888" 06:11:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725888" 06:11:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725888" 06:11:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725888" 06:11:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725888" 06:11:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725888" 06:11:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725888" 06:11:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725888" 06:11:28 INFO [http_cmd]: http status code OK 06:11:28 INFO [http_cmd]: http status code OK 06:11:28 DEBUG --- status code --- 06:11:28 DEBUG --- status code --- 06:11:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:28 DEBUG --- http response --- 06:11:28 DEBUG --- http response --- 06:11:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:11:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:11:28 DEBUG ----- output ----- 06:11:28 DEBUG ----- output ----- 06:11:28 DEBUG { 06:11:28 DEBUG { 06:11:28 DEBUG "status": "success", 06:11:28 DEBUG "status": "success", 06:11:28 DEBUG "data": { 06:11:28 DEBUG "data": { 06:11:28 DEBUG "resultType": "vector", 06:11:28 INFO [http_cmd]: http status code OK 06:11:28 DEBUG "resultType": "vector", 06:11:28 INFO [http_cmd]: http status code OK 06:11:28 DEBUG --- status code --- 06:11:28 INFO [http_cmd]: http status code OK 06:11:28 DEBUG "result": [ 06:11:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725888" 06:11:28 DEBUG --- status code --- 06:11:28 DEBUG "result": [ 06:11:28 INFO [http_cmd]: http status code OK 06:11:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:28 DEBUG --- status code --- 06:11:28 DEBUG { 06:11:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:28 DEBUG --- http response --- 06:11:28 DEBUG { 06:11:28 DEBUG --- status code --- 06:11:28 DEBUG --- http response --- 06:11:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:28 DEBUG "metric": { 06:11:28 INFO [http_cmd]: http status code OK 06:11:28 INFO [http_cmd]: http status code OK 06:11:28 DEBUG --- status code --- 06:11:28 DEBUG "metric": { 06:11:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:11:28 DEBUG --- http response --- 06:11:28 DEBUG "pod": "ds-idrepo-0" 06:11:28 DEBUG --- status code --- 06:11:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 06:11:28 DEBUG "pod": "ds-idrepo-0" 06:11:28 DEBUG --- http response --- 06:11:28 DEBUG ----- output ----- 06:11:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:11:28 DEBUG }, 06:11:28 DEBUG "value": [ 06:11:28 DEBUG 06:11:28 DEBUG ----- output ----- 06:11:28 DEBUG }, 06:11:28 DEBUG "value": [ 06:11:28 DEBUG 06:11:28 DEBUG ----- output ----- 06:11:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:28 DEBUG --- http response --- 06:11:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:11:28 DEBUG ----- output ----- 06:11:28 DEBUG { 06:11:28 DEBUG "status": "success", 06:11:28 DEBUG "data": { 06:11:28 DEBUG { 06:11:28 DEBUG --- http response --- 06:11:28 DEBUG { 06:11:28 DEBUG { 06:11:28 DEBUG "status": "success", 06:11:28 DEBUG "data": { 06:11:28 DEBUG "resultType": "vector", 06:11:28 DEBUG "result": [ 06:11:28 DEBUG "status": "success", 06:11:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:11:28 DEBUG "status": "success", 06:11:28 DEBUG "data": { 06:11:28 DEBUG "resultType": "vector", 06:11:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:11:28 DEBUG ----- output ----- 06:11:28 DEBUG { 06:11:28 DEBUG "data": { 06:11:28 DEBUG ----- output ----- 06:11:28 DEBUG "result": [ 06:11:28 DEBUG { 06:11:28 DEBUG "metric": { 06:11:28 DEBUG "status": "success", 06:11:28 DEBUG "data": { 06:11:28 DEBUG "resultType": "vector", 06:11:28 DEBUG "resultType": "vector", 06:11:28 DEBUG { 06:11:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:11:28 DEBUG }, 06:11:28 DEBUG "value": [ 06:11:28 DEBUG 06:11:28 DEBUG 06:11:28 DEBUG 06:11:28 DEBUG "result": [ 06:11:28 DEBUG "status": "success", 06:11:28 DEBUG "data": { 06:11:28 DEBUG "resultType": "vector", 06:11:28 DEBUG "result": [ 06:11:28 DEBUG { 06:11:28 DEBUG "metric": { 06:11:28 DEBUG 06:11:28 DEBUG { 06:11:28 DEBUG { 06:11:28 DEBUG "metric": { 06:11:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:11:28 DEBUG }, 06:11:28 DEBUG "value": [ 06:11:28 DEBUG "resultType": "vector", 06:11:28 DEBUG [...] 06:11:28 DEBUG "metric": { 06:11:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:11:28 DEBUG "result": [ 06:11:28 DEBUG 06:11:28 DEBUG 06:11:28 DEBUG 06:11:28 DEBUG "result": [ 06:11:28 DEBUG 06:11:28 DEBUG "instance": "10.142.0.25:9100" 06:11:28 DEBUG }, 06:11:28 DEBUG { 06:11:28 DEBUG [...] 06:11:28 DEBUG 06:11:28 DEBUG 06:11:28 DEBUG { 06:11:28 DEBUG 06:11:28 DEBUG }, 06:11:28 DEBUG "value": [ 06:11:28 INFO [http_cmd]: http status code OK 06:11:28 DEBUG "metric": { 06:11:28 DEBUG 06:11:28 DEBUG 06:11:28 DEBUG [...] 06:11:28 DEBUG "metric": { 06:11:28 DEBUG 06:11:28 DEBUG "value": [ 06:11:28 DEBUG 06:11:28 DEBUG --- status code --- 06:11:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:28 DEBUG --- http response --- 06:11:28 DEBUG 06:11:28 DEBUG 06:11:28 DEBUG "pod": "ds-idrepo-0" 06:11:28 DEBUG "pod": "ds-cts-2" 06:11:28 DEBUG 06:11:28 DEBUG 06:11:28 DEBUG 06:11:28 DEBUG 06:11:28 DEBUG [...] 06:11:28 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:11:28 DEBUG 06:11:28 DEBUG }, 06:11:28 DEBUG }, 06:11:28 DEBUG 06:11:28 DEBUG [...] 06:11:28 DEBUG 06:11:28 DEBUG 06:11:28 DEBUG 06:11:28 DEBUG }, 06:11:28 DEBUG 06:11:28 DEBUG "value": [ 06:11:28 DEBUG 06:11:28 DEBUG 06:11:28 DEBUG [...] 06:11:28 DEBUG 06:11:28 DEBUG 06:11:28 DEBUG 06:11:28 DEBUG "value": [ 06:11:28 DEBUG "pod": "ds-idrepo-1" 06:11:28 DEBUG }, 06:11:28 DEBUG 06:11:28 DEBUG "instance": "10.142.0.77:9100" 06:11:28 DEBUG }, 06:11:28 DEBUG "value": [ 06:11:28 DEBUG 1712725888, 06:11:28 DEBUG "42881888.5" 06:11:28 DEBUG 1712725888, 06:11:28 DEBUG "value": [ 06:11:28 DEBUG 1712725888, 06:11:28 DEBUG 06:11:28 DEBUG ] 06:11:28 DEBUG } 06:11:28 DEBUG ] 06:11:28 DEBUG } 06:11:28 DEBUG } 06:11:28 DEBUG "0" 06:11:28 DEBUG "0" 06:11:28 DEBUG ] 06:11:28 DEBUG [...] 06:11:28 DEBUG 06:11:28 DEBUG 06:11:28 DEBUG 06:11:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:11:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:11:28 DEBUG } 06:11:28 DEBUG ] 06:11:28 DEBUG 06:11:28 DEBUG 06:11:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:11:28 DEBUG }, 06:11:28 DEBUG "value": [ 06:11:28 DEBUG 1712725888, 06:11:28 DEBUG "1450.4666666666667" 06:11:28 DEBUG ] 06:11:28 DEBUG } 06:11:28 DEBUG 06:11:28 DEBUG ] 06:11:28 DEBUG ] 06:11:28 DEBUG }, 06:11:28 DEBUG "value": [ 06:11:28 DEBUG 1712725888, 06:11:28 DEBUG ----- output ----- 06:11:28 DEBUG { 06:11:28 DEBUG "pod": "ds-cts-2" 06:11:28 DEBUG } 06:11:28 DEBUG ] 06:11:28 DEBUG "258577.5" 06:11:28 DEBUG ] 06:11:28 DEBUG } 06:11:28 DEBUG ] 06:11:28 DEBUG } 06:11:28 DEBUG } 06:11:28 DEBUG "status": "success", 06:11:28 DEBUG "instance": "10.142.0.25:9100" 06:11:28 DEBUG }, 06:11:28 DEBUG "value": [ 06:11:28 DEBUG } 06:11:28 DEBUG "value": [ 06:11:28 DEBUG 1712725888, 06:11:28 DEBUG "29.8" 06:11:28 DEBUG }, 06:11:28 DEBUG "value": [ 06:11:28 DEBUG 1712725888, 06:11:28 DEBUG "0" 06:11:28 DEBUG ] 06:11:28 DEBUG } 06:11:28 DEBUG ] 06:11:28 DEBUG } 06:11:28 DEBUG } 06:11:28 DEBUG "data": { 06:11:28 DEBUG "resultType": "vector", 06:11:28 DEBUG "result": [ 06:11:28 DEBUG { 06:11:28 DEBUG "metric": { 06:11:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:11:28 DEBUG }, 06:11:28 DEBUG "value": [ 06:11:28 DEBUG 06:11:28 DEBUG 06:11:28 DEBUG 06:11:28 DEBUG [...] 06:11:28 DEBUG 06:11:28 DEBUG 06:11:28 DEBUG 06:11:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:11:28 DEBUG }, 06:11:28 DEBUG "value": [ 06:11:28 DEBUG 1712725888, 06:11:28 DEBUG "1450.0333333333333" 06:11:28 DEBUG ] 06:11:28 DEBUG } 06:11:28 DEBUG ] 06:11:28 DEBUG } 06:11:28 DEBUG } 06:11:28 DEBUG } 06:11:28 DEBUG } 06:11:28 DEBUG 06:11:28 DEBUG 06:11:28 DEBUG 06:11:28 DEBUG [...] 06:11:28 DEBUG 06:11:28 DEBUG 06:11:28 DEBUG 06:11:28 DEBUG "instance": "10.142.0.77:9100" 06:11:28 DEBUG }, 06:11:28 DEBUG "value": [ 06:11:28 DEBUG 1712725888, 06:11:28 DEBUG "363278388224" 06:11:28 DEBUG ] 06:11:28 DEBUG } 06:11:28 DEBUG ] 06:11:28 DEBUG } 06:11:28 DEBUG } 06:11:28 DEBUG } 06:11:28 DEBUG } 06:11:28 DEBUG } 06:11:28 DEBUG ] 06:11:28 DEBUG } 06:11:28 DEBUG ] 06:11:28 DEBUG } 06:11:28 DEBUG } 06:11:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725888" 06:11:28 INFO [http_cmd]: http status code OK 06:11:28 DEBUG --- status code --- 06:11:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:28 DEBUG --- http response --- 06:11:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:11:28 DEBUG ----- output ----- 06:11:28 DEBUG { 06:11:28 DEBUG "status": "success", 06:11:28 DEBUG "data": { 06:11:28 DEBUG "resultType": "vector", 06:11:28 DEBUG "result": [ 06:11:28 DEBUG { 06:11:28 DEBUG "metric": { 06:11:28 DEBUG "instance": "10.142.0.25:9100" 06:11:28 DEBUG }, 06:11:28 DEBUG "value": [ 06:11:28 DEBUG 06:11:28 DEBUG 06:11:28 DEBUG 06:11:28 DEBUG [...] 06:11:28 DEBUG 06:11:28 DEBUG 06:11:28 DEBUG 06:11:28 DEBUG "instance": "10.142.0.77:9100" 06:11:28 DEBUG }, 06:11:28 DEBUG "value": [ 06:11:28 DEBUG 1712725888, 06:11:28 DEBUG "0.15879999999999783" 06:11:28 DEBUG ] 06:11:28 DEBUG } 06:11:28 DEBUG ] 06:11:28 DEBUG } 06:11:28 DEBUG } 06:11:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725889" 06:11:29 INFO [http_cmd]: http status code OK 06:11:29 DEBUG --- status code --- 06:11:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:29 DEBUG --- http response --- 06:11:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:11:29 DEBUG ----- output ----- 06:11:29 DEBUG { 06:11:29 DEBUG "status": "success", 06:11:29 DEBUG "data": { 06:11:29 DEBUG "resultType": "vector", 06:11:29 DEBUG "result": [ 06:11:29 DEBUG { 06:11:29 DEBUG "metric": { 06:11:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:11:29 DEBUG }, 06:11:29 DEBUG "value": [ 06:11:29 DEBUG 06:11:29 DEBUG 06:11:29 DEBUG 06:11:29 DEBUG [...] 06:11:29 DEBUG 06:11:29 DEBUG 06:11:29 DEBUG 06:11:29 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:11:29 DEBUG }, 06:11:29 DEBUG "value": [ 06:11:29 DEBUG 1712725889, 06:11:29 DEBUG "59245134.22192986" 06:11:29 DEBUG ] 06:11:29 DEBUG } 06:11:29 DEBUG ] 06:11:29 DEBUG } 06:11:29 DEBUG } 06:11:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712725889" 06:11:29 INFO [http_cmd]: http status code OK 06:11:29 DEBUG --- status code --- 06:11:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:29 DEBUG --- http response --- 06:11:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:11:29 DEBUG ----- output ----- 06:11:29 DEBUG { 06:11:29 DEBUG "status": "success", 06:11:29 DEBUG "data": { 06:11:29 DEBUG "resultType": "vector", 06:11:29 DEBUG "result": [ 06:11:29 DEBUG { 06:11:29 DEBUG "metric": { 06:11:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:11:29 DEBUG }, 06:11:29 DEBUG "value": [ 06:11:29 DEBUG 06:11:29 DEBUG 06:11:29 DEBUG 06:11:29 DEBUG [...] 06:11:29 DEBUG 06:11:29 DEBUG 06:11:29 DEBUG 06:11:29 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:11:29 DEBUG }, 06:11:29 DEBUG "value": [ 06:11:29 DEBUG 1712725889, 06:11:29 DEBUG "4.199707604769774" 06:11:29 DEBUG ] 06:11:29 DEBUG } 06:11:29 DEBUG ] 06:11:29 DEBUG } 06:11:29 DEBUG } 06:11:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712725889" 06:11:29 INFO [http_cmd]: http status code OK 06:11:29 DEBUG --- status code --- 06:11:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:29 DEBUG --- http response --- 06:11:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:11:29 DEBUG ----- output ----- 06:11:29 DEBUG { 06:11:29 DEBUG "status": "success", 06:11:29 DEBUG "data": { 06:11:29 DEBUG "resultType": "vector", 06:11:29 DEBUG "result": [ 06:11:29 DEBUG { 06:11:29 DEBUG "metric": { 06:11:29 DEBUG "pod": "ds-idrepo-0" 06:11:29 DEBUG }, 06:11:29 DEBUG "value": [ 06:11:29 DEBUG 06:11:29 DEBUG 06:11:29 DEBUG 06:11:29 DEBUG [...] 06:11:29 DEBUG 06:11:29 DEBUG 06:11:29 DEBUG 06:11:29 DEBUG "pod": "ds-cts-2" 06:11:29 DEBUG }, 06:11:29 DEBUG "value": [ 06:11:29 DEBUG 1712725889, 06:11:29 DEBUG "0" 06:11:29 DEBUG ] 06:11:29 DEBUG } 06:11:29 DEBUG ] 06:11:29 DEBUG } 06:11:29 DEBUG } 06:11:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725889" 06:11:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725889" 06:11:29 INFO [http_cmd]: http status code OK 06:11:29 DEBUG --- status code --- 06:11:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:29 DEBUG --- http response --- 06:11:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:11:29 DEBUG ----- output ----- 06:11:29 DEBUG { 06:11:29 DEBUG "status": "success", 06:11:29 DEBUG "data": { 06:11:29 DEBUG "resultType": "vector", 06:11:29 DEBUG "result": [ 06:11:29 DEBUG { 06:11:29 DEBUG "metric": { 06:11:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:11:29 DEBUG }, 06:11:29 DEBUG "value": [ 06:11:29 DEBUG 06:11:29 DEBUG 06:11:29 DEBUG 06:11:29 DEBUG [...] 06:11:29 DEBUG 06:11:29 DEBUG 06:11:29 DEBUG 06:11:29 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:11:29 DEBUG }, 06:11:29 DEBUG "value": [ 06:11:29 DEBUG 1712725889, 06:11:29 DEBUG "2.551031839075158" 06:11:29 DEBUG ] 06:11:29 DEBUG } 06:11:29 DEBUG ] 06:11:29 DEBUG } 06:11:29 DEBUG } 06:11:29 INFO [http_cmd]: http status code OK 06:11:29 DEBUG --- status code --- 06:11:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:29 DEBUG --- http response --- 06:11:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:11:29 DEBUG ----- output ----- 06:11:29 DEBUG { 06:11:29 DEBUG "status": "success", 06:11:29 DEBUG "data": { 06:11:29 DEBUG "resultType": "vector", 06:11:29 DEBUG "result": [ 06:11:29 DEBUG { 06:11:29 DEBUG "metric": { 06:11:29 DEBUG "pod": "ds-idrepo-2" 06:11:29 DEBUG }, 06:11:29 DEBUG "value": [ 06:11:29 DEBUG 06:11:29 DEBUG 06:11:29 DEBUG 06:11:29 DEBUG [...] 06:11:29 DEBUG 06:11:29 DEBUG 06:11:29 DEBUG 06:11:29 DEBUG "pod": "ds-cts-2" 06:11:29 DEBUG }, 06:11:29 DEBUG "value": [ 06:11:29 DEBUG 1712725889, 06:11:29 DEBUG "0.00004466000000000001" 06:11:29 DEBUG ] 06:11:29 DEBUG } 06:11:29 DEBUG ] 06:11:29 DEBUG } 06:11:29 DEBUG } 06:11:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725889" 06:11:29 INFO [http_cmd]: http status code OK 06:11:29 DEBUG --- status code --- 06:11:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:29 DEBUG --- http response --- 06:11:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:11:29 DEBUG ----- output ----- 06:11:29 DEBUG { 06:11:29 DEBUG "status": "success", 06:11:29 DEBUG "data": { 06:11:29 DEBUG "resultType": "vector", 06:11:29 DEBUG "result": [ 06:11:29 DEBUG { 06:11:29 DEBUG "metric": { 06:11:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:11:29 DEBUG }, 06:11:29 DEBUG "value": [ 06:11:29 DEBUG 06:11:29 DEBUG 06:11:29 DEBUG 06:11:29 DEBUG [...] 06:11:29 DEBUG 06:11:29 DEBUG 06:11:29 DEBUG 06:11:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:11:29 DEBUG }, 06:11:29 DEBUG "value": [ 06:11:29 DEBUG 1712725889, 06:11:29 DEBUG "1450.6666666666667" 06:11:29 DEBUG ] 06:11:29 DEBUG } 06:11:29 DEBUG ] 06:11:29 DEBUG } 06:11:29 DEBUG } 06:11:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712725889" 06:11:29 INFO [http_cmd]: http status code OK 06:11:29 DEBUG --- status code --- 06:11:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:29 DEBUG --- http response --- 06:11:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:11:29 DEBUG ----- output ----- 06:11:29 DEBUG { 06:11:29 DEBUG "status": "success", 06:11:29 DEBUG "data": { 06:11:29 DEBUG "resultType": "vector", 06:11:29 DEBUG "result": [ 06:11:29 DEBUG { 06:11:29 DEBUG "metric": { 06:11:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:11:29 DEBUG }, 06:11:29 DEBUG "value": [ 06:11:29 DEBUG 06:11:29 DEBUG 06:11:29 DEBUG 06:11:29 DEBUG [...] 06:11:29 DEBUG 06:11:29 DEBUG 06:11:29 DEBUG 06:11:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:11:29 DEBUG }, 06:11:29 DEBUG "value": [ 06:11:29 DEBUG 1712725889, 06:11:29 DEBUG "0.0495" 06:11:29 DEBUG ] 06:11:29 DEBUG } 06:11:29 DEBUG ] 06:11:29 DEBUG } 06:11:29 DEBUG } 06:11:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725890" 06:11:30 INFO [http_cmd]: http status code OK 06:11:30 DEBUG --- status code --- 06:11:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:30 DEBUG --- http response --- 06:11:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:11:30 DEBUG ----- output ----- 06:11:30 DEBUG { 06:11:30 DEBUG "status": "success", 06:11:30 DEBUG "data": { 06:11:30 DEBUG "resultType": "vector", 06:11:30 DEBUG "result": [ 06:11:30 DEBUG { 06:11:30 DEBUG "metric": { 06:11:30 DEBUG "pod": "ds-idrepo-0" 06:11:30 DEBUG }, 06:11:30 DEBUG "value": [ 06:11:30 DEBUG 06:11:30 DEBUG 06:11:30 DEBUG 06:11:30 DEBUG [...] 06:11:30 DEBUG 06:11:30 DEBUG 06:11:30 DEBUG 06:11:30 DEBUG "pod": "ds-idrepo-1" 06:11:30 DEBUG }, 06:11:30 DEBUG "value": [ 06:11:30 DEBUG 1712725890, 06:11:30 DEBUG "0" 06:11:30 DEBUG ] 06:11:30 DEBUG } 06:11:30 DEBUG ] 06:11:30 DEBUG } 06:11:30 DEBUG } 06:11:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725890" 06:11:30 INFO [http_cmd]: http status code OK 06:11:30 DEBUG --- status code --- 06:11:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:30 DEBUG --- http response --- 06:11:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:11:30 DEBUG ----- output ----- 06:11:30 DEBUG { 06:11:30 DEBUG "status": "success", 06:11:30 DEBUG "data": { 06:11:30 DEBUG "resultType": "vector", 06:11:30 DEBUG "result": [ 06:11:30 DEBUG { 06:11:30 DEBUG "metric": { 06:11:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:11:30 DEBUG }, 06:11:30 DEBUG "value": [ 06:11:30 DEBUG 06:11:30 DEBUG 06:11:30 DEBUG 06:11:30 DEBUG [...] 06:11:30 DEBUG 06:11:30 DEBUG 06:11:30 DEBUG 06:11:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:11:30 DEBUG }, 06:11:30 DEBUG "value": [ 06:11:30 DEBUG 1712725890, 06:11:30 DEBUG "1449.2333333333333" 06:11:30 DEBUG ] 06:11:30 DEBUG } 06:11:30 DEBUG ] 06:11:30 DEBUG } 06:11:30 DEBUG } 06:11:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725890" 06:11:30 INFO [http_cmd]: http status code OK 06:11:30 DEBUG --- status code --- 06:11:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:30 DEBUG --- http response --- 06:11:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:11:30 DEBUG ----- output ----- 06:11:30 DEBUG { 06:11:30 DEBUG "status": "success", 06:11:30 DEBUG "data": { 06:11:30 DEBUG "resultType": "vector", 06:11:30 DEBUG "result": [ 06:11:30 DEBUG { 06:11:30 DEBUG "metric": { 06:11:30 DEBUG "pod": "ds-idrepo-0" 06:11:30 DEBUG }, 06:11:30 DEBUG "value": [ 06:11:30 DEBUG 06:11:30 DEBUG 06:11:30 DEBUG 06:11:30 DEBUG [...] 06:11:30 DEBUG 06:11:30 DEBUG 06:11:30 DEBUG 06:11:30 DEBUG "pod": "ds-idrepo-1" 06:11:30 DEBUG }, 06:11:30 DEBUG "value": [ 06:11:30 DEBUG 1712725890, 06:11:30 DEBUG "0" 06:11:30 DEBUG ] 06:11:30 DEBUG } 06:11:30 DEBUG ] 06:11:30 DEBUG } 06:11:30 DEBUG } 06:11:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725890" 06:11:30 INFO [http_cmd]: http status code OK 06:11:30 DEBUG --- status code --- 06:11:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:30 DEBUG --- http response --- 06:11:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:11:30 DEBUG ----- output ----- 06:11:30 DEBUG { 06:11:30 DEBUG "status": "success", 06:11:30 DEBUG "data": { 06:11:30 DEBUG "resultType": "vector", 06:11:30 DEBUG "result": [ 06:11:30 DEBUG { 06:11:30 DEBUG "metric": { 06:11:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:11:30 DEBUG }, 06:11:30 DEBUG "value": [ 06:11:30 DEBUG 06:11:30 DEBUG 06:11:30 DEBUG 06:11:30 DEBUG [...] 06:11:30 DEBUG 06:11:30 DEBUG 06:11:30 DEBUG 06:11:30 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:11:30 DEBUG }, 06:11:30 DEBUG "value": [ 06:11:30 DEBUG 1712725890, 06:11:30 DEBUG "0" 06:11:30 DEBUG ] 06:11:30 DEBUG } 06:11:30 DEBUG ] 06:11:30 DEBUG } 06:11:30 DEBUG } 06:11:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712725890" 06:11:30 INFO [http_cmd]: http status code OK 06:11:30 DEBUG --- status code --- 06:11:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:30 DEBUG --- http response --- 06:11:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:11:30 DEBUG ----- output ----- 06:11:30 DEBUG { 06:11:30 DEBUG "status": "success", 06:11:30 DEBUG "data": { 06:11:30 DEBUG "resultType": "vector", 06:11:30 DEBUG "result": [ 06:11:30 DEBUG { 06:11:30 DEBUG "metric": { 06:11:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:11:30 DEBUG }, 06:11:30 DEBUG "value": [ 06:11:30 DEBUG 06:11:30 DEBUG 06:11:30 DEBUG 06:11:30 DEBUG [...] 06:11:30 DEBUG 06:11:30 DEBUG 06:11:30 DEBUG 06:11:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:11:30 DEBUG }, 06:11:30 DEBUG "value": [ 06:11:30 DEBUG 1712725890, 06:11:30 DEBUG "2489364480" 06:11:30 DEBUG ] 06:11:30 DEBUG } 06:11:30 DEBUG ] 06:11:30 DEBUG } 06:11:30 DEBUG } 06:11:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725890" 06:11:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725890" 06:11:30 INFO [http_cmd]: http status code OK 06:11:30 DEBUG --- status code --- 06:11:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:30 DEBUG --- http response --- 06:11:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:11:30 DEBUG ----- output ----- 06:11:30 DEBUG { 06:11:30 DEBUG "status": "success", 06:11:30 DEBUG "data": { 06:11:30 DEBUG "resultType": "vector", 06:11:30 DEBUG "result": [ 06:11:30 DEBUG { 06:11:30 DEBUG "metric": { 06:11:30 DEBUG "instance": "10.142.0.25:9100" 06:11:30 DEBUG }, 06:11:30 DEBUG "value": [ 06:11:30 DEBUG 06:11:30 DEBUG 06:11:30 DEBUG 06:11:30 DEBUG [...] 06:11:30 DEBUG 06:11:30 DEBUG 06:11:30 DEBUG 06:11:30 DEBUG "instance": "10.142.0.77:9100" 06:11:30 DEBUG }, 06:11:30 DEBUG "value": [ 06:11:30 DEBUG 1712725890, 06:11:30 DEBUG "42884331.16666667" 06:11:30 DEBUG ] 06:11:30 DEBUG } 06:11:30 DEBUG ] 06:11:30 DEBUG } 06:11:30 DEBUG } 06:11:30 INFO [http_cmd]: http status code OK 06:11:30 DEBUG --- status code --- 06:11:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:30 DEBUG --- http response --- 06:11:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:11:30 DEBUG ----- output ----- 06:11:30 DEBUG { 06:11:30 DEBUG "status": "success", 06:11:30 DEBUG "data": { 06:11:30 DEBUG "resultType": "vector", 06:11:30 DEBUG "result": [ 06:11:30 DEBUG { 06:11:30 DEBUG "metric": { 06:11:30 DEBUG "instance": "10.142.0.25:9100" 06:11:30 DEBUG }, 06:11:30 DEBUG "value": [ 06:11:30 DEBUG 06:11:30 DEBUG 06:11:30 DEBUG 06:11:30 DEBUG [...] 06:11:30 DEBUG 06:11:30 DEBUG 06:11:30 DEBUG 06:11:30 DEBUG "instance": "10.142.0.77:9100" 06:11:30 DEBUG }, 06:11:30 DEBUG "value": [ 06:11:30 DEBUG 1712725890, 06:11:30 DEBUG "0.0529999999999997" 06:11:30 DEBUG ] 06:11:30 DEBUG } 06:11:30 DEBUG ] 06:11:30 DEBUG } 06:11:30 DEBUG } 06:11:48 INFO 06:11:48 INFO [loop_until]: kubectl --namespace=xlou top pods 06:11:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:11:48 INFO [loop_until]: OK (rc = 0) 06:11:48 DEBUG --- stdout --- 06:11:48 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10626m 8998Mi am-654c4fcbcf-9scrj 10060m 8452Mi am-654c4fcbcf-v5qb5 8539m 8447Mi ds-cts-0 8360m 12702Mi ds-cts-1 8637m 12873Mi ds-cts-2 7983m 13091Mi ds-idrepo-0 10732m 5622Mi ds-idrepo-1 368m 5671Mi ds-idrepo-2 384m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 5Mi idm-6bfb7c96cc-qklk9 4m 1191Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 5m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4099m 2248Mi 06:11:48 DEBUG --- stderr --- 06:11:48 DEBUG 06:11:56 INFO 06:11:56 INFO [loop_until]: kubectl --namespace=xlou top node 06:11:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:11:56 INFO [loop_until]: OK (rc = 0) 06:11:56 DEBUG --- stdout --- 06:11:56 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 96m 0% 2306Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8980m 56% 9318Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 80m 0% 3260Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 12980m 81% 6595Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10977m 69% 9338Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11927m 75% 9805Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 78m 0% 857Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10843m 68% 6298Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8532m 53% 13381Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 8295m 52% 14007Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 447m 2% 6383Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8491m 53% 13600Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 425m 2% 6350Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4289m 26% 5875Mi 10% 06:11:56 DEBUG --- stderr --- 06:11:56 DEBUG 06:12:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725947" 06:12:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712725947" 06:12:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725947" 06:12:27 ERROR [http_cmd]: ERROR 06:12:27 INFO [http_cmd]: http status code OK 06:12:27 INFO [http_cmd]: http status code OK 06:12:27 ERROR --- status code --- 06:12:27 DEBUG --- status code --- 06:12:27 DEBUG --- status code --- 06:12:27 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:12:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:27 ERROR --- http response --- 06:12:27 DEBUG --- http response --- 06:12:27 DEBUG --- http response --- 06:12:27 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:12:27 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:12:27 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:12:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:12:27 DEBUG ----- output ----- 06:12:27 DEBUG { 06:12:27 DEBUG "status": "success", 06:12:27 DEBUG "data": { 06:12:27 DEBUG "resultType": "vector", 06:12:27 DEBUG "result": [ 06:12:27 DEBUG { 06:12:27 DEBUG "metric": { 06:12:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:12:27 DEBUG }, 06:12:27 DEBUG "value": [ 06:12:27 DEBUG 06:12:27 DEBUG 06:12:27 DEBUG 06:12:27 DEBUG [...] 06:12:27 DEBUG 06:12:27 DEBUG 06:12:27 DEBUG 06:12:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:12:27 DEBUG }, 06:12:27 DEBUG "value": [ 06:12:27 DEBUG 1712725947, 06:12:27 DEBUG "0" 06:12:27 DEBUG ] 06:12:27 DEBUG } 06:12:27 DEBUG ] 06:12:27 DEBUG } 06:12:27 DEBUG } 06:12:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725948" 06:12:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725948" 06:12:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725948" 06:12:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725948" 06:12:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725948" 06:12:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725948" 06:12:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725948" 06:12:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725948" 06:12:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725948" 06:12:28 INFO [http_cmd]: http status code OK 06:12:28 DEBUG --- status code --- 06:12:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:28 DEBUG --- http response --- 06:12:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:12:28 INFO [http_cmd]: http status code OK 06:12:28 DEBUG ----- output ----- 06:12:28 DEBUG --- status code --- 06:12:28 DEBUG { 06:12:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:28 DEBUG --- http response --- 06:12:28 DEBUG "status": "success", 06:12:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:12:28 DEBUG ----- output ----- 06:12:28 DEBUG "data": { 06:12:28 DEBUG { 06:12:28 DEBUG "status": "success", 06:12:28 DEBUG "data": { 06:12:28 DEBUG "resultType": "vector", 06:12:28 DEBUG "result": [ 06:12:28 DEBUG { 06:12:28 DEBUG "resultType": "vector", 06:12:28 DEBUG "metric": { 06:12:28 DEBUG "result": [ 06:12:28 INFO [http_cmd]: http status code OK 06:12:28 INFO [http_cmd]: http status code OK 06:12:28 INFO [http_cmd]: http status code OK 06:12:28 DEBUG "instance": "10.142.0.25:9100" 06:12:28 INFO [http_cmd]: http status code OK 06:12:28 DEBUG { 06:12:28 DEBUG --- status code --- 06:12:28 INFO [http_cmd]: http status code OK 06:12:28 DEBUG --- status code --- 06:12:28 DEBUG --- status code --- 06:12:28 DEBUG }, 06:12:28 DEBUG --- status code --- 06:12:28 INFO [http_cmd]: http status code OK 06:12:28 DEBUG "metric": { 06:12:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:28 DEBUG --- status code --- 06:12:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:28 DEBUG "value": [ 06:12:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:28 DEBUG --- status code --- 06:12:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:12:28 DEBUG --- http response --- 06:12:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:28 DEBUG --- http response --- 06:12:28 DEBUG --- http response --- 06:12:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:12:28 DEBUG ----- output ----- 06:12:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:28 DEBUG --- http response --- 06:12:28 DEBUG }, 06:12:28 DEBUG --- http response --- 06:12:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:12:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:12:28 DEBUG { 06:12:28 DEBUG 06:12:28 DEBUG 06:12:28 DEBUG 06:12:28 DEBUG "value": [ 06:12:28 INFO [http_cmd]: http status code OK 06:12:28 DEBUG --- status code --- 06:12:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:28 DEBUG ----- output ----- 06:12:28 DEBUG "status": "success", 06:12:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:12:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:12:28 DEBUG ----- output ----- 06:12:28 DEBUG { 06:12:28 DEBUG --- http response --- 06:12:28 DEBUG { 06:12:28 DEBUG "data": { 06:12:28 DEBUG "resultType": "vector", 06:12:28 DEBUG 06:12:28 DEBUG 06:12:28 DEBUG 06:12:28 DEBUG [...] 06:12:28 DEBUG "status": "success", 06:12:28 DEBUG "result": [ 06:12:28 DEBUG { 06:12:28 DEBUG "status": "success", 06:12:28 DEBUG [...] 06:12:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 06:12:28 DEBUG 06:12:28 DEBUG 06:12:28 DEBUG "metric": { 06:12:28 DEBUG "pod": "ds-idrepo-0" 06:12:28 DEBUG --- http response --- 06:12:28 DEBUG ----- output ----- 06:12:28 DEBUG "data": { 06:12:28 DEBUG 06:12:28 DEBUG ----- output ----- 06:12:28 DEBUG 06:12:28 DEBUG "data": { 06:12:28 DEBUG }, 06:12:28 DEBUG ----- output ----- 06:12:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:12:28 DEBUG { 06:12:28 DEBUG "status": "success", 06:12:28 DEBUG "resultType": "vector", 06:12:28 DEBUG "result": [ 06:12:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:12:28 DEBUG "resultType": "vector", 06:12:28 DEBUG "value": [ 06:12:28 DEBUG { 06:12:28 DEBUG ----- output ----- 06:12:28 DEBUG 06:12:28 DEBUG 06:12:28 DEBUG "data": { 06:12:28 DEBUG "resultType": "vector", 06:12:28 DEBUG }, 06:12:28 DEBUG "result": [ 06:12:28 DEBUG 06:12:28 DEBUG "status": "success", 06:12:28 DEBUG { 06:12:28 DEBUG "instance": "10.142.0.77:9100" 06:12:28 DEBUG }, 06:12:28 DEBUG { 06:12:28 DEBUG "result": [ 06:12:28 DEBUG "value": [ 06:12:28 DEBUG { 06:12:28 DEBUG 06:12:28 DEBUG 06:12:28 DEBUG "status": "success", 06:12:28 DEBUG "data": { 06:12:28 DEBUG { 06:12:28 DEBUG "status": "success", 06:12:28 DEBUG { 06:12:28 DEBUG 1712725948, 06:12:28 DEBUG "metric": { 06:12:28 DEBUG [...] 06:12:28 DEBUG "data": { 06:12:28 DEBUG "resultType": "vector", 06:12:28 DEBUG "value": [ 06:12:28 DEBUG "metric": { 06:12:28 DEBUG "data": { 06:12:28 DEBUG "metric": { 06:12:28 DEBUG "258507.5" 06:12:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:12:28 DEBUG 06:12:28 DEBUG "resultType": "vector", 06:12:28 DEBUG "result": [ 06:12:28 DEBUG { 06:12:28 DEBUG "pod": "ds-idrepo-0" 06:12:28 DEBUG "resultType": "vector", 06:12:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:12:28 DEBUG ] 06:12:28 DEBUG }, 06:12:28 DEBUG 06:12:28 DEBUG "result": [ 06:12:28 DEBUG "metric": { 06:12:28 DEBUG "instance": "10.142.0.25:9100" 06:12:28 DEBUG }, 06:12:28 DEBUG "result": [ 06:12:28 DEBUG }, 06:12:28 DEBUG } 06:12:28 DEBUG "value": [ 06:12:28 DEBUG 06:12:28 DEBUG { 06:12:28 DEBUG }, 06:12:28 DEBUG 1712725948, 06:12:28 DEBUG "value": [ 06:12:28 DEBUG { 06:12:28 DEBUG "value": [ 06:12:28 DEBUG 06:12:28 DEBUG 06:12:28 DEBUG "pod": "ds-idrepo-1" 06:12:28 DEBUG }, 06:12:28 DEBUG "value": [ 06:12:28 DEBUG "42297487.06666666" 06:12:28 DEBUG 06:12:28 DEBUG "metric": { 06:12:28 DEBUG ] 06:12:28 DEBUG 06:12:28 DEBUG 06:12:28 DEBUG "value": [ 06:12:28 DEBUG 1712725948, 06:12:28 DEBUG 06:12:28 DEBUG ] 06:12:28 DEBUG 06:12:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:12:28 DEBUG } 06:12:28 DEBUG } 06:12:28 DEBUG 06:12:28 DEBUG [...] 06:12:28 DEBUG "metric": { 06:12:28 DEBUG 06:12:28 DEBUG } 06:12:28 DEBUG 06:12:28 DEBUG }, 06:12:28 DEBUG 06:12:28 DEBUG "0" 06:12:28 DEBUG ] 06:12:28 DEBUG "pod": "ds-idrepo-0" 06:12:28 DEBUG }, 06:12:28 DEBUG ] 06:12:28 DEBUG [...] 06:12:28 DEBUG 06:12:28 DEBUG [...] 06:12:28 DEBUG } 06:12:28 DEBUG 06:12:28 DEBUG 06:12:28 DEBUG "value": [ 06:12:28 DEBUG } 06:12:28 DEBUG } 06:12:28 DEBUG 06:12:28 DEBUG 06:12:28 DEBUG ] 06:12:28 DEBUG 06:12:28 DEBUG [...] 06:12:28 DEBUG 06:12:28 DEBUG "value": [ 06:12:28 DEBUG 06:12:28 DEBUG "pod": "ds-cts-2" 06:12:28 DEBUG }, 06:12:28 DEBUG "value": [ 06:12:28 DEBUG 06:12:28 DEBUG 06:12:28 DEBUG 06:12:28 DEBUG 06:12:28 DEBUG } 06:12:28 DEBUG 1712725948, 06:12:28 DEBUG 06:12:28 DEBUG 06:12:28 DEBUG 06:12:28 DEBUG 06:12:28 DEBUG 06:12:28 DEBUG } 06:12:28 DEBUG "29.466666666666665" 06:12:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:12:28 DEBUG 06:12:28 DEBUG [...] 06:12:28 DEBUG 06:12:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:12:28 DEBUG ] 06:12:28 DEBUG }, 06:12:28 DEBUG "instance": "10.142.0.77:9100" 06:12:28 DEBUG }, 06:12:28 DEBUG "value": [ 06:12:28 DEBUG }, 06:12:28 DEBUG } 06:12:28 DEBUG "value": [ 06:12:28 DEBUG 1712725948, 06:12:28 DEBUG "365139002368" 06:12:28 DEBUG ] 06:12:28 DEBUG "value": [ 06:12:28 DEBUG ] 06:12:28 DEBUG 1712725948, 06:12:28 DEBUG 06:12:28 DEBUG 06:12:28 DEBUG 06:12:28 DEBUG "pod": "ds-cts-2" 06:12:28 DEBUG } 06:12:28 DEBUG } 06:12:28 DEBUG "1444.6333333333332" 06:12:28 DEBUG } 06:12:28 DEBUG ] 06:12:28 DEBUG } 06:12:28 DEBUG } 06:12:28 DEBUG ] 06:12:28 DEBUG [...] 06:12:28 DEBUG }, 06:12:28 DEBUG "value": [ 06:12:28 DEBUG } 06:12:28 DEBUG 06:12:28 DEBUG 1712725948, 06:12:28 DEBUG "0" 06:12:28 DEBUG ] 06:12:28 DEBUG 06:12:28 DEBUG 06:12:28 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:12:28 DEBUG } 06:12:28 DEBUG }, 06:12:28 DEBUG "value": [ 06:12:28 DEBUG 1712725948, 06:12:28 DEBUG "0" 06:12:28 DEBUG ] 06:12:28 DEBUG } 06:12:28 DEBUG } 06:12:28 DEBUG 1712725948, 06:12:28 DEBUG "1445" 06:12:28 DEBUG ] 06:12:28 DEBUG } 06:12:28 DEBUG ] 06:12:28 DEBUG } 06:12:28 DEBUG } 06:12:28 DEBUG ] 06:12:28 DEBUG ] 06:12:28 DEBUG } 06:12:28 DEBUG } 06:12:28 DEBUG ] 06:12:28 DEBUG } 06:12:28 DEBUG } 06:12:28 DEBUG } 06:12:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725948" 06:12:28 INFO [http_cmd]: http status code OK 06:12:28 DEBUG --- status code --- 06:12:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:28 DEBUG --- http response --- 06:12:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:12:28 DEBUG ----- output ----- 06:12:28 DEBUG { 06:12:28 DEBUG "status": "success", 06:12:28 DEBUG "data": { 06:12:28 DEBUG "resultType": "vector", 06:12:28 DEBUG "result": [ 06:12:28 DEBUG { 06:12:28 DEBUG "metric": { 06:12:28 DEBUG "instance": "10.142.0.25:9100" 06:12:28 DEBUG }, 06:12:28 DEBUG "value": [ 06:12:28 DEBUG 06:12:28 DEBUG 06:12:28 DEBUG 06:12:28 DEBUG [...] 06:12:28 DEBUG 06:12:28 DEBUG 06:12:28 DEBUG 06:12:28 DEBUG "instance": "10.142.0.77:9100" 06:12:28 DEBUG }, 06:12:28 DEBUG "value": [ 06:12:28 DEBUG 1712725948, 06:12:28 DEBUG "0.15323333333333455" 06:12:28 DEBUG ] 06:12:28 DEBUG } 06:12:28 DEBUG ] 06:12:28 DEBUG } 06:12:28 DEBUG } 06:12:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725949" 06:12:29 INFO [http_cmd]: http status code OK 06:12:29 DEBUG --- status code --- 06:12:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:29 DEBUG --- http response --- 06:12:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:12:29 DEBUG ----- output ----- 06:12:29 DEBUG { 06:12:29 DEBUG "status": "success", 06:12:29 DEBUG "data": { 06:12:29 DEBUG "resultType": "vector", 06:12:29 DEBUG "result": [ 06:12:29 DEBUG { 06:12:29 DEBUG "metric": { 06:12:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:12:29 DEBUG }, 06:12:29 DEBUG "value": [ 06:12:29 DEBUG 06:12:29 DEBUG 06:12:29 DEBUG 06:12:29 DEBUG [...] 06:12:29 DEBUG 06:12:29 DEBUG 06:12:29 DEBUG 06:12:29 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:12:29 DEBUG }, 06:12:29 DEBUG "value": [ 06:12:29 DEBUG 1712725949, 06:12:29 DEBUG "66668941.41061248" 06:12:29 DEBUG ] 06:12:29 DEBUG } 06:12:29 DEBUG ] 06:12:29 DEBUG } 06:12:29 DEBUG } 06:12:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712725949" 06:12:29 INFO [http_cmd]: http status code OK 06:12:29 DEBUG --- status code --- 06:12:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:29 DEBUG --- http response --- 06:12:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:12:29 DEBUG ----- output ----- 06:12:29 DEBUG { 06:12:29 DEBUG "status": "success", 06:12:29 DEBUG "data": { 06:12:29 DEBUG "resultType": "vector", 06:12:29 DEBUG "result": [ 06:12:29 DEBUG { 06:12:29 DEBUG "metric": { 06:12:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:12:29 DEBUG }, 06:12:29 DEBUG "value": [ 06:12:29 DEBUG 06:12:29 DEBUG 06:12:29 DEBUG 06:12:29 DEBUG [...] 06:12:29 DEBUG 06:12:29 DEBUG 06:12:29 DEBUG 06:12:29 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:12:29 DEBUG }, 06:12:29 DEBUG "value": [ 06:12:29 DEBUG 1712725949, 06:12:29 DEBUG "4.202418857339831" 06:12:29 DEBUG ] 06:12:29 DEBUG } 06:12:29 DEBUG ] 06:12:29 DEBUG } 06:12:29 DEBUG } 06:12:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712725949" 06:12:29 INFO [http_cmd]: http status code OK 06:12:29 DEBUG --- status code --- 06:12:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:29 DEBUG --- http response --- 06:12:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:12:29 DEBUG ----- output ----- 06:12:29 DEBUG { 06:12:29 DEBUG "status": "success", 06:12:29 DEBUG "data": { 06:12:29 DEBUG "resultType": "vector", 06:12:29 DEBUG "result": [ 06:12:29 DEBUG { 06:12:29 DEBUG "metric": { 06:12:29 DEBUG "pod": "ds-idrepo-0" 06:12:29 DEBUG }, 06:12:29 DEBUG "value": [ 06:12:29 DEBUG 06:12:29 DEBUG 06:12:29 DEBUG 06:12:29 DEBUG [...] 06:12:29 DEBUG 06:12:29 DEBUG 06:12:29 DEBUG 06:12:29 DEBUG "pod": "ds-cts-2" 06:12:29 DEBUG }, 06:12:29 DEBUG "value": [ 06:12:29 DEBUG 1712725949, 06:12:29 DEBUG "0" 06:12:29 DEBUG ] 06:12:29 DEBUG } 06:12:29 DEBUG ] 06:12:29 DEBUG } 06:12:29 DEBUG } 06:12:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725949" 06:12:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725949" 06:12:29 INFO [http_cmd]: http status code OK 06:12:29 INFO [http_cmd]: http status code OK 06:12:29 DEBUG --- status code --- 06:12:29 DEBUG --- status code --- 06:12:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:29 DEBUG --- http response --- 06:12:29 DEBUG --- http response --- 06:12:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:12:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:12:29 DEBUG ----- output ----- 06:12:29 DEBUG ----- output ----- 06:12:29 DEBUG { 06:12:29 DEBUG { 06:12:29 DEBUG "status": "success", 06:12:29 DEBUG "data": { 06:12:29 DEBUG "resultType": "vector", 06:12:29 DEBUG "status": "success", 06:12:29 DEBUG "result": [ 06:12:29 DEBUG "data": { 06:12:29 DEBUG { 06:12:29 DEBUG "resultType": "vector", 06:12:29 DEBUG "metric": { 06:12:29 DEBUG "result": [ 06:12:29 DEBUG "pod": "ds-idrepo-2" 06:12:29 DEBUG { 06:12:29 DEBUG }, 06:12:29 DEBUG "metric": { 06:12:29 DEBUG "value": [ 06:12:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:12:29 DEBUG 06:12:29 DEBUG }, 06:12:29 DEBUG 06:12:29 DEBUG "value": [ 06:12:29 DEBUG 06:12:29 DEBUG 06:12:29 DEBUG [...] 06:12:29 DEBUG 06:12:29 DEBUG 06:12:29 DEBUG 06:12:29 DEBUG 06:12:29 DEBUG [...] 06:12:29 DEBUG 06:12:29 DEBUG 06:12:29 DEBUG "pod": "ds-cts-2" 06:12:29 DEBUG 06:12:29 DEBUG }, 06:12:29 DEBUG 06:12:29 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:12:29 DEBUG }, 06:12:29 DEBUG "value": [ 06:12:29 DEBUG 1712725949, 06:12:29 DEBUG "0.0019600738940784164" 06:12:29 DEBUG ] 06:12:29 DEBUG } 06:12:29 DEBUG ] 06:12:29 DEBUG } 06:12:29 DEBUG } 06:12:29 DEBUG "value": [ 06:12:29 DEBUG 1712725949, 06:12:29 DEBUG "0.000033333333333333335" 06:12:29 DEBUG ] 06:12:29 DEBUG } 06:12:29 DEBUG ] 06:12:29 DEBUG } 06:12:29 DEBUG } 06:12:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725949" 06:12:29 INFO [http_cmd]: http status code OK 06:12:29 DEBUG --- status code --- 06:12:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:29 DEBUG --- http response --- 06:12:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:12:29 DEBUG ----- output ----- 06:12:29 DEBUG { 06:12:29 DEBUG "status": "success", 06:12:29 DEBUG "data": { 06:12:29 DEBUG "resultType": "vector", 06:12:29 DEBUG "result": [ 06:12:29 DEBUG { 06:12:29 DEBUG "metric": { 06:12:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:12:29 DEBUG }, 06:12:29 DEBUG "value": [ 06:12:29 DEBUG 06:12:29 DEBUG 06:12:29 DEBUG 06:12:29 DEBUG [...] 06:12:29 DEBUG 06:12:29 DEBUG 06:12:29 DEBUG 06:12:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:12:29 DEBUG }, 06:12:29 DEBUG "value": [ 06:12:29 DEBUG 1712725949, 06:12:29 DEBUG "1444.7333333333333" 06:12:29 DEBUG ] 06:12:29 DEBUG } 06:12:29 DEBUG ] 06:12:29 DEBUG } 06:12:29 DEBUG } 06:12:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712725949" 06:12:29 INFO [http_cmd]: http status code OK 06:12:29 DEBUG --- status code --- 06:12:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:29 DEBUG --- http response --- 06:12:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:12:29 DEBUG ----- output ----- 06:12:29 DEBUG { 06:12:29 DEBUG "status": "success", 06:12:29 DEBUG "data": { 06:12:29 DEBUG "resultType": "vector", 06:12:29 DEBUG "result": [ 06:12:29 DEBUG { 06:12:29 DEBUG "metric": { 06:12:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:12:29 DEBUG }, 06:12:29 DEBUG "value": [ 06:12:29 DEBUG 06:12:29 DEBUG 06:12:29 DEBUG 06:12:29 DEBUG [...] 06:12:29 DEBUG 06:12:29 DEBUG 06:12:29 DEBUG 06:12:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:12:29 DEBUG }, 06:12:29 DEBUG "value": [ 06:12:29 DEBUG 1712725949, 06:12:29 DEBUG "0.05" 06:12:29 DEBUG ] 06:12:29 DEBUG } 06:12:29 DEBUG ] 06:12:29 DEBUG } 06:12:29 DEBUG } 06:12:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725950" 06:12:30 INFO [http_cmd]: http status code OK 06:12:30 DEBUG --- status code --- 06:12:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:30 DEBUG --- http response --- 06:12:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:12:30 DEBUG ----- output ----- 06:12:30 DEBUG { 06:12:30 DEBUG "status": "success", 06:12:30 DEBUG "data": { 06:12:30 DEBUG "resultType": "vector", 06:12:30 DEBUG "result": [ 06:12:30 DEBUG { 06:12:30 DEBUG "metric": { 06:12:30 DEBUG "pod": "ds-idrepo-0" 06:12:30 DEBUG }, 06:12:30 DEBUG "value": [ 06:12:30 DEBUG 06:12:30 DEBUG 06:12:30 DEBUG 06:12:30 DEBUG [...] 06:12:30 DEBUG 06:12:30 DEBUG 06:12:30 DEBUG 06:12:30 DEBUG "pod": "ds-idrepo-1" 06:12:30 DEBUG }, 06:12:30 DEBUG "value": [ 06:12:30 DEBUG 1712725950, 06:12:30 DEBUG "0" 06:12:30 DEBUG ] 06:12:30 DEBUG } 06:12:30 DEBUG ] 06:12:30 DEBUG } 06:12:30 DEBUG } 06:12:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725950" 06:12:30 INFO [http_cmd]: http status code OK 06:12:30 DEBUG --- status code --- 06:12:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:30 DEBUG --- http response --- 06:12:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:12:30 DEBUG ----- output ----- 06:12:30 DEBUG { 06:12:30 DEBUG "status": "success", 06:12:30 DEBUG "data": { 06:12:30 DEBUG "resultType": "vector", 06:12:30 DEBUG "result": [ 06:12:30 DEBUG { 06:12:30 DEBUG "metric": { 06:12:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:12:30 DEBUG }, 06:12:30 DEBUG "value": [ 06:12:30 DEBUG 06:12:30 DEBUG 06:12:30 DEBUG 06:12:30 DEBUG [...] 06:12:30 DEBUG 06:12:30 DEBUG 06:12:30 DEBUG 06:12:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:12:30 DEBUG }, 06:12:30 DEBUG "value": [ 06:12:30 DEBUG 1712725950, 06:12:30 DEBUG "1432.7333333333333" 06:12:30 DEBUG ] 06:12:30 DEBUG } 06:12:30 DEBUG ] 06:12:30 DEBUG } 06:12:30 DEBUG } 06:12:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725950" 06:12:30 INFO [http_cmd]: http status code OK 06:12:30 DEBUG --- status code --- 06:12:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:30 DEBUG --- http response --- 06:12:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:12:30 DEBUG ----- output ----- 06:12:30 DEBUG { 06:12:30 DEBUG "status": "success", 06:12:30 DEBUG "data": { 06:12:30 DEBUG "resultType": "vector", 06:12:30 DEBUG "result": [ 06:12:30 DEBUG { 06:12:30 DEBUG "metric": { 06:12:30 DEBUG "pod": "ds-idrepo-0" 06:12:30 DEBUG }, 06:12:30 DEBUG "value": [ 06:12:30 DEBUG 06:12:30 DEBUG 06:12:30 DEBUG 06:12:30 DEBUG [...] 06:12:30 DEBUG 06:12:30 DEBUG 06:12:30 DEBUG 06:12:30 DEBUG "pod": "ds-idrepo-1" 06:12:30 DEBUG }, 06:12:30 DEBUG "value": [ 06:12:30 DEBUG 1712725950, 06:12:30 DEBUG "0" 06:12:30 DEBUG ] 06:12:30 DEBUG } 06:12:30 DEBUG ] 06:12:30 DEBUG } 06:12:30 DEBUG } 06:12:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712725950" 06:12:30 INFO [http_cmd]: http status code OK 06:12:30 DEBUG --- status code --- 06:12:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:30 DEBUG --- http response --- 06:12:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:12:30 DEBUG ----- output ----- 06:12:30 DEBUG { 06:12:30 DEBUG "status": "success", 06:12:30 DEBUG "data": { 06:12:30 DEBUG "resultType": "vector", 06:12:30 DEBUG "result": [ 06:12:30 DEBUG { 06:12:30 DEBUG "metric": { 06:12:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:12:30 DEBUG }, 06:12:30 DEBUG "value": [ 06:12:30 DEBUG 06:12:30 DEBUG 06:12:30 DEBUG 06:12:30 DEBUG [...] 06:12:30 DEBUG 06:12:30 DEBUG 06:12:30 DEBUG 06:12:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:12:30 DEBUG }, 06:12:30 DEBUG "value": [ 06:12:30 DEBUG 1712725950, 06:12:30 DEBUG "0" 06:12:30 DEBUG ] 06:12:30 DEBUG } 06:12:30 DEBUG ] 06:12:30 DEBUG } 06:12:30 DEBUG } 06:12:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712725950" 06:12:30 INFO [http_cmd]: http status code OK 06:12:30 DEBUG --- status code --- 06:12:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:30 DEBUG --- http response --- 06:12:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:12:30 DEBUG ----- output ----- 06:12:30 DEBUG { 06:12:30 DEBUG "status": "success", 06:12:30 DEBUG "data": { 06:12:30 DEBUG "resultType": "vector", 06:12:30 DEBUG "result": [ 06:12:30 DEBUG { 06:12:30 DEBUG "metric": { 06:12:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:12:30 DEBUG }, 06:12:30 DEBUG "value": [ 06:12:30 DEBUG 06:12:30 DEBUG 06:12:30 DEBUG 06:12:30 DEBUG [...] 06:12:30 DEBUG 06:12:30 DEBUG 06:12:30 DEBUG 06:12:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:12:30 DEBUG }, 06:12:30 DEBUG "value": [ 06:12:30 DEBUG 1712725950, 06:12:30 DEBUG "2489720832" 06:12:30 DEBUG ] 06:12:30 DEBUG } 06:12:30 DEBUG ] 06:12:30 DEBUG } 06:12:30 DEBUG } 06:12:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725950" 06:12:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712725950" 06:12:30 INFO [http_cmd]: http status code OK 06:12:30 DEBUG --- status code --- 06:12:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:30 DEBUG --- http response --- 06:12:30 INFO [http_cmd]: http status code OK 06:12:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:12:30 DEBUG ----- output ----- 06:12:30 DEBUG { 06:12:30 DEBUG "status": "success", 06:12:30 DEBUG "data": { 06:12:30 DEBUG "resultType": "vector", 06:12:30 DEBUG "result": [ 06:12:30 DEBUG --- status code --- 06:12:30 DEBUG { 06:12:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:30 DEBUG "metric": { 06:12:30 DEBUG --- http response --- 06:12:30 DEBUG "instance": "10.142.0.25:9100" 06:12:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:12:30 DEBUG }, 06:12:30 DEBUG ----- output ----- 06:12:30 DEBUG "value": [ 06:12:30 DEBUG { 06:12:30 DEBUG 06:12:30 DEBUG "status": "success", 06:12:30 DEBUG 06:12:30 DEBUG "data": { 06:12:30 DEBUG 06:12:30 DEBUG "resultType": "vector", 06:12:30 DEBUG [...] 06:12:30 DEBUG 06:12:30 DEBUG "result": [ 06:12:30 DEBUG 06:12:30 DEBUG { 06:12:30 DEBUG 06:12:30 DEBUG "metric": { 06:12:30 DEBUG "instance": "10.142.0.77:9100" 06:12:30 DEBUG }, 06:12:30 DEBUG "instance": "10.142.0.25:9100" 06:12:30 DEBUG "value": [ 06:12:30 DEBUG }, 06:12:30 DEBUG 1712725950, 06:12:30 DEBUG "value": [ 06:12:30 DEBUG "0.04933333333333246" 06:12:30 DEBUG 06:12:30 DEBUG ] 06:12:30 DEBUG 06:12:30 DEBUG } 06:12:30 DEBUG 06:12:30 DEBUG ] 06:12:30 DEBUG [...] 06:12:30 DEBUG } 06:12:30 DEBUG 06:12:30 DEBUG } 06:12:30 DEBUG 06:12:30 DEBUG 06:12:30 DEBUG "instance": "10.142.0.77:9100" 06:12:30 DEBUG }, 06:12:30 DEBUG "value": [ 06:12:30 DEBUG 1712725950, 06:12:30 DEBUG "42301356.56666666" 06:12:30 DEBUG ] 06:12:30 DEBUG } 06:12:30 DEBUG ] 06:12:30 DEBUG } 06:12:30 DEBUG } 06:12:48 INFO 06:12:48 INFO [loop_until]: kubectl --namespace=xlou top pods 06:12:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:12:48 INFO [loop_until]: OK (rc = 0) 06:12:48 DEBUG --- stdout --- 06:12:48 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10747m 8999Mi am-654c4fcbcf-9scrj 10034m 8445Mi am-654c4fcbcf-v5qb5 8552m 8445Mi ds-cts-0 8187m 13320Mi ds-cts-1 8313m 13390Mi ds-cts-2 8227m 12610Mi ds-idrepo-0 10753m 5622Mi ds-idrepo-1 386m 5673Mi ds-idrepo-2 473m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 10m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4209m 2250Mi 06:12:48 DEBUG --- stderr --- 06:12:48 DEBUG 06:12:56 INFO 06:12:56 INFO [loop_until]: kubectl --namespace=xlou top node 06:12:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:12:56 INFO [loop_until]: OK (rc = 0) 06:12:56 DEBUG --- stdout --- 06:12:56 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 95m 0% 2307Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9292m 58% 9317Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 81m 0% 3260Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13377m 84% 6603Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10939m 68% 9339Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11865m 74% 9808Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 78m 0% 854Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10897m 68% 6298Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8386m 52% 13987Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 8160m 51% 13393Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 472m 2% 6387Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8666m 54% 13180Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 412m 2% 6349Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4281m 26% 5877Mi 10% 06:12:56 DEBUG --- stderr --- 06:12:56 DEBUG 06:13:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712726007" 06:13:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726007" 06:13:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726007" 06:13:27 INFO [http_cmd]: http status code OK 06:13:27 DEBUG --- status code --- 06:13:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:27 DEBUG --- http response --- 06:13:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:13:27 DEBUG ----- output ----- 06:13:27 DEBUG { 06:13:27 DEBUG "status": "success", 06:13:27 DEBUG "data": { 06:13:27 DEBUG "resultType": "vector", 06:13:27 DEBUG "result": [ 06:13:27 DEBUG { 06:13:27 DEBUG "metric": { 06:13:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:13:27 DEBUG }, 06:13:27 DEBUG "value": [ 06:13:27 DEBUG 06:13:27 DEBUG 06:13:27 DEBUG 06:13:27 DEBUG [...] 06:13:27 DEBUG 06:13:27 DEBUG 06:13:27 DEBUG 06:13:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:13:27 DEBUG }, 06:13:27 DEBUG "value": [ 06:13:27 DEBUG 1712726007, 06:13:27 DEBUG "0" 06:13:27 DEBUG ] 06:13:27 DEBUG } 06:13:27 DEBUG ] 06:13:27 DEBUG } 06:13:27 DEBUG } 06:13:27 ERROR [http_cmd]: ERROR 06:13:27 ERROR --- status code --- 06:13:27 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:13:27 INFO [http_cmd]: http status code OK 06:13:27 ERROR --- http response --- 06:13:27 DEBUG --- status code --- 06:13:27 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:13:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:27 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:13:27 DEBUG --- http response --- 06:13:27 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:13:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726008" 06:13:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726008" 06:13:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726008" 06:13:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726008" 06:13:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726008" 06:13:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726008" 06:13:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726008" 06:13:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726008" 06:13:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726008" 06:13:28 INFO [http_cmd]: http status code OK 06:13:28 INFO [http_cmd]: http status code OK 06:13:28 DEBUG --- status code --- 06:13:28 DEBUG --- status code --- 06:13:28 INFO [http_cmd]: http status code OK 06:13:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:28 DEBUG --- status code --- 06:13:28 INFO [http_cmd]: http status code OK 06:13:28 DEBUG --- http response --- 06:13:28 INFO [http_cmd]: http status code OK 06:13:28 DEBUG --- status code --- 06:13:28 INFO [http_cmd]: http status code OK 06:13:28 DEBUG --- status code --- 06:13:28 DEBUG --- status code --- 06:13:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:28 DEBUG --- http response --- 06:13:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:13:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:28 INFO [http_cmd]: http status code OK 06:13:28 INFO [http_cmd]: http status code OK 06:13:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:28 INFO [http_cmd]: http status code OK 06:13:28 DEBUG --- http response --- 06:13:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:13:28 DEBUG ----- output ----- 06:13:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:28 DEBUG --- http response --- 06:13:28 DEBUG --- status code --- 06:13:28 DEBUG --- status code --- 06:13:28 DEBUG --- http response --- 06:13:28 DEBUG --- status code --- 06:13:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:13:28 DEBUG ----- output ----- 06:13:28 DEBUG { 06:13:28 DEBUG --- http response --- 06:13:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:13:28 DEBUG ----- output ----- 06:13:28 DEBUG { 06:13:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:28 DEBUG ----- output ----- 06:13:28 DEBUG { 06:13:28 DEBUG "status": "success", 06:13:28 DEBUG "data": { 06:13:28 DEBUG "resultType": "vector", 06:13:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:28 DEBUG --- http response --- 06:13:28 DEBUG "status": "success", 06:13:28 DEBUG --- http response --- 06:13:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:13:28 DEBUG "status": "success", 06:13:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:13:28 DEBUG ----- output ----- 06:13:28 DEBUG --- http response --- 06:13:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:13:28 DEBUG "data": { 06:13:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 06:13:28 DEBUG ----- output ----- 06:13:28 DEBUG { 06:13:28 DEBUG "data": { 06:13:28 DEBUG "resultType": "vector", 06:13:28 DEBUG { 06:13:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:13:28 DEBUG ----- output ----- 06:13:28 DEBUG "resultType": "vector", 06:13:28 DEBUG ----- output ----- 06:13:28 DEBUG "status": "success", 06:13:28 DEBUG "data": { 06:13:28 DEBUG "resultType": "vector", 06:13:28 DEBUG "result": [ 06:13:28 DEBUG "status": "success", 06:13:28 DEBUG "data": { 06:13:28 DEBUG "resultType": "vector", 06:13:28 DEBUG "result": [ 06:13:28 DEBUG { 06:13:28 DEBUG "result": [ 06:13:28 DEBUG { 06:13:28 DEBUG "metric": { 06:13:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:13:28 DEBUG }, 06:13:28 DEBUG "value": [ 06:13:28 DEBUG 06:13:28 DEBUG { 06:13:28 DEBUG "status": "success", 06:13:28 DEBUG { 06:13:28 DEBUG { 06:13:28 DEBUG "status": "success", 06:13:28 DEBUG "data": { 06:13:28 DEBUG "result": [ 06:13:28 DEBUG { 06:13:28 DEBUG "status": "success", 06:13:28 DEBUG "metric": { 06:13:28 DEBUG "data": { 06:13:28 DEBUG "metric": { 06:13:28 DEBUG ----- output ----- 06:13:28 DEBUG "resultType": "vector", 06:13:28 DEBUG "result": [ 06:13:28 DEBUG { 06:13:28 DEBUG "metric": { 06:13:28 DEBUG "result": [ 06:13:28 DEBUG "pod": "ds-idrepo-0" 06:13:28 DEBUG "resultType": "vector", 06:13:28 DEBUG "pod": "ds-idrepo-0" 06:13:28 DEBUG { 06:13:28 DEBUG "status": "success", 06:13:28 DEBUG "pod": "ds-idrepo-0" 06:13:28 DEBUG }, 06:13:28 DEBUG "value": [ 06:13:28 DEBUG { 06:13:28 DEBUG }, 06:13:28 DEBUG "result": [ 06:13:28 DEBUG }, 06:13:28 DEBUG "data": { 06:13:28 DEBUG "resultType": "vector", 06:13:28 DEBUG { 06:13:28 DEBUG "metric": { 06:13:28 DEBUG "instance": "10.142.0.25:9100" 06:13:28 DEBUG "metric": { 06:13:28 DEBUG "value": [ 06:13:28 DEBUG { 06:13:28 DEBUG "value": [ 06:13:28 DEBUG "result": [ 06:13:28 DEBUG { 06:13:28 DEBUG "metric": { 06:13:28 DEBUG 06:13:28 DEBUG 06:13:28 DEBUG [...] 06:13:28 DEBUG 06:13:28 DEBUG "metric": { 06:13:28 DEBUG 06:13:28 DEBUG }, 06:13:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:13:28 DEBUG }, 06:13:28 DEBUG "value": [ 06:13:28 DEBUG 06:13:28 DEBUG 06:13:28 DEBUG 06:13:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:13:28 DEBUG 06:13:28 DEBUG "value": [ 06:13:28 DEBUG 06:13:28 DEBUG 06:13:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:13:28 DEBUG 06:13:28 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:13:28 DEBUG 06:13:28 DEBUG }, 06:13:28 DEBUG 06:13:28 DEBUG 06:13:28 DEBUG 06:13:28 DEBUG 06:13:28 DEBUG 06:13:28 DEBUG }, 06:13:28 DEBUG 06:13:28 DEBUG [...] 06:13:28 DEBUG "value": [ 06:13:28 DEBUG [...] 06:13:28 DEBUG [...] 06:13:28 DEBUG "data": { 06:13:28 DEBUG }, 06:13:28 DEBUG [...] 06:13:28 DEBUG "value": [ 06:13:28 DEBUG 06:13:28 DEBUG 06:13:28 DEBUG 06:13:28 DEBUG 06:13:28 DEBUG 06:13:28 DEBUG "resultType": "vector", 06:13:28 DEBUG "value": [ 06:13:28 DEBUG 06:13:28 DEBUG 1712726008, 06:13:28 DEBUG 06:13:28 DEBUG 06:13:28 DEBUG 06:13:28 DEBUG 06:13:28 DEBUG 06:13:28 DEBUG "result": [ 06:13:28 DEBUG { 06:13:28 DEBUG 06:13:28 DEBUG "0" 06:13:28 DEBUG [...] 06:13:28 DEBUG 06:13:28 DEBUG [...] 06:13:28 DEBUG 06:13:28 DEBUG 06:13:28 DEBUG 06:13:28 DEBUG "metric": { 06:13:28 DEBUG "instance": "10.142.0.25:9100" 06:13:28 DEBUG ] 06:13:28 DEBUG 06:13:28 DEBUG "pod": "ds-idrepo-1" 06:13:28 DEBUG 06:13:28 DEBUG 06:13:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:13:28 DEBUG 06:13:28 DEBUG }, 06:13:28 DEBUG 06:13:28 DEBUG "instance": "10.142.0.77:9100" 06:13:28 DEBUG 06:13:28 DEBUG }, 06:13:28 DEBUG 06:13:28 DEBUG "pod": "ds-cts-2" 06:13:28 DEBUG }, 06:13:28 DEBUG 06:13:28 DEBUG "value": [ 06:13:28 DEBUG } 06:13:28 DEBUG ] 06:13:28 DEBUG 06:13:28 DEBUG "value": [ 06:13:28 DEBUG 06:13:28 DEBUG }, 06:13:28 DEBUG "value": [ 06:13:28 DEBUG [...] 06:13:28 DEBUG 06:13:28 DEBUG 06:13:28 DEBUG } 06:13:28 DEBUG "pod": "ds-cts-2" 06:13:28 DEBUG 1712726008, 06:13:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:13:28 DEBUG "value": [ 06:13:28 DEBUG 1712726008, 06:13:28 DEBUG 06:13:28 DEBUG 06:13:28 DEBUG }, 06:13:28 DEBUG "value": [ 06:13:28 DEBUG }, 06:13:28 DEBUG "0" 06:13:28 DEBUG }, 06:13:28 DEBUG 1712726008, 06:13:28 DEBUG "1441.5" 06:13:28 DEBUG 06:13:28 DEBUG [...] 06:13:28 DEBUG 1712726008, 06:13:28 DEBUG "367291706368" 06:13:28 DEBUG ] 06:13:28 DEBUG "value": [ 06:13:28 DEBUG 1712726008, 06:13:28 DEBUG "29.4" 06:13:28 DEBUG ] 06:13:28 DEBUG 06:13:28 DEBUG 06:13:28 DEBUG } 06:13:28 DEBUG ] 06:13:28 DEBUG } 06:13:28 DEBUG "0" 06:13:28 DEBUG ] 06:13:28 DEBUG ] 06:13:28 DEBUG } 06:13:28 DEBUG ] 06:13:28 DEBUG } 06:13:28 DEBUG } 06:13:28 DEBUG } 06:13:28 DEBUG } 06:13:28 DEBUG } 06:13:28 DEBUG "value": [ 06:13:28 DEBUG 1712726008, 06:13:28 DEBUG 06:13:28 DEBUG 06:13:28 DEBUG "instance": "10.142.0.77:9100" 06:13:28 DEBUG }, 06:13:28 DEBUG "value": [ 06:13:28 DEBUG 1712726008, 06:13:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:13:28 DEBUG ] 06:13:28 DEBUG } 06:13:28 DEBUG ] 06:13:28 DEBUG "1441.5666666666666" 06:13:28 DEBUG "42525130.9" 06:13:28 DEBUG }, 06:13:28 DEBUG } 06:13:28 DEBUG } 06:13:28 DEBUG ] 06:13:28 DEBUG ] 06:13:28 DEBUG ] 06:13:28 DEBUG "value": [ 06:13:28 DEBUG } 06:13:28 DEBUG } 06:13:28 DEBUG } 06:13:28 DEBUG } 06:13:28 DEBUG ] 06:13:28 DEBUG } 06:13:28 DEBUG } 06:13:28 DEBUG } 06:13:28 DEBUG ] 06:13:28 DEBUG ] 06:13:28 DEBUG 1712726008, 06:13:28 DEBUG } 06:13:28 DEBUG } 06:13:28 DEBUG "258057" 06:13:28 DEBUG } 06:13:28 DEBUG } 06:13:28 DEBUG ] 06:13:28 DEBUG } 06:13:28 DEBUG ] 06:13:28 DEBUG } 06:13:28 DEBUG } 06:13:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726008" 06:13:28 INFO [http_cmd]: http status code OK 06:13:28 DEBUG --- status code --- 06:13:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:28 DEBUG --- http response --- 06:13:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:13:28 DEBUG ----- output ----- 06:13:28 DEBUG { 06:13:28 DEBUG "status": "success", 06:13:28 DEBUG "data": { 06:13:28 DEBUG "resultType": "vector", 06:13:28 DEBUG "result": [ 06:13:28 DEBUG { 06:13:28 DEBUG "metric": { 06:13:28 DEBUG "instance": "10.142.0.25:9100" 06:13:28 DEBUG }, 06:13:28 DEBUG "value": [ 06:13:28 DEBUG 06:13:28 DEBUG 06:13:28 DEBUG 06:13:28 DEBUG [...] 06:13:28 DEBUG 06:13:28 DEBUG 06:13:28 DEBUG 06:13:28 DEBUG "instance": "10.142.0.77:9100" 06:13:28 DEBUG }, 06:13:28 DEBUG "value": [ 06:13:28 DEBUG 1712726008, 06:13:28 DEBUG "0.054433333333336144" 06:13:28 DEBUG ] 06:13:28 DEBUG } 06:13:28 DEBUG ] 06:13:28 DEBUG } 06:13:28 DEBUG } 06:13:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726009" 06:13:29 INFO [http_cmd]: http status code OK 06:13:29 DEBUG --- status code --- 06:13:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:29 DEBUG --- http response --- 06:13:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:13:29 DEBUG ----- output ----- 06:13:29 DEBUG { 06:13:29 DEBUG "status": "success", 06:13:29 DEBUG "data": { 06:13:29 DEBUG "resultType": "vector", 06:13:29 DEBUG "result": [ 06:13:29 DEBUG { 06:13:29 DEBUG "metric": { 06:13:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:13:29 DEBUG }, 06:13:29 DEBUG "value": [ 06:13:29 DEBUG 06:13:29 DEBUG 06:13:29 DEBUG 06:13:29 DEBUG [...] 06:13:29 DEBUG 06:13:29 DEBUG 06:13:29 DEBUG 06:13:29 DEBUG "pod": "ds-cts-0" 06:13:29 DEBUG }, 06:13:29 DEBUG "value": [ 06:13:29 DEBUG 1712726009, 06:13:29 DEBUG "73289.20139484365" 06:13:29 DEBUG ] 06:13:29 DEBUG } 06:13:29 DEBUG ] 06:13:29 DEBUG } 06:13:29 DEBUG } 06:13:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712726009" 06:13:29 INFO [http_cmd]: http status code OK 06:13:29 DEBUG --- status code --- 06:13:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:29 DEBUG --- http response --- 06:13:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:13:29 DEBUG ----- output ----- 06:13:29 DEBUG { 06:13:29 DEBUG "status": "success", 06:13:29 DEBUG "data": { 06:13:29 DEBUG "resultType": "vector", 06:13:29 DEBUG "result": [ 06:13:29 DEBUG { 06:13:29 DEBUG "metric": { 06:13:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:13:29 DEBUG }, 06:13:29 DEBUG "value": [ 06:13:29 DEBUG 06:13:29 DEBUG 06:13:29 DEBUG 06:13:29 DEBUG [...] 06:13:29 DEBUG 06:13:29 DEBUG 06:13:29 DEBUG 06:13:29 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:13:29 DEBUG }, 06:13:29 DEBUG "value": [ 06:13:29 DEBUG 1712726009, 06:13:29 DEBUG "4.1842269805385035" 06:13:29 DEBUG ] 06:13:29 DEBUG } 06:13:29 DEBUG ] 06:13:29 DEBUG } 06:13:29 DEBUG } 06:13:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712726009" 06:13:29 INFO [http_cmd]: http status code OK 06:13:29 DEBUG --- status code --- 06:13:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:29 DEBUG --- http response --- 06:13:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:13:29 DEBUG ----- output ----- 06:13:29 DEBUG { 06:13:29 DEBUG "status": "success", 06:13:29 DEBUG "data": { 06:13:29 DEBUG "resultType": "vector", 06:13:29 DEBUG "result": [ 06:13:29 DEBUG { 06:13:29 DEBUG "metric": { 06:13:29 DEBUG "pod": "ds-idrepo-0" 06:13:29 DEBUG }, 06:13:29 DEBUG "value": [ 06:13:29 DEBUG 06:13:29 DEBUG 06:13:29 DEBUG 06:13:29 DEBUG [...] 06:13:29 DEBUG 06:13:29 DEBUG 06:13:29 DEBUG 06:13:29 DEBUG "pod": "ds-cts-2" 06:13:29 DEBUG }, 06:13:29 DEBUG "value": [ 06:13:29 DEBUG 1712726009, 06:13:29 DEBUG "0" 06:13:29 DEBUG ] 06:13:29 DEBUG } 06:13:29 DEBUG ] 06:13:29 DEBUG } 06:13:29 DEBUG } 06:13:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726009" 06:13:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726009" 06:13:29 INFO [http_cmd]: http status code OK 06:13:29 INFO [http_cmd]: http status code OK 06:13:29 DEBUG --- status code --- 06:13:29 DEBUG --- status code --- 06:13:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:29 DEBUG --- http response --- 06:13:29 DEBUG --- http response --- 06:13:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:13:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:13:29 DEBUG ----- output ----- 06:13:29 DEBUG { 06:13:29 DEBUG ----- output ----- 06:13:29 DEBUG "status": "success", 06:13:29 DEBUG { 06:13:29 DEBUG "data": { 06:13:29 DEBUG "resultType": "vector", 06:13:29 DEBUG "status": "success", 06:13:29 DEBUG "result": [ 06:13:29 DEBUG "data": { 06:13:29 DEBUG { 06:13:29 DEBUG "metric": { 06:13:29 DEBUG "resultType": "vector", 06:13:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:13:29 DEBUG }, 06:13:29 DEBUG "value": [ 06:13:29 DEBUG 06:13:29 DEBUG 06:13:29 DEBUG 06:13:29 DEBUG [...] 06:13:29 DEBUG 06:13:29 DEBUG 06:13:29 DEBUG 06:13:29 DEBUG "pod": "ds-cts-0" 06:13:29 DEBUG "result": [ 06:13:29 DEBUG }, 06:13:29 DEBUG { 06:13:29 DEBUG "value": [ 06:13:29 DEBUG "metric": { 06:13:29 DEBUG 1712726009, 06:13:29 DEBUG "8.311028132367115" 06:13:29 DEBUG ] 06:13:29 DEBUG } 06:13:29 DEBUG ] 06:13:29 DEBUG "pod": "ds-idrepo-2" 06:13:29 DEBUG } 06:13:29 DEBUG }, 06:13:29 DEBUG } 06:13:29 DEBUG "value": [ 06:13:29 DEBUG 06:13:29 DEBUG 06:13:29 DEBUG 06:13:29 DEBUG [...] 06:13:29 DEBUG 06:13:29 DEBUG 06:13:29 DEBUG 06:13:29 DEBUG "pod": "ds-cts-2" 06:13:29 DEBUG }, 06:13:29 DEBUG "value": [ 06:13:29 DEBUG 1712726009, 06:13:29 DEBUG "0.00003333333333333331" 06:13:29 DEBUG ] 06:13:29 DEBUG } 06:13:29 DEBUG ] 06:13:29 DEBUG } 06:13:29 DEBUG } 06:13:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726010" 06:13:30 INFO [http_cmd]: http status code OK 06:13:30 DEBUG --- status code --- 06:13:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:30 DEBUG --- http response --- 06:13:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:13:30 DEBUG ----- output ----- 06:13:30 DEBUG { 06:13:30 DEBUG "status": "success", 06:13:30 DEBUG "data": { 06:13:30 DEBUG "resultType": "vector", 06:13:30 DEBUG "result": [ 06:13:30 DEBUG { 06:13:30 DEBUG "metric": { 06:13:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:13:30 DEBUG }, 06:13:30 DEBUG "value": [ 06:13:30 DEBUG 06:13:30 DEBUG 06:13:30 DEBUG 06:13:30 DEBUG [...] 06:13:30 DEBUG 06:13:30 DEBUG 06:13:30 DEBUG 06:13:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:13:30 DEBUG }, 06:13:30 DEBUG "value": [ 06:13:30 DEBUG 1712726010, 06:13:30 DEBUG "1441.2666666666667" 06:13:30 DEBUG ] 06:13:30 DEBUG } 06:13:30 DEBUG ] 06:13:30 DEBUG } 06:13:30 DEBUG } 06:13:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712726010" 06:13:30 INFO [http_cmd]: http status code OK 06:13:30 DEBUG --- status code --- 06:13:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:30 DEBUG --- http response --- 06:13:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:13:30 DEBUG ----- output ----- 06:13:30 DEBUG { 06:13:30 DEBUG "status": "success", 06:13:30 DEBUG "data": { 06:13:30 DEBUG "resultType": "vector", 06:13:30 DEBUG "result": [ 06:13:30 DEBUG { 06:13:30 DEBUG "metric": { 06:13:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:13:30 DEBUG }, 06:13:30 DEBUG "value": [ 06:13:30 DEBUG 06:13:30 DEBUG 06:13:30 DEBUG 06:13:30 DEBUG [...] 06:13:30 DEBUG 06:13:30 DEBUG 06:13:30 DEBUG 06:13:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:13:30 DEBUG }, 06:13:30 DEBUG "value": [ 06:13:30 DEBUG 1712726010, 06:13:30 DEBUG "0.05" 06:13:30 DEBUG ] 06:13:30 DEBUG } 06:13:30 DEBUG ] 06:13:30 DEBUG } 06:13:30 DEBUG } 06:13:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726010" 06:13:30 INFO [http_cmd]: http status code OK 06:13:30 DEBUG --- status code --- 06:13:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:30 DEBUG --- http response --- 06:13:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:13:30 DEBUG ----- output ----- 06:13:30 DEBUG { 06:13:30 DEBUG "status": "success", 06:13:30 DEBUG "data": { 06:13:30 DEBUG "resultType": "vector", 06:13:30 DEBUG "result": [ 06:13:30 DEBUG { 06:13:30 DEBUG "metric": { 06:13:30 DEBUG "pod": "ds-idrepo-0" 06:13:30 DEBUG }, 06:13:30 DEBUG "value": [ 06:13:30 DEBUG 06:13:30 DEBUG 06:13:30 DEBUG 06:13:30 DEBUG [...] 06:13:30 DEBUG 06:13:30 DEBUG 06:13:30 DEBUG 06:13:30 DEBUG "pod": "ds-idrepo-1" 06:13:30 DEBUG }, 06:13:30 DEBUG "value": [ 06:13:30 DEBUG 1712726010, 06:13:30 DEBUG "0" 06:13:30 DEBUG ] 06:13:30 DEBUG } 06:13:30 DEBUG ] 06:13:30 DEBUG } 06:13:30 DEBUG } 06:13:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726010" 06:13:30 INFO [http_cmd]: http status code OK 06:13:30 DEBUG --- status code --- 06:13:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:30 DEBUG --- http response --- 06:13:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:13:30 DEBUG ----- output ----- 06:13:30 DEBUG { 06:13:30 DEBUG "status": "success", 06:13:30 DEBUG "data": { 06:13:30 DEBUG "resultType": "vector", 06:13:30 DEBUG "result": [ 06:13:30 DEBUG { 06:13:30 DEBUG "metric": { 06:13:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:13:30 DEBUG }, 06:13:30 DEBUG "value": [ 06:13:30 DEBUG 06:13:30 DEBUG 06:13:30 DEBUG 06:13:30 DEBUG [...] 06:13:30 DEBUG 06:13:30 DEBUG 06:13:30 DEBUG 06:13:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:13:30 DEBUG }, 06:13:30 DEBUG "value": [ 06:13:30 DEBUG 1712726010, 06:13:30 DEBUG "1468.9333333333334" 06:13:30 DEBUG ] 06:13:30 DEBUG } 06:13:30 DEBUG ] 06:13:30 DEBUG } 06:13:30 DEBUG } 06:13:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726010" 06:13:30 INFO [http_cmd]: http status code OK 06:13:30 DEBUG --- status code --- 06:13:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:30 DEBUG --- http response --- 06:13:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:13:30 DEBUG ----- output ----- 06:13:30 DEBUG { 06:13:30 DEBUG "status": "success", 06:13:30 DEBUG "data": { 06:13:30 DEBUG "resultType": "vector", 06:13:30 DEBUG "result": [ 06:13:30 DEBUG { 06:13:30 DEBUG "metric": { 06:13:30 DEBUG "pod": "ds-idrepo-0" 06:13:30 DEBUG }, 06:13:30 DEBUG "value": [ 06:13:30 DEBUG 06:13:30 DEBUG 06:13:30 DEBUG 06:13:30 DEBUG [...] 06:13:30 DEBUG 06:13:30 DEBUG 06:13:30 DEBUG 06:13:30 DEBUG "pod": "ds-idrepo-1" 06:13:30 DEBUG }, 06:13:30 DEBUG "value": [ 06:13:30 DEBUG 1712726010, 06:13:30 DEBUG "0" 06:13:30 DEBUG ] 06:13:30 DEBUG } 06:13:30 DEBUG ] 06:13:30 DEBUG } 06:13:30 DEBUG } 06:13:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726010" 06:13:30 INFO [http_cmd]: http status code OK 06:13:30 DEBUG --- status code --- 06:13:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:30 DEBUG --- http response --- 06:13:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:13:30 DEBUG ----- output ----- 06:13:30 DEBUG { 06:13:30 DEBUG "status": "success", 06:13:30 DEBUG "data": { 06:13:30 DEBUG "resultType": "vector", 06:13:30 DEBUG "result": [ 06:13:30 DEBUG { 06:13:30 DEBUG "metric": { 06:13:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:13:30 DEBUG }, 06:13:30 DEBUG "value": [ 06:13:30 DEBUG 06:13:30 DEBUG 06:13:30 DEBUG 06:13:30 DEBUG [...] 06:13:30 DEBUG 06:13:30 DEBUG 06:13:30 DEBUG 06:13:30 DEBUG "pod": "ds-cts-0" 06:13:30 DEBUG }, 06:13:30 DEBUG "value": [ 06:13:30 DEBUG 1712726010, 06:13:30 DEBUG "14955563.025210084" 06:13:30 DEBUG ] 06:13:30 DEBUG } 06:13:30 DEBUG ] 06:13:30 DEBUG } 06:13:30 DEBUG } 06:13:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712726010" 06:13:30 INFO [http_cmd]: http status code OK 06:13:30 DEBUG --- status code --- 06:13:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:30 DEBUG --- http response --- 06:13:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:13:30 DEBUG ----- output ----- 06:13:30 DEBUG { 06:13:30 DEBUG "status": "success", 06:13:30 DEBUG "data": { 06:13:30 DEBUG "resultType": "vector", 06:13:30 DEBUG "result": [ 06:13:30 DEBUG { 06:13:30 DEBUG "metric": { 06:13:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:13:30 DEBUG }, 06:13:30 DEBUG "value": [ 06:13:30 DEBUG 06:13:30 DEBUG 06:13:30 DEBUG 06:13:30 DEBUG [...] 06:13:30 DEBUG 06:13:30 DEBUG 06:13:30 DEBUG 06:13:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:13:30 DEBUG }, 06:13:30 DEBUG "value": [ 06:13:30 DEBUG 1712726010, 06:13:30 DEBUG "2487988224" 06:13:30 DEBUG ] 06:13:30 DEBUG } 06:13:30 DEBUG ] 06:13:30 DEBUG } 06:13:30 DEBUG } 06:13:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726011" 06:13:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726011" 06:13:31 INFO [http_cmd]: http status code OK 06:13:31 DEBUG --- status code --- 06:13:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:31 DEBUG --- http response --- 06:13:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:13:31 DEBUG ----- output ----- 06:13:31 DEBUG { 06:13:31 DEBUG "status": "success", 06:13:31 DEBUG "data": { 06:13:31 DEBUG "resultType": "vector", 06:13:31 DEBUG "result": [ 06:13:31 DEBUG { 06:13:31 DEBUG "metric": { 06:13:31 DEBUG "instance": "10.142.0.25:9100" 06:13:31 DEBUG }, 06:13:31 DEBUG "value": [ 06:13:31 DEBUG 06:13:31 DEBUG 06:13:31 DEBUG 06:13:31 DEBUG [...] 06:13:31 DEBUG 06:13:31 DEBUG 06:13:31 DEBUG 06:13:31 DEBUG "instance": "10.142.0.77:9100" 06:13:31 DEBUG }, 06:13:31 DEBUG "value": [ 06:13:31 DEBUG 1712726011, 06:13:31 DEBUG "42527037.86666667" 06:13:31 DEBUG ] 06:13:31 DEBUG } 06:13:31 DEBUG ] 06:13:31 DEBUG } 06:13:31 DEBUG } 06:13:31 INFO [http_cmd]: http status code OK 06:13:31 DEBUG --- status code --- 06:13:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:13:31 DEBUG --- http response --- 06:13:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:13:31 DEBUG ----- output ----- 06:13:31 DEBUG { 06:13:31 DEBUG "status": "success", 06:13:31 DEBUG "data": { 06:13:31 DEBUG "resultType": "vector", 06:13:31 DEBUG "result": [ 06:13:31 DEBUG { 06:13:31 DEBUG "metric": { 06:13:31 DEBUG "instance": "10.142.0.25:9100" 06:13:31 DEBUG }, 06:13:31 DEBUG "value": [ 06:13:31 DEBUG 06:13:31 DEBUG 06:13:31 DEBUG 06:13:31 DEBUG [...] 06:13:31 DEBUG 06:13:31 DEBUG 06:13:31 DEBUG 06:13:31 DEBUG "instance": "10.142.0.77:9100" 06:13:31 DEBUG }, 06:13:31 DEBUG "value": [ 06:13:31 DEBUG 1712726011, 06:13:31 DEBUG "0.01699999999999975" 06:13:31 DEBUG ] 06:13:31 DEBUG } 06:13:31 DEBUG ] 06:13:31 DEBUG } 06:13:31 DEBUG } 06:13:48 INFO 06:13:48 INFO [loop_until]: kubectl --namespace=xlou top pods 06:13:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:13:48 INFO [loop_until]: OK (rc = 0) 06:13:48 DEBUG --- stdout --- 06:13:48 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 11093m 8980Mi am-654c4fcbcf-9scrj 10083m 8443Mi am-654c4fcbcf-v5qb5 8521m 8445Mi ds-cts-0 8493m 13385Mi ds-cts-1 8410m 13213Mi ds-cts-2 8043m 13383Mi ds-idrepo-0 10788m 5622Mi ds-idrepo-1 363m 5672Mi ds-idrepo-2 384m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 2m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3962m 2255Mi 06:13:48 DEBUG --- stderr --- 06:13:48 DEBUG 06:13:56 INFO 06:13:56 INFO [loop_until]: kubectl --namespace=xlou top node 06:13:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:13:56 INFO [loop_until]: OK (rc = 0) 06:13:56 DEBUG --- stdout --- 06:13:56 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 96m 0% 2309Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9397m 59% 9316Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 80m 0% 3258Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13614m 85% 6615Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10521m 66% 9328Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11943m 75% 9799Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 81m 0% 858Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10520m 66% 6298Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8568m 53% 14068Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 8233m 51% 14058Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 447m 2% 6387Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8610m 54% 13919Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 415m 2% 6349Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4342m 27% 5871Mi 10% 06:13:56 DEBUG --- stderr --- 06:13:56 DEBUG 06:14:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726067" 06:14:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712726067" 06:14:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726067" 06:14:27 ERROR [http_cmd]: ERROR 06:14:27 INFO [http_cmd]: http status code OK 06:14:27 ERROR --- status code --- 06:14:27 DEBUG --- status code --- 06:14:27 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:14:27 INFO [http_cmd]: http status code OK 06:14:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:27 ERROR --- http response --- 06:14:27 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:14:27 DEBUG --- http response --- 06:14:27 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:14:27 DEBUG --- status code --- 06:14:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:14:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:27 DEBUG ----- output ----- 06:14:27 DEBUG --- http response --- 06:14:27 DEBUG { 06:14:27 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:14:27 DEBUG "status": "success", 06:14:27 DEBUG "data": { 06:14:27 DEBUG "resultType": "vector", 06:14:27 DEBUG "result": [ 06:14:27 DEBUG { 06:14:27 DEBUG "metric": { 06:14:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:14:27 DEBUG }, 06:14:27 DEBUG "value": [ 06:14:27 DEBUG 06:14:27 DEBUG 06:14:27 DEBUG 06:14:27 DEBUG [...] 06:14:27 DEBUG 06:14:27 DEBUG 06:14:27 DEBUG 06:14:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:14:27 DEBUG }, 06:14:27 DEBUG "value": [ 06:14:27 DEBUG 1712726067, 06:14:27 DEBUG "0" 06:14:27 DEBUG ] 06:14:27 DEBUG } 06:14:27 DEBUG ] 06:14:27 DEBUG } 06:14:27 DEBUG } 06:14:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726068" 06:14:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726068" 06:14:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726068" 06:14:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726068" 06:14:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726068" 06:14:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726068" 06:14:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726068" 06:14:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726068" 06:14:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726068" 06:14:28 INFO [http_cmd]: http status code OK 06:14:28 INFO [http_cmd]: http status code OK 06:14:28 INFO [http_cmd]: http status code OK 06:14:28 DEBUG --- status code --- 06:14:28 DEBUG --- status code --- 06:14:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:28 DEBUG --- http response --- 06:14:28 INFO [http_cmd]: http status code OK 06:14:28 DEBUG --- status code --- 06:14:28 INFO [http_cmd]: http status code OK 06:14:28 DEBUG --- status code --- 06:14:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:14:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:28 DEBUG --- http response --- 06:14:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:14:28 DEBUG ----- output ----- 06:14:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:28 DEBUG { 06:14:28 DEBUG "status": "success", 06:14:28 DEBUG "data": { 06:14:28 DEBUG ----- output ----- 06:14:28 DEBUG --- http response --- 06:14:28 DEBUG --- status code --- 06:14:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:28 INFO [http_cmd]: http status code OK 06:14:28 DEBUG --- status code --- 06:14:28 INFO [http_cmd]: http status code OK 06:14:28 DEBUG --- status code --- 06:14:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:28 DEBUG --- http response --- 06:14:28 DEBUG --- http response --- 06:14:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:28 DEBUG --- http response --- 06:14:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:28 INFO [http_cmd]: http status code OK 06:14:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 06:14:28 DEBUG { 06:14:28 DEBUG "status": "success", 06:14:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:14:28 DEBUG ----- output ----- 06:14:28 DEBUG { 06:14:28 DEBUG --- http response --- 06:14:28 DEBUG --- status code --- 06:14:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:28 DEBUG "data": { 06:14:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:14:28 DEBUG ----- output ----- 06:14:28 DEBUG { 06:14:28 DEBUG "status": "success", 06:14:28 DEBUG "data": { 06:14:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:14:28 DEBUG --- http response --- 06:14:28 DEBUG ----- output ----- 06:14:28 DEBUG "resultType": "vector", 06:14:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:14:28 INFO [http_cmd]: http status code OK 06:14:28 DEBUG --- status code --- 06:14:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:28 DEBUG --- http response --- 06:14:28 DEBUG ----- output ----- 06:14:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:14:28 DEBUG { 06:14:28 DEBUG "result": [ 06:14:28 DEBUG ----- output ----- 06:14:28 DEBUG "resultType": "vector", 06:14:28 DEBUG "status": "success", 06:14:28 DEBUG "resultType": "vector", 06:14:28 DEBUG "result": [ 06:14:28 DEBUG { 06:14:28 DEBUG ----- output ----- 06:14:28 DEBUG { 06:14:28 DEBUG "status": "success", 06:14:28 DEBUG { 06:14:28 DEBUG "status": "success", 06:14:28 DEBUG "data": { 06:14:28 DEBUG "resultType": "vector", 06:14:28 DEBUG "metric": { 06:14:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:14:28 DEBUG "data": { 06:14:28 DEBUG "status": "success", 06:14:28 DEBUG { 06:14:28 DEBUG "metric": { 06:14:28 DEBUG "pod": "ds-idrepo-0" 06:14:28 DEBUG "result": [ 06:14:28 DEBUG { 06:14:28 DEBUG "instance": "10.142.0.25:9100" 06:14:28 DEBUG ----- output ----- 06:14:28 DEBUG "resultType": "vector", 06:14:28 DEBUG "data": { 06:14:28 DEBUG }, 06:14:28 DEBUG "result": [ 06:14:28 DEBUG "data": { 06:14:28 DEBUG "resultType": "vector", 06:14:28 DEBUG "status": "success", 06:14:28 DEBUG }, 06:14:28 DEBUG { 06:14:28 DEBUG "status": "success", 06:14:28 DEBUG "resultType": "vector", 06:14:28 DEBUG "result": [ 06:14:28 DEBUG { 06:14:28 DEBUG "result": [ 06:14:28 DEBUG { 06:14:28 DEBUG "metric": { 06:14:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:14:28 DEBUG }, 06:14:28 DEBUG "data": { 06:14:28 DEBUG { 06:14:28 DEBUG "value": [ 06:14:28 DEBUG "metric": { 06:14:28 DEBUG "value": [ 06:14:28 DEBUG "data": { 06:14:28 DEBUG "value": [ 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG "resultType": "vector", 06:14:28 DEBUG "metric": { 06:14:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:14:28 DEBUG "pod": "ds-idrepo-0" 06:14:28 DEBUG }, 06:14:28 DEBUG "resultType": "vector", 06:14:28 DEBUG 06:14:28 DEBUG [...] 06:14:28 DEBUG "result": [ 06:14:28 DEBUG { 06:14:28 DEBUG }, 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG "result": [ 06:14:28 DEBUG { 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG "metric": { 06:14:28 DEBUG "value": [ 06:14:28 DEBUG 06:14:28 DEBUG "value": [ 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG "metric": { 06:14:28 DEBUG 06:14:28 DEBUG "instance": "10.142.0.77:9100" 06:14:28 DEBUG }, 06:14:28 DEBUG 06:14:28 DEBUG [...] 06:14:28 DEBUG 06:14:28 DEBUG [...] 06:14:28 DEBUG "instance": "10.142.0.25:9100" 06:14:28 DEBUG }, 06:14:28 DEBUG "value": [ 06:14:28 DEBUG 1712726068, 06:14:28 DEBUG "369396438016" 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG "value": [ 06:14:28 DEBUG 06:14:28 DEBUG ] 06:14:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:14:28 DEBUG 06:14:28 DEBUG [...] 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG [...] 06:14:28 DEBUG }, 06:14:28 DEBUG { 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG } 06:14:28 DEBUG ] 06:14:28 DEBUG "value": [ 06:14:28 DEBUG "metric": { 06:14:28 DEBUG "pod": "ds-idrepo-0" 06:14:28 DEBUG }, 06:14:28 DEBUG "value": [ 06:14:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:14:28 DEBUG }, 06:14:28 DEBUG "value": [ 06:14:28 DEBUG 1712726068, 06:14:28 DEBUG "258809" 06:14:28 DEBUG ] 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG } 06:14:28 DEBUG "result": [ 06:14:28 DEBUG { 06:14:28 DEBUG "metric": { 06:14:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:14:28 DEBUG }, 06:14:28 DEBUG "value": [ 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG [...] 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:14:28 DEBUG }, 06:14:28 DEBUG "value": [ 06:14:28 DEBUG 1712726068, 06:14:28 DEBUG "1439.2666666666667" 06:14:28 DEBUG ] 06:14:28 DEBUG } 06:14:28 DEBUG ] 06:14:28 DEBUG } 06:14:28 DEBUG } 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG [...] 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:14:28 DEBUG }, 06:14:28 DEBUG "value": [ 06:14:28 DEBUG 1712726068, 06:14:28 DEBUG "1439.0666666666666" 06:14:28 DEBUG ] 06:14:28 DEBUG } 06:14:28 DEBUG ] 06:14:28 DEBUG } 06:14:28 DEBUG } 06:14:28 DEBUG } 06:14:28 DEBUG [...] 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG "pod": "ds-idrepo-1" 06:14:28 DEBUG }, 06:14:28 DEBUG "value": [ 06:14:28 DEBUG 1712726068, 06:14:28 DEBUG "0" 06:14:28 DEBUG ] 06:14:28 DEBUG } 06:14:28 DEBUG ] 06:14:28 DEBUG } 06:14:28 DEBUG } 06:14:28 DEBUG [...] 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG "pod": "ds-cts-2" 06:14:28 DEBUG }, 06:14:28 DEBUG "value": [ 06:14:28 DEBUG 1712726068, 06:14:28 DEBUG "0" 06:14:28 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:14:28 DEBUG 06:14:28 DEBUG } 06:14:28 DEBUG ] 06:14:28 DEBUG } 06:14:28 DEBUG ] 06:14:28 DEBUG }, 06:14:28 DEBUG "value": [ 06:14:28 DEBUG 1712726068, 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG "0" 06:14:28 DEBUG ] 06:14:28 DEBUG } 06:14:28 DEBUG ] 06:14:28 DEBUG } 06:14:28 DEBUG "pod": "ds-cts-2" 06:14:28 DEBUG } 06:14:28 DEBUG ] 06:14:28 DEBUG } 06:14:28 DEBUG } 06:14:28 DEBUG }, 06:14:28 DEBUG "instance": "10.142.0.77:9100" 06:14:28 DEBUG }, 06:14:28 DEBUG "value": [ 06:14:28 DEBUG } 06:14:28 DEBUG } 06:14:28 DEBUG "value": [ 06:14:28 DEBUG 1712726068, 06:14:28 DEBUG "28.9" 06:14:28 DEBUG ] 06:14:28 DEBUG } 06:14:28 DEBUG ] 06:14:28 DEBUG } 06:14:28 DEBUG } 06:14:28 DEBUG 1712726068, 06:14:28 DEBUG "42696378.733333334" 06:14:28 DEBUG ] 06:14:28 DEBUG } 06:14:28 DEBUG ] 06:14:28 DEBUG } 06:14:28 DEBUG } 06:14:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726068" 06:14:28 INFO [http_cmd]: http status code OK 06:14:28 DEBUG --- status code --- 06:14:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:28 DEBUG --- http response --- 06:14:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:14:28 DEBUG ----- output ----- 06:14:28 DEBUG { 06:14:28 DEBUG "status": "success", 06:14:28 DEBUG "data": { 06:14:28 DEBUG "resultType": "vector", 06:14:28 DEBUG "result": [ 06:14:28 DEBUG { 06:14:28 DEBUG "metric": { 06:14:28 DEBUG "instance": "10.142.0.25:9100" 06:14:28 DEBUG }, 06:14:28 DEBUG "value": [ 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG [...] 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG 06:14:28 DEBUG "instance": "10.142.0.77:9100" 06:14:28 DEBUG }, 06:14:28 DEBUG "value": [ 06:14:28 DEBUG 1712726068, 06:14:28 DEBUG "0.1444666666666688" 06:14:28 DEBUG ] 06:14:28 DEBUG } 06:14:28 DEBUG ] 06:14:28 DEBUG } 06:14:28 DEBUG } 06:14:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726069" 06:14:29 INFO [http_cmd]: http status code OK 06:14:29 DEBUG --- status code --- 06:14:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:29 DEBUG --- http response --- 06:14:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:14:29 DEBUG ----- output ----- 06:14:29 DEBUG { 06:14:29 DEBUG "status": "success", 06:14:29 DEBUG "data": { 06:14:29 DEBUG "resultType": "vector", 06:14:29 DEBUG "result": [ 06:14:29 DEBUG { 06:14:29 DEBUG "metric": { 06:14:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:14:29 DEBUG }, 06:14:29 DEBUG "value": [ 06:14:29 DEBUG 06:14:29 DEBUG 06:14:29 DEBUG 06:14:29 DEBUG [...] 06:14:29 DEBUG 06:14:29 DEBUG 06:14:29 DEBUG 06:14:29 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:14:29 DEBUG }, 06:14:29 DEBUG "value": [ 06:14:29 DEBUG 1712726069, 06:14:29 DEBUG "332.51720835195715" 06:14:29 DEBUG ] 06:14:29 DEBUG } 06:14:29 DEBUG ] 06:14:29 DEBUG } 06:14:29 DEBUG } 06:14:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712726069" 06:14:29 INFO [http_cmd]: http status code OK 06:14:29 DEBUG --- status code --- 06:14:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:29 DEBUG --- http response --- 06:14:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:14:29 DEBUG ----- output ----- 06:14:29 DEBUG { 06:14:29 DEBUG "status": "success", 06:14:29 DEBUG "data": { 06:14:29 DEBUG "resultType": "vector", 06:14:29 DEBUG "result": [ 06:14:29 DEBUG { 06:14:29 DEBUG "metric": { 06:14:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:14:29 DEBUG }, 06:14:29 DEBUG "value": [ 06:14:29 DEBUG 06:14:29 DEBUG 06:14:29 DEBUG 06:14:29 DEBUG [...] 06:14:29 DEBUG 06:14:29 DEBUG 06:14:29 DEBUG 06:14:29 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:14:29 DEBUG }, 06:14:29 DEBUG "value": [ 06:14:29 DEBUG 1712726069, 06:14:29 DEBUG "4.2152937734636176" 06:14:29 DEBUG ] 06:14:29 DEBUG } 06:14:29 DEBUG ] 06:14:29 DEBUG } 06:14:29 DEBUG } 06:14:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712726069" 06:14:29 INFO [http_cmd]: http status code OK 06:14:29 DEBUG --- status code --- 06:14:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:29 DEBUG --- http response --- 06:14:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:14:29 DEBUG ----- output ----- 06:14:29 DEBUG { 06:14:29 DEBUG "status": "success", 06:14:29 DEBUG "data": { 06:14:29 DEBUG "resultType": "vector", 06:14:29 DEBUG "result": [ 06:14:29 DEBUG { 06:14:29 DEBUG "metric": { 06:14:29 DEBUG "pod": "ds-idrepo-0" 06:14:29 DEBUG }, 06:14:29 DEBUG "value": [ 06:14:29 DEBUG 06:14:29 DEBUG 06:14:29 DEBUG 06:14:29 DEBUG [...] 06:14:29 DEBUG 06:14:29 DEBUG 06:14:29 DEBUG 06:14:29 DEBUG "pod": "ds-cts-2" 06:14:29 DEBUG }, 06:14:29 DEBUG "value": [ 06:14:29 DEBUG 1712726069, 06:14:29 DEBUG "0" 06:14:29 DEBUG ] 06:14:29 DEBUG } 06:14:29 DEBUG ] 06:14:29 DEBUG } 06:14:29 DEBUG } 06:14:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726070" 06:14:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726070" 06:14:30 INFO [http_cmd]: http status code OK 06:14:30 INFO [http_cmd]: http status code OK 06:14:30 DEBUG --- status code --- 06:14:30 DEBUG --- status code --- 06:14:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:30 DEBUG --- http response --- 06:14:30 DEBUG --- http response --- 06:14:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:14:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:14:30 DEBUG ----- output ----- 06:14:30 DEBUG { 06:14:30 DEBUG "status": "success", 06:14:30 DEBUG ----- output ----- 06:14:30 DEBUG "data": { 06:14:30 DEBUG { 06:14:30 DEBUG "resultType": "vector", 06:14:30 DEBUG "status": "success", 06:14:30 DEBUG "result": [ 06:14:30 DEBUG "data": { 06:14:30 DEBUG { 06:14:30 DEBUG "resultType": "vector", 06:14:30 DEBUG "metric": { 06:14:30 DEBUG "result": [ 06:14:30 DEBUG "pod": "ds-idrepo-2" 06:14:30 DEBUG { 06:14:30 DEBUG }, 06:14:30 DEBUG "metric": { 06:14:30 DEBUG "value": [ 06:14:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:14:30 DEBUG 06:14:30 DEBUG }, 06:14:30 DEBUG 06:14:30 DEBUG "value": [ 06:14:30 DEBUG 06:14:30 DEBUG 06:14:30 DEBUG [...] 06:14:30 DEBUG 06:14:30 DEBUG 06:14:30 DEBUG 06:14:30 DEBUG 06:14:30 DEBUG [...] 06:14:30 DEBUG 06:14:30 DEBUG 06:14:30 DEBUG "pod": "ds-cts-2" 06:14:30 DEBUG }, 06:14:30 DEBUG 06:14:30 DEBUG "value": [ 06:14:30 DEBUG 06:14:30 DEBUG 1712726070, 06:14:30 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:14:30 DEBUG "0.00005621888888888889" 06:14:30 DEBUG }, 06:14:30 DEBUG ] 06:14:30 DEBUG "value": [ 06:14:30 DEBUG } 06:14:30 DEBUG ] 06:14:30 DEBUG 1712726070, 06:14:30 DEBUG } 06:14:30 DEBUG "0.0013682155847087876" 06:14:30 DEBUG } 06:14:30 DEBUG ] 06:14:30 DEBUG } 06:14:30 DEBUG ] 06:14:30 DEBUG } 06:14:30 DEBUG } 06:14:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726070" 06:14:30 INFO [http_cmd]: http status code OK 06:14:30 DEBUG --- status code --- 06:14:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:30 DEBUG --- http response --- 06:14:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:14:30 DEBUG ----- output ----- 06:14:30 DEBUG { 06:14:30 DEBUG "status": "success", 06:14:30 DEBUG "data": { 06:14:30 DEBUG "resultType": "vector", 06:14:30 DEBUG "result": [ 06:14:30 DEBUG { 06:14:30 DEBUG "metric": { 06:14:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:14:30 DEBUG }, 06:14:30 DEBUG "value": [ 06:14:30 DEBUG 06:14:30 DEBUG 06:14:30 DEBUG 06:14:30 DEBUG [...] 06:14:30 DEBUG 06:14:30 DEBUG 06:14:30 DEBUG 06:14:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:14:30 DEBUG }, 06:14:30 DEBUG "value": [ 06:14:30 DEBUG 1712726070, 06:14:30 DEBUG "1439.4333333333334" 06:14:30 DEBUG ] 06:14:30 DEBUG } 06:14:30 DEBUG ] 06:14:30 DEBUG } 06:14:30 DEBUG } 06:14:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712726070" 06:14:30 INFO [http_cmd]: http status code OK 06:14:30 DEBUG --- status code --- 06:14:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:30 DEBUG --- http response --- 06:14:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:14:30 DEBUG ----- output ----- 06:14:30 DEBUG { 06:14:30 DEBUG "status": "success", 06:14:30 DEBUG "data": { 06:14:30 DEBUG "resultType": "vector", 06:14:30 DEBUG "result": [ 06:14:30 DEBUG { 06:14:30 DEBUG "metric": { 06:14:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:14:30 DEBUG }, 06:14:30 DEBUG "value": [ 06:14:30 DEBUG 06:14:30 DEBUG 06:14:30 DEBUG 06:14:30 DEBUG [...] 06:14:30 DEBUG 06:14:30 DEBUG 06:14:30 DEBUG 06:14:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:14:30 DEBUG }, 06:14:30 DEBUG "value": [ 06:14:30 DEBUG 1712726070, 06:14:30 DEBUG "0.05" 06:14:30 DEBUG ] 06:14:30 DEBUG } 06:14:30 DEBUG ] 06:14:30 DEBUG } 06:14:30 DEBUG } 06:14:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726070" 06:14:30 INFO [http_cmd]: http status code OK 06:14:30 DEBUG --- status code --- 06:14:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:30 DEBUG --- http response --- 06:14:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:14:30 DEBUG ----- output ----- 06:14:30 DEBUG { 06:14:30 DEBUG "status": "success", 06:14:30 DEBUG "data": { 06:14:30 DEBUG "resultType": "vector", 06:14:30 DEBUG "result": [ 06:14:30 DEBUG { 06:14:30 DEBUG "metric": { 06:14:30 DEBUG "pod": "ds-idrepo-0" 06:14:30 DEBUG }, 06:14:30 DEBUG "value": [ 06:14:30 DEBUG 06:14:30 DEBUG 06:14:30 DEBUG 06:14:30 DEBUG [...] 06:14:30 DEBUG 06:14:30 DEBUG 06:14:30 DEBUG 06:14:30 DEBUG "pod": "ds-idrepo-1" 06:14:30 DEBUG }, 06:14:30 DEBUG "value": [ 06:14:30 DEBUG 1712726070, 06:14:30 DEBUG "0" 06:14:30 DEBUG ] 06:14:30 DEBUG } 06:14:30 DEBUG ] 06:14:30 DEBUG } 06:14:30 DEBUG } 06:14:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726070" 06:14:30 INFO [http_cmd]: http status code OK 06:14:30 DEBUG --- status code --- 06:14:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:30 DEBUG --- http response --- 06:14:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:14:30 DEBUG ----- output ----- 06:14:30 DEBUG { 06:14:30 DEBUG "status": "success", 06:14:30 DEBUG "data": { 06:14:30 DEBUG "resultType": "vector", 06:14:30 DEBUG "result": [ 06:14:30 DEBUG { 06:14:30 DEBUG "metric": { 06:14:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:14:30 DEBUG }, 06:14:30 DEBUG "value": [ 06:14:30 DEBUG 06:14:30 DEBUG 06:14:30 DEBUG 06:14:30 DEBUG [...] 06:14:30 DEBUG 06:14:30 DEBUG 06:14:30 DEBUG 06:14:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:14:30 DEBUG }, 06:14:30 DEBUG "value": [ 06:14:30 DEBUG 1712726070, 06:14:30 DEBUG "1445.9" 06:14:30 DEBUG ] 06:14:30 DEBUG } 06:14:30 DEBUG ] 06:14:30 DEBUG } 06:14:30 DEBUG } 06:14:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726070" 06:14:30 INFO [http_cmd]: http status code OK 06:14:30 DEBUG --- status code --- 06:14:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:30 DEBUG --- http response --- 06:14:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:14:30 DEBUG ----- output ----- 06:14:30 DEBUG { 06:14:30 DEBUG "status": "success", 06:14:30 DEBUG "data": { 06:14:30 DEBUG "resultType": "vector", 06:14:30 DEBUG "result": [ 06:14:30 DEBUG { 06:14:30 DEBUG "metric": { 06:14:30 DEBUG "pod": "ds-idrepo-0" 06:14:30 DEBUG }, 06:14:30 DEBUG "value": [ 06:14:30 DEBUG 06:14:30 DEBUG 06:14:30 DEBUG 06:14:30 DEBUG [...] 06:14:30 DEBUG 06:14:30 DEBUG 06:14:30 DEBUG 06:14:30 DEBUG "pod": "ds-idrepo-1" 06:14:30 DEBUG }, 06:14:30 DEBUG "value": [ 06:14:30 DEBUG 1712726070, 06:14:30 DEBUG "0" 06:14:30 DEBUG ] 06:14:30 DEBUG } 06:14:30 DEBUG ] 06:14:30 DEBUG } 06:14:30 DEBUG } 06:14:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726071" 06:14:31 INFO [http_cmd]: http status code OK 06:14:31 DEBUG --- status code --- 06:14:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:31 DEBUG --- http response --- 06:14:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:14:31 DEBUG ----- output ----- 06:14:31 DEBUG { 06:14:31 DEBUG "status": "success", 06:14:31 DEBUG "data": { 06:14:31 DEBUG "resultType": "vector", 06:14:31 DEBUG "result": [ 06:14:31 DEBUG { 06:14:31 DEBUG "metric": { 06:14:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:14:31 DEBUG }, 06:14:31 DEBUG "value": [ 06:14:31 DEBUG 06:14:31 DEBUG 06:14:31 DEBUG 06:14:31 DEBUG [...] 06:14:31 DEBUG 06:14:31 DEBUG 06:14:31 DEBUG 06:14:31 DEBUG "pod": "end-user-ui-6d9fd5487b-2s47b" 06:14:31 DEBUG }, 06:14:31 DEBUG "value": [ 06:14:31 DEBUG 1712726071, 06:14:31 DEBUG "0" 06:14:31 DEBUG ] 06:14:31 DEBUG } 06:14:31 DEBUG ] 06:14:31 DEBUG } 06:14:31 DEBUG } 06:14:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712726071" 06:14:31 INFO [http_cmd]: http status code OK 06:14:31 DEBUG --- status code --- 06:14:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:31 DEBUG --- http response --- 06:14:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:14:31 DEBUG ----- output ----- 06:14:31 DEBUG { 06:14:31 DEBUG "status": "success", 06:14:31 DEBUG "data": { 06:14:31 DEBUG "resultType": "vector", 06:14:31 DEBUG "result": [ 06:14:31 DEBUG { 06:14:31 DEBUG "metric": { 06:14:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:14:31 DEBUG }, 06:14:31 DEBUG "value": [ 06:14:31 DEBUG 06:14:31 DEBUG 06:14:31 DEBUG 06:14:31 DEBUG [...] 06:14:31 DEBUG 06:14:31 DEBUG 06:14:31 DEBUG 06:14:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:14:31 DEBUG }, 06:14:31 DEBUG "value": [ 06:14:31 DEBUG 1712726071, 06:14:31 DEBUG "2487742464" 06:14:31 DEBUG ] 06:14:31 DEBUG } 06:14:31 DEBUG ] 06:14:31 DEBUG } 06:14:31 DEBUG } 06:14:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726071" 06:14:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726071" 06:14:31 INFO [http_cmd]: http status code OK 06:14:31 DEBUG --- status code --- 06:14:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:31 DEBUG --- http response --- 06:14:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:14:31 DEBUG ----- output ----- 06:14:31 DEBUG { 06:14:31 DEBUG "status": "success", 06:14:31 DEBUG "data": { 06:14:31 DEBUG "resultType": "vector", 06:14:31 DEBUG "result": [ 06:14:31 DEBUG { 06:14:31 DEBUG "metric": { 06:14:31 DEBUG "instance": "10.142.0.25:9100" 06:14:31 DEBUG }, 06:14:31 DEBUG "value": [ 06:14:31 DEBUG 06:14:31 DEBUG 06:14:31 DEBUG 06:14:31 DEBUG [...] 06:14:31 DEBUG 06:14:31 DEBUG 06:14:31 DEBUG 06:14:31 DEBUG "instance": "10.142.0.77:9100" 06:14:31 DEBUG }, 06:14:31 DEBUG "value": [ 06:14:31 DEBUG 1712726071, 06:14:31 DEBUG "42700249.63333333" 06:14:31 DEBUG ] 06:14:31 DEBUG } 06:14:31 DEBUG ] 06:14:31 DEBUG } 06:14:31 DEBUG } 06:14:31 INFO [http_cmd]: http status code OK 06:14:31 DEBUG --- status code --- 06:14:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:14:31 DEBUG --- http response --- 06:14:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:14:31 DEBUG ----- output ----- 06:14:31 DEBUG { 06:14:31 DEBUG "status": "success", 06:14:31 DEBUG "data": { 06:14:31 DEBUG "resultType": "vector", 06:14:31 DEBUG "result": [ 06:14:31 DEBUG { 06:14:31 DEBUG "metric": { 06:14:31 DEBUG "instance": "10.142.0.25:9100" 06:14:31 DEBUG }, 06:14:31 DEBUG "value": [ 06:14:31 DEBUG 06:14:31 DEBUG 06:14:31 DEBUG 06:14:31 DEBUG [...] 06:14:31 DEBUG 06:14:31 DEBUG 06:14:31 DEBUG 06:14:31 DEBUG "instance": "10.142.0.77:9100" 06:14:31 DEBUG }, 06:14:31 DEBUG "value": [ 06:14:31 DEBUG 1712726071, 06:14:31 DEBUG "0.044999999999999984" 06:14:31 DEBUG ] 06:14:31 DEBUG } 06:14:31 DEBUG ] 06:14:31 DEBUG } 06:14:31 DEBUG } 06:14:48 INFO 06:14:48 INFO [loop_until]: kubectl --namespace=xlou top pods 06:14:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:14:49 INFO [loop_until]: OK (rc = 0) 06:14:49 DEBUG --- stdout --- 06:14:49 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10839m 8990Mi am-654c4fcbcf-9scrj 9636m 8446Mi am-654c4fcbcf-v5qb5 8562m 8447Mi ds-cts-0 8137m 12793Mi ds-cts-1 8531m 12705Mi ds-cts-2 7817m 12908Mi ds-idrepo-0 10681m 5622Mi ds-idrepo-1 357m 5672Mi ds-idrepo-2 373m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 5m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4123m 2259Mi 06:14:49 DEBUG --- stderr --- 06:14:49 DEBUG 06:14:56 INFO 06:14:56 INFO [loop_until]: kubectl --namespace=xlou top node 06:14:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:14:56 INFO [loop_until]: OK (rc = 0) 06:14:56 DEBUG --- stdout --- 06:14:56 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 93m 0% 2310Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9298m 58% 9319Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 81m 0% 3257Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13198m 83% 6641Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10950m 68% 9332Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11971m 75% 9799Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 76m 0% 858Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10823m 68% 6299Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8334m 52% 13488Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 7866m 49% 13470Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 431m 2% 6388Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8679m 54% 13363Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 402m 2% 6351Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4343m 27% 5875Mi 10% 06:14:56 DEBUG --- stderr --- 06:14:56 DEBUG 06:15:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712726127" 06:15:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726127" 06:15:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726127" 06:15:27 INFO [http_cmd]: http status code OK 06:15:27 DEBUG --- status code --- 06:15:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:27 DEBUG --- http response --- 06:15:27 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:15:27 INFO [http_cmd]: http status code OK 06:15:27 ERROR [http_cmd]: ERROR 06:15:27 DEBUG --- status code --- 06:15:27 ERROR --- status code --- 06:15:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:27 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:15:27 DEBUG --- http response --- 06:15:27 ERROR --- http response --- 06:15:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:15:27 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:15:27 DEBUG ----- output ----- 06:15:27 DEBUG { 06:15:27 DEBUG "status": "success", 06:15:27 DEBUG "data": { 06:15:27 DEBUG "resultType": "vector", 06:15:27 DEBUG "result": [ 06:15:27 DEBUG { 06:15:27 DEBUG "metric": { 06:15:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:15:27 DEBUG }, 06:15:27 DEBUG "value": [ 06:15:27 DEBUG 06:15:27 DEBUG 06:15:27 DEBUG 06:15:27 DEBUG [...] 06:15:27 DEBUG 06:15:27 DEBUG 06:15:27 DEBUG 06:15:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:15:27 DEBUG }, 06:15:27 DEBUG "value": [ 06:15:27 DEBUG 1712726127, 06:15:27 DEBUG "0" 06:15:27 DEBUG ] 06:15:27 DEBUG } 06:15:27 DEBUG ] 06:15:27 DEBUG } 06:15:27 DEBUG } 06:15:27 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:15:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726128" 06:15:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726128" 06:15:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726128" 06:15:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726128" 06:15:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726128" 06:15:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726128" 06:15:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726128" 06:15:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726128" 06:15:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726128" 06:15:28 INFO [http_cmd]: http status code OK 06:15:28 INFO [http_cmd]: http status code OK 06:15:28 DEBUG --- status code --- 06:15:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:28 INFO [http_cmd]: http status code OK 06:15:28 INFO [http_cmd]: http status code OK 06:15:28 DEBUG --- http response --- 06:15:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:15:28 DEBUG ----- output ----- 06:15:28 DEBUG --- status code --- 06:15:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:28 DEBUG { 06:15:28 DEBUG "status": "success", 06:15:28 DEBUG --- status code --- 06:15:28 INFO [http_cmd]: http status code OK 06:15:28 DEBUG --- status code --- 06:15:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:28 DEBUG --- http response --- 06:15:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:28 DEBUG --- http response --- 06:15:28 DEBUG --- status code --- 06:15:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:28 INFO [http_cmd]: http status code OK 06:15:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:15:28 DEBUG --- http response --- 06:15:28 INFO [http_cmd]: http status code OK 06:15:28 DEBUG --- http response --- 06:15:28 DEBUG "data": { 06:15:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:15:28 INFO [http_cmd]: http status code OK 06:15:28 INFO [http_cmd]: http status code OK 06:15:28 DEBUG --- status code --- 06:15:28 DEBUG ----- output ----- 06:15:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:15:28 DEBUG --- status code --- 06:15:28 DEBUG "resultType": "vector", 06:15:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:15:28 DEBUG ----- output ----- 06:15:28 DEBUG --- status code --- 06:15:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:28 DEBUG { 06:15:28 DEBUG ----- output ----- 06:15:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:28 DEBUG "result": [ 06:15:28 DEBUG ----- output ----- 06:15:28 DEBUG --- status code --- 06:15:28 DEBUG { 06:15:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:28 DEBUG --- http response --- 06:15:28 DEBUG "status": "success", 06:15:28 DEBUG { 06:15:28 DEBUG --- http response --- 06:15:28 DEBUG { 06:15:28 DEBUG { 06:15:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:28 DEBUG "status": "success", 06:15:28 DEBUG --- http response --- 06:15:28 DEBUG "data": { 06:15:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:15:28 DEBUG "status": "success", 06:15:28 DEBUG "data": { 06:15:28 DEBUG "resultType": "vector", 06:15:28 DEBUG "status": "success", 06:15:28 DEBUG --- http response --- 06:15:28 DEBUG "data": { 06:15:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:15:28 DEBUG "resultType": "vector", 06:15:28 DEBUG ----- output ----- 06:15:28 DEBUG { 06:15:28 DEBUG "status": "success", 06:15:28 DEBUG "metric": { 06:15:28 DEBUG "data": { 06:15:28 DEBUG "resultType": "vector", 06:15:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:15:28 DEBUG ----- output ----- 06:15:28 DEBUG "result": [ 06:15:28 DEBUG { 06:15:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:15:28 DEBUG "result": [ 06:15:28 DEBUG { 06:15:28 DEBUG "metric": { 06:15:28 DEBUG "resultType": "vector", 06:15:28 DEBUG { 06:15:28 DEBUG "status": "success", 06:15:28 DEBUG "metric": { 06:15:28 DEBUG "instance": "10.142.0.25:9100" 06:15:28 DEBUG ----- output ----- 06:15:28 DEBUG "result": [ 06:15:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:15:28 DEBUG }, 06:15:28 DEBUG "result": [ 06:15:28 DEBUG "data": { 06:15:28 DEBUG ----- output ----- 06:15:28 DEBUG }, 06:15:28 DEBUG "value": [ 06:15:28 DEBUG { 06:15:28 DEBUG { 06:15:28 DEBUG "value": [ 06:15:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:15:28 DEBUG }, 06:15:28 DEBUG "resultType": "vector", 06:15:28 DEBUG { 06:15:28 DEBUG 06:15:28 DEBUG 06:15:28 DEBUG "status": "success", 06:15:28 DEBUG "metric": { 06:15:28 DEBUG 06:15:28 DEBUG "value": [ 06:15:28 DEBUG { 06:15:28 DEBUG "result": [ 06:15:28 DEBUG "status": "success", 06:15:28 DEBUG 06:15:28 DEBUG [...] 06:15:28 DEBUG 06:15:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:15:28 DEBUG 06:15:28 DEBUG 06:15:28 DEBUG "metric": { 06:15:28 DEBUG "pod": "ds-idrepo-0" 06:15:28 DEBUG }, 06:15:28 DEBUG 06:15:28 DEBUG "data": { 06:15:28 DEBUG "data": { 06:15:28 DEBUG }, 06:15:28 DEBUG 06:15:28 DEBUG 06:15:28 DEBUG "value": [ 06:15:28 DEBUG 06:15:28 DEBUG 06:15:28 DEBUG { 06:15:28 DEBUG "metric": { 06:15:28 DEBUG "resultType": "vector", 06:15:28 DEBUG "value": [ 06:15:28 DEBUG [...] 06:15:28 DEBUG 06:15:28 DEBUG 06:15:28 DEBUG 06:15:28 DEBUG "instance": "10.142.0.77:9100" 06:15:28 DEBUG "instance": "10.142.0.25:9100" 06:15:28 DEBUG }, 06:15:28 DEBUG "value": [ 06:15:28 DEBUG "result": [ 06:15:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:15:28 DEBUG }, 06:15:28 DEBUG "value": [ 06:15:28 DEBUG 1712726128, 06:15:28 DEBUG }, 06:15:28 DEBUG "value": [ 06:15:28 DEBUG 1712726128, 06:15:28 DEBUG 06:15:28 DEBUG { 06:15:28 DEBUG "1440.5333333333333" 06:15:28 DEBUG ] 06:15:28 DEBUG 06:15:28 DEBUG [...] 06:15:28 DEBUG "371241337856" 06:15:28 DEBUG ] 06:15:28 DEBUG 06:15:28 DEBUG 06:15:28 DEBUG 06:15:28 DEBUG [...] 06:15:28 DEBUG 06:15:28 DEBUG 06:15:28 DEBUG 06:15:28 DEBUG } 06:15:28 DEBUG ] 06:15:28 DEBUG "metric": { 06:15:28 DEBUG "instance": "10.142.0.77:9100" 06:15:28 DEBUG }, 06:15:28 DEBUG "value": [ 06:15:28 DEBUG 1712726128, 06:15:28 DEBUG "42332596.43333334" 06:15:28 DEBUG ] 06:15:28 DEBUG } 06:15:28 DEBUG } 06:15:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:15:28 DEBUG "resultType": "vector", 06:15:28 DEBUG "result": [ 06:15:28 DEBUG 06:15:28 DEBUG 06:15:28 DEBUG 06:15:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:15:28 DEBUG }, 06:15:28 DEBUG "value": [ 06:15:28 DEBUG { 06:15:28 DEBUG "metric": { 06:15:28 DEBUG }, 06:15:28 DEBUG "value": [ 06:15:28 DEBUG 1712726128, 06:15:28 DEBUG "258698.5" 06:15:28 DEBUG 06:15:28 DEBUG 06:15:28 DEBUG "pod": "ds-idrepo-0" 06:15:28 DEBUG }, 06:15:28 DEBUG "value": [ 06:15:28 DEBUG 06:15:28 DEBUG 06:15:28 DEBUG 06:15:28 DEBUG 06:15:28 DEBUG [...] 06:15:28 DEBUG [...] 06:15:28 DEBUG 06:15:28 DEBUG 06:15:28 DEBUG 06:15:28 DEBUG ] 06:15:28 DEBUG } 06:15:28 DEBUG 06:15:28 DEBUG 06:15:28 DEBUG "pod": "ds-cts-2" 06:15:28 DEBUG "data": { 06:15:28 DEBUG 06:15:28 DEBUG 06:15:28 DEBUG 06:15:28 DEBUG ] 06:15:28 DEBUG 06:15:28 DEBUG } 06:15:28 DEBUG }, 06:15:28 DEBUG "resultType": "vector", 06:15:28 DEBUG 06:15:28 DEBUG [...] 06:15:28 DEBUG 06:15:28 DEBUG } 06:15:28 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:15:28 DEBUG }, 06:15:28 DEBUG "value": [ 06:15:28 DEBUG "result": [ 06:15:28 DEBUG [...] 06:15:28 DEBUG 06:15:28 DEBUG 06:15:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:15:28 DEBUG }, 06:15:28 DEBUG "value": [ 06:15:28 DEBUG 1712726128, 06:15:28 DEBUG { 06:15:28 DEBUG 06:15:28 DEBUG } 06:15:28 DEBUG 1712726128, 06:15:28 DEBUG "1441.5" 06:15:28 DEBUG ] 06:15:28 DEBUG "0" 06:15:28 DEBUG "metric": { 06:15:28 DEBUG ] 06:15:28 DEBUG 06:15:28 DEBUG 06:15:28 DEBUG } 06:15:28 DEBUG "value": [ 06:15:28 DEBUG ] 06:15:28 DEBUG "pod": "ds-idrepo-0" 06:15:28 DEBUG }, 06:15:28 DEBUG "pod": "ds-idrepo-1" 06:15:28 DEBUG } 06:15:28 DEBUG ] 06:15:28 DEBUG 1712726128, 06:15:28 DEBUG } 06:15:28 DEBUG "value": [ 06:15:28 DEBUG 06:15:28 DEBUG }, 06:15:28 DEBUG ] 06:15:28 DEBUG } 06:15:28 DEBUG } 06:15:28 DEBUG ] 06:15:28 DEBUG "0" 06:15:28 DEBUG ] 06:15:28 DEBUG "value": [ 06:15:28 DEBUG } 06:15:28 DEBUG } 06:15:28 DEBUG 06:15:28 DEBUG } 06:15:28 DEBUG ] 06:15:28 DEBUG 1712726128, 06:15:28 DEBUG } 06:15:28 DEBUG } 06:15:28 DEBUG 06:15:28 DEBUG } 06:15:28 DEBUG } 06:15:28 DEBUG "0" 06:15:28 DEBUG } 06:15:28 DEBUG } 06:15:28 DEBUG ] 06:15:28 DEBUG } 06:15:28 DEBUG ] 06:15:28 DEBUG } 06:15:28 DEBUG } 06:15:28 DEBUG [...] 06:15:28 DEBUG 06:15:28 DEBUG 06:15:28 DEBUG 06:15:28 DEBUG "pod": "ds-cts-2" 06:15:28 DEBUG }, 06:15:28 DEBUG "value": [ 06:15:28 DEBUG 1712726128, 06:15:28 DEBUG "29.833333333333332" 06:15:28 DEBUG ] 06:15:28 DEBUG } 06:15:28 DEBUG ] 06:15:28 DEBUG } 06:15:28 DEBUG } 06:15:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726128" 06:15:28 INFO [http_cmd]: http status code OK 06:15:28 DEBUG --- status code --- 06:15:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:28 DEBUG --- http response --- 06:15:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:15:28 DEBUG ----- output ----- 06:15:28 DEBUG { 06:15:28 DEBUG "status": "success", 06:15:28 DEBUG "data": { 06:15:28 DEBUG "resultType": "vector", 06:15:28 DEBUG "result": [ 06:15:28 DEBUG { 06:15:28 DEBUG "metric": { 06:15:28 DEBUG "instance": "10.142.0.25:9100" 06:15:28 DEBUG }, 06:15:28 DEBUG "value": [ 06:15:28 DEBUG 06:15:28 DEBUG 06:15:28 DEBUG 06:15:28 DEBUG [...] 06:15:28 DEBUG 06:15:28 DEBUG 06:15:28 DEBUG 06:15:28 DEBUG "instance": "10.142.0.77:9100" 06:15:28 DEBUG }, 06:15:28 DEBUG "value": [ 06:15:28 DEBUG 1712726128, 06:15:28 DEBUG "0.1504333333333354" 06:15:28 DEBUG ] 06:15:28 DEBUG } 06:15:28 DEBUG ] 06:15:28 DEBUG } 06:15:28 DEBUG } 06:15:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726129" 06:15:29 INFO [http_cmd]: http status code OK 06:15:29 DEBUG --- status code --- 06:15:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:29 DEBUG --- http response --- 06:15:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:15:29 DEBUG ----- output ----- 06:15:29 DEBUG { 06:15:29 DEBUG "status": "success", 06:15:29 DEBUG "data": { 06:15:29 DEBUG "resultType": "vector", 06:15:29 DEBUG "result": [ 06:15:29 DEBUG { 06:15:29 DEBUG "metric": { 06:15:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:15:29 DEBUG }, 06:15:29 DEBUG "value": [ 06:15:29 DEBUG 06:15:29 DEBUG 06:15:29 DEBUG 06:15:29 DEBUG [...] 06:15:29 DEBUG 06:15:29 DEBUG 06:15:29 DEBUG 06:15:29 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:15:29 DEBUG }, 06:15:29 DEBUG "value": [ 06:15:29 DEBUG 1712726129, 06:15:29 DEBUG "76602252.34254707" 06:15:29 DEBUG ] 06:15:29 DEBUG } 06:15:29 DEBUG ] 06:15:29 DEBUG } 06:15:29 DEBUG } 06:15:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712726129" 06:15:29 INFO [http_cmd]: http status code OK 06:15:29 DEBUG --- status code --- 06:15:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:29 DEBUG --- http response --- 06:15:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:15:29 DEBUG ----- output ----- 06:15:29 DEBUG { 06:15:29 DEBUG "status": "success", 06:15:29 DEBUG "data": { 06:15:29 DEBUG "resultType": "vector", 06:15:29 DEBUG "result": [ 06:15:29 DEBUG { 06:15:29 DEBUG "metric": { 06:15:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:15:29 DEBUG }, 06:15:29 DEBUG "value": [ 06:15:29 DEBUG 06:15:29 DEBUG 06:15:29 DEBUG 06:15:29 DEBUG [...] 06:15:29 DEBUG 06:15:29 DEBUG 06:15:29 DEBUG 06:15:29 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:15:29 DEBUG }, 06:15:29 DEBUG "value": [ 06:15:29 DEBUG 1712726129, 06:15:29 DEBUG "4.247439073574545" 06:15:29 DEBUG ] 06:15:29 DEBUG } 06:15:29 DEBUG ] 06:15:29 DEBUG } 06:15:29 DEBUG } 06:15:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712726129" 06:15:29 INFO [http_cmd]: http status code OK 06:15:29 DEBUG --- status code --- 06:15:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:29 DEBUG --- http response --- 06:15:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:15:29 DEBUG ----- output ----- 06:15:29 DEBUG { 06:15:29 DEBUG "status": "success", 06:15:29 DEBUG "data": { 06:15:29 DEBUG "resultType": "vector", 06:15:29 DEBUG "result": [ 06:15:29 DEBUG { 06:15:29 DEBUG "metric": { 06:15:29 DEBUG "pod": "ds-idrepo-0" 06:15:29 DEBUG }, 06:15:29 DEBUG "value": [ 06:15:29 DEBUG 06:15:29 DEBUG 06:15:29 DEBUG 06:15:29 DEBUG [...] 06:15:29 DEBUG 06:15:29 DEBUG 06:15:29 DEBUG 06:15:29 DEBUG "pod": "ds-cts-2" 06:15:29 DEBUG }, 06:15:29 DEBUG "value": [ 06:15:29 DEBUG 1712726129, 06:15:29 DEBUG "0" 06:15:29 DEBUG ] 06:15:29 DEBUG } 06:15:29 DEBUG ] 06:15:29 DEBUG } 06:15:29 DEBUG } 06:15:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726130" 06:15:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726130" 06:15:30 INFO [http_cmd]: http status code OK 06:15:30 INFO [http_cmd]: http status code OK 06:15:30 DEBUG --- status code --- 06:15:30 DEBUG --- status code --- 06:15:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:30 DEBUG --- http response --- 06:15:30 DEBUG --- http response --- 06:15:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:15:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:15:30 DEBUG ----- output ----- 06:15:30 DEBUG ----- output ----- 06:15:30 DEBUG { 06:15:30 DEBUG { 06:15:30 DEBUG "status": "success", 06:15:30 DEBUG "status": "success", 06:15:30 DEBUG "data": { 06:15:30 DEBUG "data": { 06:15:30 DEBUG "resultType": "vector", 06:15:30 DEBUG "resultType": "vector", 06:15:30 DEBUG "result": [ 06:15:30 DEBUG "result": [ 06:15:30 DEBUG { 06:15:30 DEBUG { 06:15:30 DEBUG "metric": { 06:15:30 DEBUG "metric": { 06:15:30 DEBUG "pod": "ds-idrepo-2" 06:15:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:15:30 DEBUG }, 06:15:30 DEBUG }, 06:15:30 DEBUG "value": [ 06:15:30 DEBUG 06:15:30 DEBUG "value": [ 06:15:30 DEBUG 06:15:30 DEBUG 06:15:30 DEBUG 06:15:30 DEBUG [...] 06:15:30 DEBUG 06:15:30 DEBUG 06:15:30 DEBUG 06:15:30 DEBUG 06:15:30 DEBUG "pod": "ds-cts-2" 06:15:30 DEBUG }, 06:15:30 DEBUG 06:15:30 DEBUG [...] 06:15:30 DEBUG 06:15:30 DEBUG 06:15:30 DEBUG 06:15:30 DEBUG "value": [ 06:15:30 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:15:30 DEBUG 1712726130, 06:15:30 DEBUG }, 06:15:30 DEBUG "value": [ 06:15:30 DEBUG 1712726130, 06:15:30 DEBUG "0.006816024086264458" 06:15:30 DEBUG ] 06:15:30 DEBUG } 06:15:30 DEBUG ] 06:15:30 DEBUG } 06:15:30 DEBUG } 06:15:30 DEBUG "0.000022885555555555554" 06:15:30 DEBUG ] 06:15:30 DEBUG } 06:15:30 DEBUG ] 06:15:30 DEBUG } 06:15:30 DEBUG } 06:15:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726130" 06:15:30 INFO [http_cmd]: http status code OK 06:15:30 DEBUG --- status code --- 06:15:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:30 DEBUG --- http response --- 06:15:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:15:30 DEBUG ----- output ----- 06:15:30 DEBUG { 06:15:30 DEBUG "status": "success", 06:15:30 DEBUG "data": { 06:15:30 DEBUG "resultType": "vector", 06:15:30 DEBUG "result": [ 06:15:30 DEBUG { 06:15:30 DEBUG "metric": { 06:15:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:15:30 DEBUG }, 06:15:30 DEBUG "value": [ 06:15:30 DEBUG 06:15:30 DEBUG 06:15:30 DEBUG 06:15:30 DEBUG [...] 06:15:30 DEBUG 06:15:30 DEBUG 06:15:30 DEBUG 06:15:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:15:30 DEBUG }, 06:15:30 DEBUG "value": [ 06:15:30 DEBUG 1712726130, 06:15:30 DEBUG "1442.0333333333333" 06:15:30 DEBUG ] 06:15:30 DEBUG } 06:15:30 DEBUG ] 06:15:30 DEBUG } 06:15:30 DEBUG } 06:15:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712726130" 06:15:30 INFO [http_cmd]: http status code OK 06:15:30 DEBUG --- status code --- 06:15:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:30 DEBUG --- http response --- 06:15:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:15:30 DEBUG ----- output ----- 06:15:30 DEBUG { 06:15:30 DEBUG "status": "success", 06:15:30 DEBUG "data": { 06:15:30 DEBUG "resultType": "vector", 06:15:30 DEBUG "result": [ 06:15:30 DEBUG { 06:15:30 DEBUG "metric": { 06:15:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:15:30 DEBUG }, 06:15:30 DEBUG "value": [ 06:15:30 DEBUG 06:15:30 DEBUG 06:15:30 DEBUG 06:15:30 DEBUG [...] 06:15:30 DEBUG 06:15:30 DEBUG 06:15:30 DEBUG 06:15:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:15:30 DEBUG }, 06:15:30 DEBUG "value": [ 06:15:30 DEBUG 1712726130, 06:15:30 DEBUG "0.05" 06:15:30 DEBUG ] 06:15:30 DEBUG } 06:15:30 DEBUG ] 06:15:30 DEBUG } 06:15:30 DEBUG } 06:15:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726130" 06:15:30 INFO [http_cmd]: http status code OK 06:15:30 DEBUG --- status code --- 06:15:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:30 DEBUG --- http response --- 06:15:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:15:30 DEBUG ----- output ----- 06:15:30 DEBUG { 06:15:30 DEBUG "status": "success", 06:15:30 DEBUG "data": { 06:15:30 DEBUG "resultType": "vector", 06:15:30 DEBUG "result": [ 06:15:30 DEBUG { 06:15:30 DEBUG "metric": { 06:15:30 DEBUG "pod": "ds-idrepo-0" 06:15:30 DEBUG }, 06:15:30 DEBUG "value": [ 06:15:30 DEBUG 06:15:30 DEBUG 06:15:30 DEBUG 06:15:30 DEBUG [...] 06:15:30 DEBUG 06:15:30 DEBUG 06:15:30 DEBUG 06:15:30 DEBUG "pod": "ds-idrepo-1" 06:15:30 DEBUG }, 06:15:30 DEBUG "value": [ 06:15:30 DEBUG 1712726130, 06:15:30 DEBUG "0" 06:15:30 DEBUG ] 06:15:30 DEBUG } 06:15:30 DEBUG ] 06:15:30 DEBUG } 06:15:30 DEBUG } 06:15:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726130" 06:15:30 INFO [http_cmd]: http status code OK 06:15:30 DEBUG --- status code --- 06:15:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:30 DEBUG --- http response --- 06:15:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:15:30 DEBUG ----- output ----- 06:15:30 DEBUG { 06:15:30 DEBUG "status": "success", 06:15:30 DEBUG "data": { 06:15:30 DEBUG "resultType": "vector", 06:15:30 DEBUG "result": [ 06:15:30 DEBUG { 06:15:30 DEBUG "metric": { 06:15:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:15:30 DEBUG }, 06:15:30 DEBUG "value": [ 06:15:30 DEBUG 06:15:30 DEBUG 06:15:30 DEBUG 06:15:30 DEBUG [...] 06:15:30 DEBUG 06:15:30 DEBUG 06:15:30 DEBUG 06:15:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:15:30 DEBUG }, 06:15:30 DEBUG "value": [ 06:15:30 DEBUG 1712726130, 06:15:30 DEBUG "1446.3333333333333" 06:15:30 DEBUG ] 06:15:30 DEBUG } 06:15:30 DEBUG ] 06:15:30 DEBUG } 06:15:30 DEBUG } 06:15:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726130" 06:15:30 INFO [http_cmd]: http status code OK 06:15:30 DEBUG --- status code --- 06:15:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:30 DEBUG --- http response --- 06:15:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:15:30 DEBUG ----- output ----- 06:15:30 DEBUG { 06:15:30 DEBUG "status": "success", 06:15:30 DEBUG "data": { 06:15:30 DEBUG "resultType": "vector", 06:15:30 DEBUG "result": [ 06:15:30 DEBUG { 06:15:30 DEBUG "metric": { 06:15:30 DEBUG "pod": "ds-idrepo-0" 06:15:30 DEBUG }, 06:15:30 DEBUG "value": [ 06:15:30 DEBUG 06:15:30 DEBUG 06:15:30 DEBUG 06:15:30 DEBUG [...] 06:15:30 DEBUG 06:15:30 DEBUG 06:15:30 DEBUG 06:15:30 DEBUG "pod": "ds-idrepo-1" 06:15:30 DEBUG }, 06:15:30 DEBUG "value": [ 06:15:30 DEBUG 1712726130, 06:15:30 DEBUG "0" 06:15:30 DEBUG ] 06:15:30 DEBUG } 06:15:30 DEBUG ] 06:15:30 DEBUG } 06:15:30 DEBUG } 06:15:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726131" 06:15:31 INFO [http_cmd]: http status code OK 06:15:31 DEBUG --- status code --- 06:15:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:31 DEBUG --- http response --- 06:15:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:15:31 DEBUG ----- output ----- 06:15:31 DEBUG { 06:15:31 DEBUG "status": "success", 06:15:31 DEBUG "data": { 06:15:31 DEBUG "resultType": "vector", 06:15:31 DEBUG "result": [ 06:15:31 DEBUG { 06:15:31 DEBUG "metric": { 06:15:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:15:31 DEBUG }, 06:15:31 DEBUG "value": [ 06:15:31 DEBUG 06:15:31 DEBUG 06:15:31 DEBUG 06:15:31 DEBUG [...] 06:15:31 DEBUG 06:15:31 DEBUG 06:15:31 DEBUG 06:15:31 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:15:31 DEBUG }, 06:15:31 DEBUG "value": [ 06:15:31 DEBUG 1712726131, 06:15:31 DEBUG "0" 06:15:31 DEBUG ] 06:15:31 DEBUG } 06:15:31 DEBUG ] 06:15:31 DEBUG } 06:15:31 DEBUG } 06:15:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712726131" 06:15:31 INFO [http_cmd]: http status code OK 06:15:31 DEBUG --- status code --- 06:15:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:31 DEBUG --- http response --- 06:15:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:15:31 DEBUG ----- output ----- 06:15:31 DEBUG { 06:15:31 DEBUG "status": "success", 06:15:31 DEBUG "data": { 06:15:31 DEBUG "resultType": "vector", 06:15:31 DEBUG "result": [ 06:15:31 DEBUG { 06:15:31 DEBUG "metric": { 06:15:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:15:31 DEBUG }, 06:15:31 DEBUG "value": [ 06:15:31 DEBUG 06:15:31 DEBUG 06:15:31 DEBUG 06:15:31 DEBUG [...] 06:15:31 DEBUG 06:15:31 DEBUG 06:15:31 DEBUG 06:15:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:15:31 DEBUG }, 06:15:31 DEBUG "value": [ 06:15:31 DEBUG 1712726131, 06:15:31 DEBUG "2488254464" 06:15:31 DEBUG ] 06:15:31 DEBUG } 06:15:31 DEBUG ] 06:15:31 DEBUG } 06:15:31 DEBUG } 06:15:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726131" 06:15:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726131" 06:15:31 INFO [http_cmd]: http status code OK 06:15:31 DEBUG --- status code --- 06:15:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:31 DEBUG --- http response --- 06:15:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:15:31 DEBUG ----- output ----- 06:15:31 DEBUG { 06:15:31 DEBUG "status": "success", 06:15:31 DEBUG "data": { 06:15:31 DEBUG "resultType": "vector", 06:15:31 DEBUG "result": [ 06:15:31 DEBUG { 06:15:31 DEBUG "metric": { 06:15:31 DEBUG "instance": "10.142.0.25:9100" 06:15:31 DEBUG }, 06:15:31 DEBUG "value": [ 06:15:31 DEBUG 06:15:31 DEBUG 06:15:31 DEBUG 06:15:31 DEBUG [...] 06:15:31 DEBUG 06:15:31 DEBUG 06:15:31 DEBUG 06:15:31 DEBUG "instance": "10.142.0.77:9100" 06:15:31 DEBUG }, 06:15:31 DEBUG "value": [ 06:15:31 DEBUG 1712726131, 06:15:31 DEBUG "0.047666666666667" 06:15:31 DEBUG ] 06:15:31 DEBUG } 06:15:31 DEBUG ] 06:15:31 DEBUG } 06:15:31 DEBUG } 06:15:31 INFO [http_cmd]: http status code OK 06:15:31 DEBUG --- status code --- 06:15:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:15:31 DEBUG --- http response --- 06:15:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:15:31 DEBUG ----- output ----- 06:15:31 DEBUG { 06:15:31 DEBUG "status": "success", 06:15:31 DEBUG "data": { 06:15:31 DEBUG "resultType": "vector", 06:15:31 DEBUG "result": [ 06:15:31 DEBUG { 06:15:31 DEBUG "metric": { 06:15:31 DEBUG "instance": "10.142.0.25:9100" 06:15:31 DEBUG }, 06:15:31 DEBUG "value": [ 06:15:31 DEBUG 06:15:31 DEBUG 06:15:31 DEBUG 06:15:31 DEBUG [...] 06:15:31 DEBUG 06:15:31 DEBUG 06:15:31 DEBUG 06:15:31 DEBUG "instance": "10.142.0.77:9100" 06:15:31 DEBUG }, 06:15:31 DEBUG "value": [ 06:15:31 DEBUG 1712726131, 06:15:31 DEBUG "42334261.733333334" 06:15:31 DEBUG ] 06:15:31 DEBUG } 06:15:31 DEBUG ] 06:15:31 DEBUG } 06:15:31 DEBUG } 06:15:49 INFO 06:15:49 INFO [loop_until]: kubectl --namespace=xlou top pods 06:15:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:15:49 INFO [loop_until]: OK (rc = 0) 06:15:49 DEBUG --- stdout --- 06:15:49 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10797m 9013Mi am-654c4fcbcf-9scrj 10054m 8446Mi am-654c4fcbcf-v5qb5 8446m 8445Mi ds-cts-0 8267m 13392Mi ds-cts-1 8414m 13392Mi ds-cts-2 8015m 13394Mi ds-idrepo-0 10712m 5623Mi ds-idrepo-1 407m 5672Mi ds-idrepo-2 372m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 8m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4341m 2273Mi 06:15:49 DEBUG --- stderr --- 06:15:49 DEBUG 06:15:56 INFO 06:15:56 INFO [loop_until]: kubectl --namespace=xlou top node 06:15:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:15:57 INFO [loop_until]: OK (rc = 0) 06:15:57 DEBUG --- stdout --- 06:15:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 95m 0% 2305Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9309m 58% 9316Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 82m 0% 3261Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13801m 86% 6677Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 11002m 69% 9327Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11960m 75% 9815Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 81m 0% 858Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10929m 68% 6301Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8505m 53% 14064Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 8145m 51% 14057Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 438m 2% 6384Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8489m 53% 14079Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 469m 2% 6352Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4424m 27% 5882Mi 10% 06:15:57 DEBUG --- stderr --- 06:15:57 DEBUG 06:16:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726187" 06:16:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726187" 06:16:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712726187" 06:16:27 ERROR [http_cmd]: ERROR 06:16:27 ERROR --- status code --- 06:16:27 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:16:27 ERROR --- http response --- 06:16:27 INFO [http_cmd]: http status code OK 06:16:27 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:16:27 DEBUG --- status code --- 06:16:27 INFO [http_cmd]: http status code OK 06:16:27 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:16:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:27 DEBUG --- status code --- 06:16:27 DEBUG --- http response --- 06:16:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:16:27 DEBUG --- http response --- 06:16:27 DEBUG ----- output ----- 06:16:27 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:16:27 DEBUG { 06:16:27 DEBUG "status": "success", 06:16:27 DEBUG "data": { 06:16:27 DEBUG "resultType": "vector", 06:16:27 DEBUG "result": [ 06:16:27 DEBUG { 06:16:27 DEBUG "metric": { 06:16:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:16:27 DEBUG }, 06:16:27 DEBUG "value": [ 06:16:27 DEBUG 06:16:27 DEBUG 06:16:27 DEBUG 06:16:27 DEBUG [...] 06:16:27 DEBUG 06:16:27 DEBUG 06:16:27 DEBUG 06:16:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:16:27 DEBUG }, 06:16:27 DEBUG "value": [ 06:16:27 DEBUG 1712726187, 06:16:27 DEBUG "0" 06:16:27 DEBUG ] 06:16:27 DEBUG } 06:16:27 DEBUG ] 06:16:27 DEBUG } 06:16:27 DEBUG } 06:16:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726188" 06:16:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726188" 06:16:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726188" 06:16:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726188" 06:16:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726188" 06:16:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726188" 06:16:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726188" 06:16:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726188" 06:16:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726188" 06:16:28 INFO [http_cmd]: http status code OK 06:16:28 DEBUG --- status code --- 06:16:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:28 DEBUG --- http response --- 06:16:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:16:28 DEBUG ----- output ----- 06:16:28 DEBUG { 06:16:28 DEBUG "status": "success", 06:16:28 DEBUG "data": { 06:16:28 DEBUG "resultType": "vector", 06:16:28 DEBUG "result": [ 06:16:28 INFO [http_cmd]: http status code OK 06:16:28 DEBUG { 06:16:28 DEBUG --- status code --- 06:16:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:28 DEBUG --- http response --- 06:16:28 INFO [http_cmd]: http status code OK 06:16:28 DEBUG "metric": { 06:16:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:16:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:16:28 DEBUG ----- output ----- 06:16:28 DEBUG { 06:16:28 INFO [http_cmd]: http status code OK 06:16:28 INFO [http_cmd]: http status code OK 06:16:28 DEBUG "status": "success", 06:16:28 DEBUG "data": { 06:16:28 DEBUG --- status code --- 06:16:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:28 DEBUG --- status code --- 06:16:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:28 INFO [http_cmd]: http status code OK 06:16:28 DEBUG "resultType": "vector", 06:16:28 DEBUG "result": [ 06:16:28 DEBUG { 06:16:28 DEBUG "metric": { 06:16:28 DEBUG "instance": "10.142.0.25:9100" 06:16:28 DEBUG }, 06:16:28 DEBUG --- status code --- 06:16:28 DEBUG "value": [ 06:16:28 DEBUG 06:16:28 DEBUG 06:16:28 DEBUG 06:16:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:28 DEBUG --- http response --- 06:16:28 DEBUG [...] 06:16:28 DEBUG 06:16:28 DEBUG --- http response --- 06:16:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 06:16:28 DEBUG ----- output ----- 06:16:28 DEBUG { 06:16:28 DEBUG "status": "success", 06:16:28 DEBUG "data": { 06:16:28 INFO [http_cmd]: http status code OK 06:16:28 INFO [http_cmd]: http status code OK 06:16:28 INFO [http_cmd]: http status code OK 06:16:28 DEBUG --- status code --- 06:16:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:16:28 DEBUG ----- output ----- 06:16:28 DEBUG { 06:16:28 DEBUG "status": "success", 06:16:28 DEBUG --- status code --- 06:16:28 DEBUG --- status code --- 06:16:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:28 DEBUG --- http response --- 06:16:28 DEBUG --- http response --- 06:16:28 DEBUG "data": { 06:16:28 DEBUG "resultType": "vector", 06:16:28 DEBUG "result": [ 06:16:28 DEBUG { 06:16:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:28 DEBUG --- http response --- 06:16:28 DEBUG --- status code --- 06:16:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:16:28 DEBUG ----- output ----- 06:16:28 DEBUG { 06:16:28 DEBUG "resultType": "vector", 06:16:28 DEBUG "result": [ 06:16:28 DEBUG { 06:16:28 DEBUG "metric": { 06:16:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:16:28 DEBUG ----- output ----- 06:16:28 DEBUG { 06:16:28 DEBUG "status": "success", 06:16:28 DEBUG "data": { 06:16:28 DEBUG --- http response --- 06:16:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:16:28 DEBUG }, 06:16:28 DEBUG "value": [ 06:16:28 DEBUG "resultType": "vector", 06:16:28 DEBUG "result": [ 06:16:28 DEBUG { 06:16:28 DEBUG "metric": { 06:16:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:16:28 DEBUG }, 06:16:28 DEBUG "value": [ 06:16:28 DEBUG 06:16:28 DEBUG 06:16:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:16:28 DEBUG 06:16:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:28 DEBUG "metric": { 06:16:28 DEBUG "pod": "ds-idrepo-0" 06:16:28 DEBUG }, 06:16:28 DEBUG "value": [ 06:16:28 DEBUG 06:16:28 DEBUG 06:16:28 DEBUG [...] 06:16:28 DEBUG 06:16:28 DEBUG 06:16:28 DEBUG 06:16:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:16:28 DEBUG }, 06:16:28 DEBUG "value": [ 06:16:28 DEBUG 1712726188, 06:16:28 DEBUG "258695.5" 06:16:28 DEBUG ] 06:16:28 DEBUG } 06:16:28 DEBUG ] 06:16:28 DEBUG } 06:16:28 DEBUG } 06:16:28 DEBUG 06:16:28 DEBUG --- http response --- 06:16:28 DEBUG "status": "success", 06:16:28 DEBUG "data": { 06:16:28 DEBUG ----- output ----- 06:16:28 DEBUG { 06:16:28 DEBUG "status": "success", 06:16:28 DEBUG "data": { 06:16:28 DEBUG "resultType": "vector", 06:16:28 DEBUG "result": [ 06:16:28 DEBUG { 06:16:28 DEBUG "metric": { 06:16:28 DEBUG "pod": "ds-idrepo-0" 06:16:28 DEBUG }, 06:16:28 DEBUG "value": [ 06:16:28 DEBUG 06:16:28 DEBUG 06:16:28 DEBUG 06:16:28 DEBUG [...] 06:16:28 DEBUG 06:16:28 DEBUG 06:16:28 DEBUG 06:16:28 DEBUG "pod": "ds-idrepo-1" 06:16:28 DEBUG }, 06:16:28 DEBUG "value": [ 06:16:28 DEBUG 1712726188, 06:16:28 DEBUG "0" 06:16:28 DEBUG ] 06:16:28 DEBUG } 06:16:28 DEBUG ] 06:16:28 DEBUG } 06:16:28 DEBUG } 06:16:28 DEBUG 06:16:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:16:28 DEBUG ----- output ----- 06:16:28 DEBUG { 06:16:28 DEBUG "status": "success", 06:16:28 DEBUG "data": { 06:16:28 DEBUG [...] 06:16:28 DEBUG }, 06:16:28 DEBUG "resultType": "vector", 06:16:28 DEBUG "result": [ 06:16:28 DEBUG { 06:16:28 DEBUG "metric": { 06:16:28 DEBUG 06:16:28 DEBUG 06:16:28 DEBUG 06:16:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:16:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:16:28 DEBUG "resultType": "vector", 06:16:28 DEBUG 06:16:28 DEBUG 06:16:28 DEBUG 06:16:28 DEBUG "value": [ 06:16:28 DEBUG }, 06:16:28 DEBUG ----- output ----- 06:16:28 DEBUG "result": [ 06:16:28 DEBUG 06:16:28 DEBUG "instance": "10.142.0.77:9100" 06:16:28 DEBUG }, 06:16:28 DEBUG 06:16:28 DEBUG 06:16:28 DEBUG { 06:16:28 DEBUG { 06:16:28 DEBUG [...] 06:16:28 DEBUG 06:16:28 DEBUG 06:16:28 DEBUG 06:16:28 DEBUG [...] 06:16:28 DEBUG 06:16:28 DEBUG "metric": { 06:16:28 DEBUG "pod": "ds-idrepo-0" 06:16:28 DEBUG "pod": "ds-cts-2" 06:16:28 DEBUG }, 06:16:28 DEBUG 06:16:28 DEBUG "status": "success", 06:16:28 DEBUG "value": [ 06:16:28 DEBUG 06:16:28 DEBUG }, 06:16:28 DEBUG "value": [ 06:16:28 DEBUG 1712726188, 06:16:28 DEBUG 06:16:28 DEBUG "data": { 06:16:28 DEBUG "resultType": "vector", 06:16:28 DEBUG "result": [ 06:16:28 DEBUG { 06:16:28 DEBUG "metric": { 06:16:28 DEBUG "instance": "10.142.0.25:9100" 06:16:28 DEBUG }, 06:16:28 DEBUG "value": [ 06:16:28 DEBUG 06:16:28 DEBUG 06:16:28 DEBUG 06:16:28 DEBUG [...] 06:16:28 DEBUG 06:16:28 DEBUG 06:16:28 DEBUG 06:16:28 DEBUG "instance": "10.142.0.77:9100" 06:16:28 DEBUG }, 06:16:28 DEBUG "value": [ 06:16:28 DEBUG 1712726188, 06:16:28 DEBUG "42473376.233333334" 06:16:28 DEBUG ] 06:16:28 DEBUG } 06:16:28 DEBUG ] 06:16:28 DEBUG } 06:16:28 DEBUG } 06:16:28 DEBUG 06:16:28 DEBUG 06:16:28 DEBUG 06:16:28 DEBUG [...] 06:16:28 DEBUG 06:16:28 DEBUG 06:16:28 DEBUG 06:16:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:16:28 DEBUG }, 06:16:28 DEBUG "value": [ 06:16:28 DEBUG 1712726188, 06:16:28 DEBUG "1441.8666666666666" 06:16:28 DEBUG ] 06:16:28 DEBUG } 06:16:28 DEBUG ] 06:16:28 DEBUG } 06:16:28 DEBUG } 06:16:28 DEBUG "value": [ 06:16:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:16:28 DEBUG "value": [ 06:16:28 DEBUG "30" 06:16:28 DEBUG 06:16:28 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:16:28 DEBUG }, 06:16:28 DEBUG 1712726188, 06:16:28 DEBUG ] 06:16:28 DEBUG 06:16:28 DEBUG }, 06:16:28 DEBUG "value": [ 06:16:28 DEBUG "373601250304" 06:16:28 DEBUG } 06:16:28 DEBUG 06:16:28 DEBUG "value": [ 06:16:28 DEBUG 1712726188, 06:16:28 DEBUG ] 06:16:28 DEBUG ] 06:16:28 DEBUG [...] 06:16:28 DEBUG 1712726188, 06:16:28 DEBUG "0" 06:16:28 DEBUG } 06:16:28 DEBUG } 06:16:28 DEBUG 06:16:28 DEBUG ] 06:16:28 DEBUG } 06:16:28 DEBUG ] 06:16:28 DEBUG } 06:16:28 DEBUG } 06:16:28 DEBUG ] 06:16:28 DEBUG } 06:16:28 DEBUG } 06:16:28 DEBUG 06:16:28 DEBUG 06:16:28 DEBUG "pod": "ds-cts-2" 06:16:28 DEBUG }, 06:16:28 DEBUG "value": [ 06:16:28 DEBUG 1712726188, 06:16:28 DEBUG "0" 06:16:28 DEBUG ] 06:16:28 DEBUG } 06:16:28 DEBUG ] 06:16:28 DEBUG } 06:16:28 DEBUG } 06:16:28 DEBUG } 06:16:28 DEBUG "1441.2333333333333" 06:16:28 DEBUG ] 06:16:28 DEBUG } 06:16:28 DEBUG ] 06:16:28 DEBUG } 06:16:28 DEBUG } 06:16:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726188" 06:16:28 INFO [http_cmd]: http status code OK 06:16:28 DEBUG --- status code --- 06:16:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:28 DEBUG --- http response --- 06:16:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:16:28 DEBUG ----- output ----- 06:16:28 DEBUG { 06:16:28 DEBUG "status": "success", 06:16:28 DEBUG "data": { 06:16:28 DEBUG "resultType": "vector", 06:16:28 DEBUG "result": [ 06:16:28 DEBUG { 06:16:28 DEBUG "metric": { 06:16:28 DEBUG "instance": "10.142.0.25:9100" 06:16:28 DEBUG }, 06:16:28 DEBUG "value": [ 06:16:28 DEBUG 06:16:28 DEBUG 06:16:28 DEBUG 06:16:28 DEBUG [...] 06:16:28 DEBUG 06:16:28 DEBUG 06:16:28 DEBUG 06:16:28 DEBUG "instance": "10.142.0.77:9100" 06:16:28 DEBUG }, 06:16:28 DEBUG "value": [ 06:16:28 DEBUG 1712726188, 06:16:28 DEBUG "0.11610000000000073" 06:16:28 DEBUG ] 06:16:28 DEBUG } 06:16:28 DEBUG ] 06:16:28 DEBUG } 06:16:28 DEBUG } 06:16:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726189" 06:16:29 INFO [http_cmd]: http status code OK 06:16:29 DEBUG --- status code --- 06:16:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:29 DEBUG --- http response --- 06:16:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:16:29 DEBUG ----- output ----- 06:16:29 DEBUG { 06:16:29 DEBUG "status": "success", 06:16:29 DEBUG "data": { 06:16:29 DEBUG "resultType": "vector", 06:16:29 DEBUG "result": [ 06:16:29 DEBUG { 06:16:29 DEBUG "metric": { 06:16:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:16:29 DEBUG }, 06:16:29 DEBUG "value": [ 06:16:29 DEBUG 06:16:29 DEBUG 06:16:29 DEBUG 06:16:29 DEBUG [...] 06:16:29 DEBUG 06:16:29 DEBUG 06:16:29 DEBUG 06:16:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:16:29 DEBUG }, 06:16:29 DEBUG "value": [ 06:16:29 DEBUG 1712726189, 06:16:29 DEBUG "48039.37541755266" 06:16:29 DEBUG ] 06:16:29 DEBUG } 06:16:29 DEBUG ] 06:16:29 DEBUG } 06:16:29 DEBUG } 06:16:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712726189" 06:16:29 INFO [http_cmd]: http status code OK 06:16:29 DEBUG --- status code --- 06:16:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:29 DEBUG --- http response --- 06:16:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712726189" 06:16:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:16:29 DEBUG ----- output ----- 06:16:29 DEBUG { 06:16:29 DEBUG "status": "success", 06:16:29 DEBUG "data": { 06:16:29 DEBUG "resultType": "vector", 06:16:29 DEBUG "result": [ 06:16:29 DEBUG { 06:16:29 DEBUG "metric": { 06:16:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:16:29 DEBUG }, 06:16:29 DEBUG "value": [ 06:16:29 DEBUG 06:16:29 DEBUG 06:16:29 DEBUG 06:16:29 DEBUG [...] 06:16:29 DEBUG 06:16:29 DEBUG 06:16:29 DEBUG 06:16:29 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:16:29 DEBUG }, 06:16:29 DEBUG "value": [ 06:16:29 DEBUG 1712726189, 06:16:29 DEBUG "4.2412663441771645" 06:16:29 DEBUG ] 06:16:29 DEBUG } 06:16:29 DEBUG ] 06:16:29 DEBUG } 06:16:29 DEBUG } 06:16:29 INFO [http_cmd]: http status code OK 06:16:29 DEBUG --- status code --- 06:16:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:29 DEBUG --- http response --- 06:16:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:16:29 DEBUG ----- output ----- 06:16:29 DEBUG { 06:16:29 DEBUG "status": "success", 06:16:29 DEBUG "data": { 06:16:29 DEBUG "resultType": "vector", 06:16:29 DEBUG "result": [ 06:16:29 DEBUG { 06:16:29 DEBUG "metric": { 06:16:29 DEBUG "pod": "ds-idrepo-0" 06:16:29 DEBUG }, 06:16:29 DEBUG "value": [ 06:16:29 DEBUG 06:16:29 DEBUG 06:16:29 DEBUG 06:16:29 DEBUG [...] 06:16:29 DEBUG 06:16:29 DEBUG 06:16:29 DEBUG 06:16:29 DEBUG "pod": "ds-cts-2" 06:16:29 DEBUG }, 06:16:29 DEBUG "value": [ 06:16:29 DEBUG 1712726189, 06:16:29 DEBUG "0" 06:16:29 DEBUG ] 06:16:29 DEBUG } 06:16:29 DEBUG ] 06:16:29 DEBUG } 06:16:29 DEBUG } 06:16:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726190" 06:16:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726190" 06:16:30 INFO [http_cmd]: http status code OK 06:16:30 DEBUG --- status code --- 06:16:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:30 DEBUG --- http response --- 06:16:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:16:30 DEBUG ----- output ----- 06:16:30 DEBUG { 06:16:30 DEBUG "status": "success", 06:16:30 DEBUG "data": { 06:16:30 DEBUG "resultType": "vector", 06:16:30 DEBUG "result": [ 06:16:30 DEBUG { 06:16:30 DEBUG "metric": { 06:16:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:16:30 DEBUG }, 06:16:30 DEBUG "value": [ 06:16:30 DEBUG 06:16:30 DEBUG 06:16:30 DEBUG 06:16:30 DEBUG [...] 06:16:30 DEBUG 06:16:30 DEBUG 06:16:30 DEBUG 06:16:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:16:30 DEBUG }, 06:16:30 DEBUG "value": [ 06:16:30 DEBUG 1712726190, 06:16:30 DEBUG "5.211359549481391" 06:16:30 DEBUG ] 06:16:30 DEBUG } 06:16:30 DEBUG ] 06:16:30 DEBUG } 06:16:30 DEBUG } 06:16:30 INFO [http_cmd]: http status code OK 06:16:30 DEBUG --- status code --- 06:16:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:30 DEBUG --- http response --- 06:16:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:16:30 DEBUG ----- output ----- 06:16:30 DEBUG { 06:16:30 DEBUG "status": "success", 06:16:30 DEBUG "data": { 06:16:30 DEBUG "resultType": "vector", 06:16:30 DEBUG "result": [ 06:16:30 DEBUG { 06:16:30 DEBUG "metric": { 06:16:30 DEBUG "pod": "ds-idrepo-2" 06:16:30 DEBUG }, 06:16:30 DEBUG "value": [ 06:16:30 DEBUG 06:16:30 DEBUG 06:16:30 DEBUG 06:16:30 DEBUG [...] 06:16:30 DEBUG 06:16:30 DEBUG 06:16:30 DEBUG 06:16:30 DEBUG "pod": "ds-cts-2" 06:16:30 DEBUG }, 06:16:30 DEBUG "value": [ 06:16:30 DEBUG 1712726190, 06:16:30 DEBUG "0.0001415422222222222" 06:16:30 DEBUG ] 06:16:30 DEBUG } 06:16:30 DEBUG ] 06:16:30 DEBUG } 06:16:30 DEBUG } 06:16:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726190" 06:16:30 INFO [http_cmd]: http status code OK 06:16:30 DEBUG --- status code --- 06:16:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:30 DEBUG --- http response --- 06:16:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:16:30 DEBUG ----- output ----- 06:16:30 DEBUG { 06:16:30 DEBUG "status": "success", 06:16:30 DEBUG "data": { 06:16:30 DEBUG "resultType": "vector", 06:16:30 DEBUG "result": [ 06:16:30 DEBUG { 06:16:30 DEBUG "metric": { 06:16:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:16:30 DEBUG }, 06:16:30 DEBUG "value": [ 06:16:30 DEBUG 06:16:30 DEBUG 06:16:30 DEBUG 06:16:30 DEBUG [...] 06:16:30 DEBUG 06:16:30 DEBUG 06:16:30 DEBUG 06:16:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:16:30 DEBUG }, 06:16:30 DEBUG "value": [ 06:16:30 DEBUG 1712726190, 06:16:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712726190" 06:16:30 DEBUG "1441.8333333333333" 06:16:30 DEBUG ] 06:16:30 DEBUG } 06:16:30 DEBUG ] 06:16:30 DEBUG } 06:16:30 DEBUG } 06:16:30 INFO [http_cmd]: http status code OK 06:16:30 DEBUG --- status code --- 06:16:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:30 DEBUG --- http response --- 06:16:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:16:30 DEBUG ----- output ----- 06:16:30 DEBUG { 06:16:30 DEBUG "status": "success", 06:16:30 DEBUG "data": { 06:16:30 DEBUG "resultType": "vector", 06:16:30 DEBUG "result": [ 06:16:30 DEBUG { 06:16:30 DEBUG "metric": { 06:16:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:16:30 DEBUG }, 06:16:30 DEBUG "value": [ 06:16:30 DEBUG 06:16:30 DEBUG 06:16:30 DEBUG 06:16:30 DEBUG [...] 06:16:30 DEBUG 06:16:30 DEBUG 06:16:30 DEBUG 06:16:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:16:30 DEBUG }, 06:16:30 DEBUG "value": [ 06:16:30 DEBUG 1712726190, 06:16:30 DEBUG "0.05" 06:16:30 DEBUG ] 06:16:30 DEBUG } 06:16:30 DEBUG ] 06:16:30 DEBUG } 06:16:30 DEBUG } 06:16:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726190" 06:16:30 INFO [http_cmd]: http status code OK 06:16:30 DEBUG --- status code --- 06:16:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:30 DEBUG --- http response --- 06:16:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:16:30 DEBUG ----- output ----- 06:16:30 DEBUG { 06:16:30 DEBUG "status": "success", 06:16:30 DEBUG "data": { 06:16:30 DEBUG "resultType": "vector", 06:16:30 DEBUG "result": [ 06:16:30 DEBUG { 06:16:30 DEBUG "metric": { 06:16:30 DEBUG "pod": "ds-idrepo-0" 06:16:30 DEBUG }, 06:16:30 DEBUG "value": [ 06:16:30 DEBUG 06:16:30 DEBUG 06:16:30 DEBUG 06:16:30 DEBUG [...] 06:16:30 DEBUG 06:16:30 DEBUG 06:16:30 DEBUG 06:16:30 DEBUG "pod": "ds-idrepo-1" 06:16:30 DEBUG }, 06:16:30 DEBUG "value": [ 06:16:30 DEBUG 1712726190, 06:16:30 DEBUG "0" 06:16:30 DEBUG ] 06:16:30 DEBUG } 06:16:30 DEBUG ] 06:16:30 DEBUG } 06:16:30 DEBUG } 06:16:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726190" 06:16:30 INFO [http_cmd]: http status code OK 06:16:30 DEBUG --- status code --- 06:16:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:30 DEBUG --- http response --- 06:16:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:16:30 DEBUG ----- output ----- 06:16:30 DEBUG { 06:16:30 DEBUG "status": "success", 06:16:30 DEBUG "data": { 06:16:30 DEBUG "resultType": "vector", 06:16:30 DEBUG "result": [ 06:16:30 DEBUG { 06:16:30 DEBUG "metric": { 06:16:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:16:30 DEBUG }, 06:16:30 DEBUG "value": [ 06:16:30 DEBUG 06:16:30 DEBUG 06:16:30 DEBUG 06:16:30 DEBUG [...] 06:16:30 DEBUG 06:16:30 DEBUG 06:16:30 DEBUG 06:16:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:16:30 DEBUG }, 06:16:30 DEBUG "value": [ 06:16:30 DEBUG 1712726190, 06:16:30 DEBUG "1436.2" 06:16:30 DEBUG ] 06:16:30 DEBUG } 06:16:30 DEBUG ] 06:16:30 DEBUG } 06:16:30 DEBUG } 06:16:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726190" 06:16:30 INFO [http_cmd]: http status code OK 06:16:30 DEBUG --- status code --- 06:16:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:30 DEBUG --- http response --- 06:16:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:16:30 DEBUG ----- output ----- 06:16:30 DEBUG { 06:16:30 DEBUG "status": "success", 06:16:30 DEBUG "data": { 06:16:30 DEBUG "resultType": "vector", 06:16:30 DEBUG "result": [ 06:16:30 DEBUG { 06:16:30 DEBUG "metric": { 06:16:30 DEBUG "pod": "ds-idrepo-0" 06:16:30 DEBUG }, 06:16:30 DEBUG "value": [ 06:16:30 DEBUG 06:16:30 DEBUG 06:16:30 DEBUG 06:16:30 DEBUG [...] 06:16:30 DEBUG 06:16:30 DEBUG 06:16:30 DEBUG 06:16:30 DEBUG "pod": "ds-idrepo-1" 06:16:30 DEBUG }, 06:16:30 DEBUG "value": [ 06:16:30 DEBUG 1712726190, 06:16:30 DEBUG "0" 06:16:30 DEBUG ] 06:16:30 DEBUG } 06:16:30 DEBUG ] 06:16:30 DEBUG } 06:16:30 DEBUG } 06:16:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726191" 06:16:31 INFO [http_cmd]: http status code OK 06:16:31 DEBUG --- status code --- 06:16:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:31 DEBUG --- http response --- 06:16:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:16:31 DEBUG ----- output ----- 06:16:31 DEBUG { 06:16:31 DEBUG "status": "success", 06:16:31 DEBUG "data": { 06:16:31 DEBUG "resultType": "vector", 06:16:31 DEBUG "result": [ 06:16:31 DEBUG { 06:16:31 DEBUG "metric": { 06:16:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:16:31 DEBUG }, 06:16:31 DEBUG "value": [ 06:16:31 DEBUG 06:16:31 DEBUG 06:16:31 DEBUG 06:16:31 DEBUG [...] 06:16:31 DEBUG 06:16:31 DEBUG 06:16:31 DEBUG 06:16:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:16:31 DEBUG }, 06:16:31 DEBUG "value": [ 06:16:31 DEBUG 1712726191, 06:16:31 DEBUG "0" 06:16:31 DEBUG ] 06:16:31 DEBUG } 06:16:31 DEBUG ] 06:16:31 DEBUG } 06:16:31 DEBUG } 06:16:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712726191" 06:16:31 INFO [http_cmd]: http status code OK 06:16:31 DEBUG --- status code --- 06:16:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:31 DEBUG --- http response --- 06:16:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:16:31 DEBUG ----- output ----- 06:16:31 DEBUG { 06:16:31 DEBUG "status": "success", 06:16:31 DEBUG "data": { 06:16:31 DEBUG "resultType": "vector", 06:16:31 DEBUG "result": [ 06:16:31 DEBUG { 06:16:31 DEBUG "metric": { 06:16:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:16:31 DEBUG }, 06:16:31 DEBUG "value": [ 06:16:31 DEBUG 06:16:31 DEBUG 06:16:31 DEBUG 06:16:31 DEBUG [...] 06:16:31 DEBUG 06:16:31 DEBUG 06:16:31 DEBUG 06:16:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:16:31 DEBUG }, 06:16:31 DEBUG "value": [ 06:16:31 DEBUG 1712726191, 06:16:31 DEBUG "2487906304" 06:16:31 DEBUG ] 06:16:31 DEBUG } 06:16:31 DEBUG ] 06:16:31 DEBUG } 06:16:31 DEBUG } 06:16:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726191" 06:16:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726191" 06:16:31 INFO [http_cmd]: http status code OK 06:16:31 DEBUG --- status code --- 06:16:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:31 DEBUG --- http response --- 06:16:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:16:31 INFO [http_cmd]: http status code OK 06:16:31 DEBUG ----- output ----- 06:16:31 DEBUG --- status code --- 06:16:31 DEBUG { 06:16:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:16:31 DEBUG "status": "success", 06:16:31 DEBUG --- http response --- 06:16:31 DEBUG "data": { 06:16:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:16:31 DEBUG "resultType": "vector", 06:16:31 DEBUG ----- output ----- 06:16:31 DEBUG { 06:16:31 DEBUG "status": "success", 06:16:31 DEBUG "data": { 06:16:31 DEBUG "resultType": "vector", 06:16:31 DEBUG "result": [ 06:16:31 DEBUG { 06:16:31 DEBUG "metric": { 06:16:31 DEBUG "instance": "10.142.0.25:9100" 06:16:31 DEBUG }, 06:16:31 DEBUG "value": [ 06:16:31 DEBUG 06:16:31 DEBUG 06:16:31 DEBUG 06:16:31 DEBUG [...] 06:16:31 DEBUG 06:16:31 DEBUG 06:16:31 DEBUG 06:16:31 DEBUG "instance": "10.142.0.77:9100" 06:16:31 DEBUG }, 06:16:31 DEBUG "value": [ 06:16:31 DEBUG 1712726191, 06:16:31 DEBUG "42475168.3" 06:16:31 DEBUG ] 06:16:31 DEBUG } 06:16:31 DEBUG ] 06:16:31 DEBUG } 06:16:31 DEBUG } 06:16:31 DEBUG "result": [ 06:16:31 DEBUG { 06:16:31 DEBUG "metric": { 06:16:31 DEBUG "instance": "10.142.0.25:9100" 06:16:31 DEBUG }, 06:16:31 DEBUG "value": [ 06:16:31 DEBUG 06:16:31 DEBUG 06:16:31 DEBUG 06:16:31 DEBUG [...] 06:16:31 DEBUG 06:16:31 DEBUG 06:16:31 DEBUG 06:16:31 DEBUG "instance": "10.142.0.77:9100" 06:16:31 DEBUG }, 06:16:31 DEBUG "value": [ 06:16:31 DEBUG 1712726191, 06:16:31 DEBUG "0.033000000000000425" 06:16:31 DEBUG ] 06:16:31 DEBUG } 06:16:31 DEBUG ] 06:16:31 DEBUG } 06:16:31 DEBUG } 06:16:49 INFO 06:16:49 INFO [loop_until]: kubectl --namespace=xlou top pods 06:16:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:16:49 INFO [loop_until]: OK (rc = 0) 06:16:49 DEBUG --- stdout --- 06:16:49 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10882m 8978Mi am-654c4fcbcf-9scrj 10019m 8447Mi am-654c4fcbcf-v5qb5 8469m 8445Mi ds-cts-0 8472m 13389Mi ds-cts-1 8231m 12785Mi ds-cts-2 7973m 13021Mi ds-idrepo-0 11063m 5623Mi ds-idrepo-1 372m 5672Mi ds-idrepo-2 389m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 6m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4319m 2273Mi 06:16:49 DEBUG --- stderr --- 06:16:49 DEBUG 06:16:57 INFO 06:16:57 INFO [loop_until]: kubectl --namespace=xlou top node 06:16:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:16:57 INFO [loop_until]: OK (rc = 0) 06:16:57 DEBUG --- stdout --- 06:16:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 97m 0% 2295Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9376m 59% 9315Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 79m 0% 3259Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13743m 86% 6688Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10973m 69% 9330Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 12048m 75% 9802Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 80m 0% 854Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10869m 68% 6300Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8360m 52% 14080Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 7782m 48% 13654Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 447m 2% 6385Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8488m 53% 13540Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 432m 2% 6351Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4340m 27% 5883Mi 10% 06:16:57 DEBUG --- stderr --- 06:16:57 DEBUG 06:17:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712726247" 06:17:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726247" 06:17:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726247" 06:17:27 ERROR [http_cmd]: ERROR 06:17:27 ERROR --- status code --- 06:17:27 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:17:27 ERROR --- http response --- 06:17:27 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:17:27 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:17:27 INFO [http_cmd]: http status code OK 06:17:27 DEBUG --- status code --- 06:17:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:27 DEBUG --- http response --- 06:17:27 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:17:27 INFO [http_cmd]: http status code OK 06:17:27 DEBUG --- status code --- 06:17:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:27 DEBUG --- http response --- 06:17:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:17:27 DEBUG ----- output ----- 06:17:27 DEBUG { 06:17:27 DEBUG "status": "success", 06:17:27 DEBUG "data": { 06:17:27 DEBUG "resultType": "vector", 06:17:27 DEBUG "result": [ 06:17:27 DEBUG { 06:17:27 DEBUG "metric": { 06:17:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:17:27 DEBUG }, 06:17:27 DEBUG "value": [ 06:17:27 DEBUG 06:17:27 DEBUG 06:17:27 DEBUG 06:17:27 DEBUG [...] 06:17:27 DEBUG 06:17:27 DEBUG 06:17:27 DEBUG 06:17:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:17:27 DEBUG }, 06:17:27 DEBUG "value": [ 06:17:27 DEBUG 1712726247, 06:17:27 DEBUG "0" 06:17:27 DEBUG ] 06:17:27 DEBUG } 06:17:27 DEBUG ] 06:17:27 DEBUG } 06:17:27 DEBUG } 06:17:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726248" 06:17:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726248" 06:17:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726248" 06:17:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726248" 06:17:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726248" 06:17:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726248" 06:17:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726248" 06:17:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726248" 06:17:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726248" 06:17:28 INFO [http_cmd]: http status code OK 06:17:28 DEBUG --- status code --- 06:17:28 INFO [http_cmd]: http status code OK 06:17:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:28 DEBUG --- status code --- 06:17:28 DEBUG --- http response --- 06:17:28 INFO [http_cmd]: http status code OK 06:17:28 DEBUG --- status code --- 06:17:28 INFO [http_cmd]: http status code OK 06:17:28 INFO [http_cmd]: http status code OK 06:17:28 INFO [http_cmd]: http status code OK 06:17:28 INFO [http_cmd]: http status code OK 06:17:28 DEBUG --- status code --- 06:17:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:28 DEBUG --- http response --- 06:17:28 INFO [http_cmd]: http status code OK 06:17:28 DEBUG --- status code --- 06:17:28 DEBUG --- status code --- 06:17:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:17:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:28 DEBUG --- http response --- 06:17:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:17:28 DEBUG ----- output ----- 06:17:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:28 DEBUG --- status code --- 06:17:28 DEBUG --- http response --- 06:17:28 DEBUG ----- output ----- 06:17:28 DEBUG --- http response --- 06:17:28 DEBUG --- status code --- 06:17:28 DEBUG { 06:17:28 INFO [http_cmd]: http status code OK 06:17:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:17:28 DEBUG --- http response --- 06:17:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:28 DEBUG --- http response --- 06:17:28 DEBUG { 06:17:28 DEBUG "status": "success", 06:17:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:17:28 DEBUG "status": "success", 06:17:28 DEBUG --- status code --- 06:17:28 DEBUG ----- output ----- 06:17:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:17:28 DEBUG ----- output ----- 06:17:28 DEBUG { 06:17:28 DEBUG "status": "success", 06:17:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:28 DEBUG ----- output ----- 06:17:28 DEBUG "data": { 06:17:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:28 DEBUG { 06:17:28 DEBUG "data": { 06:17:28 DEBUG --- http response --- 06:17:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:17:28 DEBUG ----- output ----- 06:17:28 DEBUG { 06:17:28 DEBUG "status": "success", 06:17:28 DEBUG "data": { 06:17:28 DEBUG --- http response --- 06:17:28 DEBUG "status": "success", 06:17:28 DEBUG "resultType": "vector", 06:17:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:17:28 DEBUG { 06:17:28 DEBUG "status": "success", 06:17:28 DEBUG "data": { 06:17:28 DEBUG "data": { 06:17:28 DEBUG "resultType": "vector", 06:17:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:17:28 DEBUG "data": { 06:17:28 DEBUG "result": [ 06:17:28 DEBUG ----- output ----- 06:17:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:17:28 DEBUG ----- output ----- 06:17:28 DEBUG { 06:17:28 DEBUG "status": "success", 06:17:28 DEBUG "result": [ 06:17:28 DEBUG ----- output ----- 06:17:28 DEBUG "resultType": "vector", 06:17:28 DEBUG { 06:17:28 DEBUG { 06:17:28 DEBUG "resultType": "vector", 06:17:28 DEBUG "resultType": "vector", 06:17:28 DEBUG "result": [ 06:17:28 DEBUG { 06:17:28 DEBUG { 06:17:28 DEBUG { 06:17:28 DEBUG "result": [ 06:17:28 DEBUG "metric": { 06:17:28 DEBUG "status": "success", 06:17:28 DEBUG "result": [ 06:17:28 DEBUG { 06:17:28 DEBUG "metric": { 06:17:28 DEBUG "instance": "10.142.0.25:9100" 06:17:28 DEBUG "metric": { 06:17:28 DEBUG "status": "success", 06:17:28 DEBUG { 06:17:28 DEBUG "metric": { 06:17:28 DEBUG "data": { 06:17:28 DEBUG }, 06:17:28 DEBUG "value": [ 06:17:28 DEBUG 06:17:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:17:28 DEBUG "resultType": "vector", 06:17:28 DEBUG "data": { 06:17:28 DEBUG "pod": "ds-idrepo-0" 06:17:28 DEBUG }, 06:17:28 DEBUG "resultType": "vector", 06:17:28 DEBUG "data": { 06:17:28 DEBUG 06:17:28 DEBUG "metric": { 06:17:28 DEBUG }, 06:17:28 DEBUG "result": [ 06:17:28 DEBUG "resultType": "vector", 06:17:28 DEBUG "result": [ 06:17:28 DEBUG "value": [ 06:17:28 DEBUG "result": [ 06:17:28 DEBUG "resultType": "vector", 06:17:28 DEBUG 06:17:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:17:28 DEBUG "value": [ 06:17:28 DEBUG { 06:17:28 DEBUG "metric": { 06:17:28 DEBUG "pod": "ds-idrepo-0" 06:17:28 DEBUG 06:17:28 DEBUG { 06:17:28 DEBUG "result": [ 06:17:28 DEBUG [...] 06:17:28 DEBUG }, 06:17:28 DEBUG 06:17:28 DEBUG { 06:17:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:17:28 DEBUG }, 06:17:28 DEBUG "value": [ 06:17:28 DEBUG 06:17:28 DEBUG { 06:17:28 DEBUG 06:17:28 DEBUG "value": [ 06:17:28 DEBUG 06:17:28 DEBUG "metric": { 06:17:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:17:28 DEBUG "metric": { 06:17:28 DEBUG }, 06:17:28 DEBUG "value": [ 06:17:28 DEBUG "metric": { 06:17:28 DEBUG 06:17:28 DEBUG 06:17:28 DEBUG 06:17:28 DEBUG 06:17:28 DEBUG [...] 06:17:28 DEBUG "pod": "ds-idrepo-0" 06:17:28 DEBUG 06:17:28 DEBUG 06:17:28 DEBUG 06:17:28 DEBUG 06:17:28 DEBUG "instance": "10.142.0.77:9100" 06:17:28 DEBUG 06:17:28 DEBUG 06:17:28 DEBUG 06:17:28 DEBUG }, 06:17:28 DEBUG 06:17:28 DEBUG [...] 06:17:28 DEBUG 06:17:28 DEBUG 06:17:28 DEBUG }, 06:17:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:17:28 DEBUG }, 06:17:28 DEBUG "value": [ 06:17:28 DEBUG "value": [ 06:17:28 DEBUG 06:17:28 DEBUG 06:17:28 DEBUG "instance": "10.142.0.25:9100" 06:17:28 DEBUG }, 06:17:28 DEBUG "value": [ 06:17:28 DEBUG }, 06:17:28 DEBUG "value": [ 06:17:28 DEBUG 06:17:28 DEBUG 1712726248, 06:17:28 DEBUG "1448.5333333333333" 06:17:28 DEBUG 06:17:28 DEBUG "value": [ 06:17:28 DEBUG 06:17:28 DEBUG 1712726248, 06:17:28 DEBUG 06:17:28 DEBUG 06:17:28 DEBUG 06:17:28 DEBUG [...] 06:17:28 DEBUG 06:17:28 DEBUG [...] 06:17:28 DEBUG 06:17:28 DEBUG 06:17:28 DEBUG 06:17:28 DEBUG [...] 06:17:28 DEBUG 06:17:28 DEBUG 06:17:28 DEBUG 06:17:28 DEBUG "instance": "10.142.0.77:9100" 06:17:28 DEBUG 06:17:28 DEBUG [...] 06:17:28 DEBUG "42692469" 06:17:28 DEBUG ] 06:17:28 DEBUG 06:17:28 DEBUG 06:17:28 DEBUG [...] 06:17:28 DEBUG 06:17:28 DEBUG 06:17:28 DEBUG 06:17:28 DEBUG 06:17:28 DEBUG ] 06:17:28 DEBUG } 06:17:28 DEBUG 06:17:28 DEBUG "pod": "ds-cts-2" 06:17:28 DEBUG }, 06:17:28 DEBUG }, 06:17:28 DEBUG 06:17:28 DEBUG 06:17:28 DEBUG "pod": "ds-cts-2" 06:17:28 DEBUG ] 06:17:28 DEBUG } 06:17:28 DEBUG [...] 06:17:28 DEBUG 06:17:28 DEBUG "value": [ 06:17:28 DEBUG 1712726248, 06:17:28 DEBUG "29.7" 06:17:28 DEBUG }, 06:17:28 DEBUG "value": [ 06:17:28 DEBUG } 06:17:28 DEBUG } 06:17:28 DEBUG 06:17:28 DEBUG 06:17:28 DEBUG "value": [ 06:17:28 DEBUG 06:17:28 DEBUG ] 06:17:28 DEBUG 1712726248, 06:17:28 DEBUG 06:17:28 DEBUG 06:17:28 DEBUG 06:17:28 DEBUG "pod": "ds-idrepo-1" 06:17:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726248" 06:17:28 DEBUG 1712726248, 06:17:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:17:28 DEBUG } 06:17:28 DEBUG "0" 06:17:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:17:28 DEBUG ] 06:17:28 DEBUG } 06:17:28 DEBUG }, 06:17:28 DEBUG "375506050048" 06:17:28 DEBUG }, 06:17:28 DEBUG ] 06:17:28 DEBUG } 06:17:28 DEBUG ] 06:17:28 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:17:28 DEBUG } 06:17:28 DEBUG "value": [ 06:17:28 DEBUG ] 06:17:28 DEBUG "value": [ 06:17:28 DEBUG }, 06:17:28 DEBUG } 06:17:28 DEBUG } 06:17:28 DEBUG }, 06:17:28 DEBUG "value": [ 06:17:28 DEBUG } 06:17:28 DEBUG ] 06:17:28 DEBUG "value": [ 06:17:28 DEBUG ] 06:17:28 DEBUG 1712726248, 06:17:28 DEBUG 1712726248, 06:17:28 DEBUG "0" 06:17:28 DEBUG } 06:17:28 DEBUG } 06:17:28 DEBUG 1712726248, 06:17:28 DEBUG } 06:17:28 DEBUG } 06:17:28 DEBUG 1712726248, 06:17:28 DEBUG "0" 06:17:28 DEBUG "1448.8" 06:17:28 DEBUG ] 06:17:28 DEBUG "258747" 06:17:28 DEBUG ] 06:17:28 DEBUG ] 06:17:28 DEBUG } 06:17:28 DEBUG ] 06:17:28 DEBUG } 06:17:28 DEBUG } 06:17:28 DEBUG ] 06:17:28 DEBUG } 06:17:28 DEBUG ] 06:17:28 DEBUG ] 06:17:28 DEBUG } 06:17:28 DEBUG ] 06:17:28 DEBUG } 06:17:28 DEBUG } 06:17:28 DEBUG } 06:17:28 DEBUG } 06:17:28 DEBUG } 06:17:28 DEBUG } 06:17:28 DEBUG } 06:17:28 INFO [http_cmd]: http status code OK 06:17:28 DEBUG --- status code --- 06:17:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:28 DEBUG --- http response --- 06:17:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:17:28 DEBUG ----- output ----- 06:17:28 DEBUG { 06:17:28 DEBUG "status": "success", 06:17:28 DEBUG "data": { 06:17:28 DEBUG "resultType": "vector", 06:17:28 DEBUG "result": [ 06:17:28 DEBUG { 06:17:28 DEBUG "metric": { 06:17:28 DEBUG "instance": "10.142.0.25:9100" 06:17:28 DEBUG }, 06:17:28 DEBUG "value": [ 06:17:28 DEBUG 06:17:28 DEBUG 06:17:28 DEBUG 06:17:28 DEBUG [...] 06:17:28 DEBUG 06:17:28 DEBUG 06:17:28 DEBUG 06:17:28 DEBUG "instance": "10.142.0.77:9100" 06:17:28 DEBUG }, 06:17:28 DEBUG "value": [ 06:17:28 DEBUG 1712726248, 06:17:28 DEBUG "0.14596666666666777" 06:17:28 DEBUG ] 06:17:28 DEBUG } 06:17:28 DEBUG ] 06:17:28 DEBUG } 06:17:28 DEBUG } 06:17:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726249" 06:17:29 INFO [http_cmd]: http status code OK 06:17:29 DEBUG --- status code --- 06:17:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:29 DEBUG --- http response --- 06:17:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:17:29 DEBUG ----- output ----- 06:17:29 DEBUG { 06:17:29 DEBUG "status": "success", 06:17:29 DEBUG "data": { 06:17:29 DEBUG "resultType": "vector", 06:17:29 DEBUG "result": [ 06:17:29 DEBUG { 06:17:29 DEBUG "metric": { 06:17:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:17:29 DEBUG }, 06:17:29 DEBUG "value": [ 06:17:29 DEBUG 06:17:29 DEBUG 06:17:29 DEBUG 06:17:29 DEBUG [...] 06:17:29 DEBUG 06:17:29 DEBUG 06:17:29 DEBUG 06:17:29 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:17:29 DEBUG }, 06:17:29 DEBUG "value": [ 06:17:29 DEBUG 1712726249, 06:17:29 DEBUG "412434.7545451285" 06:17:29 DEBUG ] 06:17:29 DEBUG } 06:17:29 DEBUG ] 06:17:29 DEBUG } 06:17:29 DEBUG } 06:17:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712726249" 06:17:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712726249" 06:17:29 INFO [http_cmd]: http status code OK 06:17:29 DEBUG --- status code --- 06:17:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:29 DEBUG --- http response --- 06:17:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:17:29 DEBUG ----- output ----- 06:17:29 DEBUG { 06:17:29 DEBUG "status": "success", 06:17:29 DEBUG "data": { 06:17:29 DEBUG "resultType": "vector", 06:17:29 DEBUG "result": [ 06:17:29 DEBUG { 06:17:29 DEBUG "metric": { 06:17:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:17:29 DEBUG }, 06:17:29 DEBUG "value": [ 06:17:29 DEBUG 06:17:29 DEBUG 06:17:29 DEBUG 06:17:29 DEBUG [...] 06:17:29 DEBUG 06:17:29 DEBUG 06:17:29 DEBUG 06:17:29 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:17:29 DEBUG }, 06:17:29 DEBUG "value": [ 06:17:29 DEBUG 1712726249, 06:17:29 DEBUG "4.301064065602054" 06:17:29 DEBUG ] 06:17:29 DEBUG } 06:17:29 DEBUG ] 06:17:29 DEBUG } 06:17:29 DEBUG } 06:17:29 INFO [http_cmd]: http status code OK 06:17:29 DEBUG --- status code --- 06:17:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:29 DEBUG --- http response --- 06:17:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:17:29 DEBUG ----- output ----- 06:17:29 DEBUG { 06:17:29 DEBUG "status": "success", 06:17:29 DEBUG "data": { 06:17:29 DEBUG "resultType": "vector", 06:17:29 DEBUG "result": [ 06:17:29 DEBUG { 06:17:29 DEBUG "metric": { 06:17:29 DEBUG "pod": "ds-idrepo-0" 06:17:29 DEBUG }, 06:17:29 DEBUG "value": [ 06:17:29 DEBUG 06:17:29 DEBUG 06:17:29 DEBUG 06:17:29 DEBUG [...] 06:17:29 DEBUG 06:17:29 DEBUG 06:17:29 DEBUG 06:17:29 DEBUG "pod": "ds-cts-2" 06:17:29 DEBUG }, 06:17:29 DEBUG "value": [ 06:17:29 DEBUG 1712726249, 06:17:29 DEBUG "0" 06:17:29 DEBUG ] 06:17:29 DEBUG } 06:17:29 DEBUG ] 06:17:29 DEBUG } 06:17:29 DEBUG } 06:17:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726250" 06:17:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726250" 06:17:30 INFO [http_cmd]: http status code OK 06:17:30 DEBUG --- status code --- 06:17:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:30 DEBUG --- http response --- 06:17:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:17:30 DEBUG ----- output ----- 06:17:30 DEBUG { 06:17:30 DEBUG "status": "success", 06:17:30 DEBUG "data": { 06:17:30 DEBUG "resultType": "vector", 06:17:30 DEBUG "result": [ 06:17:30 DEBUG { 06:17:30 DEBUG "metric": { 06:17:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:17:30 DEBUG }, 06:17:30 DEBUG "value": [ 06:17:30 DEBUG 06:17:30 DEBUG 06:17:30 DEBUG 06:17:30 DEBUG [...] 06:17:30 DEBUG 06:17:30 DEBUG 06:17:30 DEBUG 06:17:30 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:17:30 DEBUG }, 06:17:30 DEBUG "value": [ 06:17:30 DEBUG 1712726250, 06:17:30 DEBUG "4.311326918169628" 06:17:30 DEBUG ] 06:17:30 DEBUG } 06:17:30 DEBUG ] 06:17:30 DEBUG } 06:17:30 DEBUG } 06:17:30 INFO [http_cmd]: http status code OK 06:17:30 DEBUG --- status code --- 06:17:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:30 DEBUG --- http response --- 06:17:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:17:30 DEBUG ----- output ----- 06:17:30 DEBUG { 06:17:30 DEBUG "status": "success", 06:17:30 DEBUG "data": { 06:17:30 DEBUG "resultType": "vector", 06:17:30 DEBUG "result": [ 06:17:30 DEBUG { 06:17:30 DEBUG "metric": { 06:17:30 DEBUG "pod": "ds-idrepo-2" 06:17:30 DEBUG }, 06:17:30 DEBUG "value": [ 06:17:30 DEBUG 06:17:30 DEBUG 06:17:30 DEBUG 06:17:30 DEBUG [...] 06:17:30 DEBUG 06:17:30 DEBUG 06:17:30 DEBUG 06:17:30 DEBUG "pod": "ds-cts-2" 06:17:30 DEBUG }, 06:17:30 DEBUG "value": [ 06:17:30 DEBUG 1712726250, 06:17:30 DEBUG "0.000022885555555555554" 06:17:30 DEBUG ] 06:17:30 DEBUG } 06:17:30 DEBUG ] 06:17:30 DEBUG } 06:17:30 DEBUG } 06:17:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726250" 06:17:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712726250" 06:17:30 INFO [http_cmd]: http status code OK 06:17:30 DEBUG --- status code --- 06:17:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:30 DEBUG --- http response --- 06:17:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:17:30 DEBUG ----- output ----- 06:17:30 DEBUG { 06:17:30 DEBUG "status": "success", 06:17:30 DEBUG "data": { 06:17:30 DEBUG "resultType": "vector", 06:17:30 DEBUG "result": [ 06:17:30 DEBUG { 06:17:30 DEBUG "metric": { 06:17:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:17:30 DEBUG }, 06:17:30 DEBUG "value": [ 06:17:30 DEBUG 06:17:30 DEBUG 06:17:30 DEBUG 06:17:30 DEBUG [...] 06:17:30 DEBUG 06:17:30 DEBUG 06:17:30 DEBUG 06:17:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:17:30 DEBUG }, 06:17:30 DEBUG "value": [ 06:17:30 DEBUG 1712726250, 06:17:30 DEBUG "1449.0333333333333" 06:17:30 DEBUG ] 06:17:30 DEBUG } 06:17:30 DEBUG ] 06:17:30 DEBUG } 06:17:30 DEBUG } 06:17:30 INFO [http_cmd]: http status code OK 06:17:30 DEBUG --- status code --- 06:17:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:30 DEBUG --- http response --- 06:17:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:17:30 DEBUG ----- output ----- 06:17:30 DEBUG { 06:17:30 DEBUG "status": "success", 06:17:30 DEBUG "data": { 06:17:30 DEBUG "resultType": "vector", 06:17:30 DEBUG "result": [ 06:17:30 DEBUG { 06:17:30 DEBUG "metric": { 06:17:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:17:30 DEBUG }, 06:17:30 DEBUG "value": [ 06:17:30 DEBUG 06:17:30 DEBUG 06:17:30 DEBUG 06:17:30 DEBUG [...] 06:17:30 DEBUG 06:17:30 DEBUG 06:17:30 DEBUG 06:17:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:17:30 DEBUG }, 06:17:30 DEBUG "value": [ 06:17:30 DEBUG 1712726250, 06:17:30 DEBUG "0.05" 06:17:30 DEBUG ] 06:17:30 DEBUG } 06:17:30 DEBUG ] 06:17:30 DEBUG } 06:17:30 DEBUG } 06:17:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726250" 06:17:30 INFO [http_cmd]: http status code OK 06:17:30 DEBUG --- status code --- 06:17:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:30 DEBUG --- http response --- 06:17:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:17:30 DEBUG ----- output ----- 06:17:30 DEBUG { 06:17:30 DEBUG "status": "success", 06:17:30 DEBUG "data": { 06:17:30 DEBUG "resultType": "vector", 06:17:30 DEBUG "result": [ 06:17:30 DEBUG { 06:17:30 DEBUG "metric": { 06:17:30 DEBUG "pod": "ds-idrepo-0" 06:17:30 DEBUG }, 06:17:30 DEBUG "value": [ 06:17:30 DEBUG 06:17:30 DEBUG 06:17:30 DEBUG 06:17:30 DEBUG [...] 06:17:30 DEBUG 06:17:30 DEBUG 06:17:30 DEBUG 06:17:30 DEBUG "pod": "ds-idrepo-1" 06:17:30 DEBUG }, 06:17:30 DEBUG "value": [ 06:17:30 DEBUG 1712726250, 06:17:30 DEBUG "0" 06:17:30 DEBUG ] 06:17:30 DEBUG } 06:17:30 DEBUG ] 06:17:30 DEBUG } 06:17:30 DEBUG } 06:17:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726250" 06:17:30 INFO [http_cmd]: http status code OK 06:17:30 DEBUG --- status code --- 06:17:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:30 DEBUG --- http response --- 06:17:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:17:30 DEBUG ----- output ----- 06:17:30 DEBUG { 06:17:30 DEBUG "status": "success", 06:17:30 DEBUG "data": { 06:17:30 DEBUG "resultType": "vector", 06:17:30 DEBUG "result": [ 06:17:30 DEBUG { 06:17:30 DEBUG "metric": { 06:17:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:17:30 DEBUG }, 06:17:30 DEBUG "value": [ 06:17:30 DEBUG 06:17:30 DEBUG 06:17:30 DEBUG 06:17:30 DEBUG [...] 06:17:30 DEBUG 06:17:30 DEBUG 06:17:30 DEBUG 06:17:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:17:30 DEBUG }, 06:17:30 DEBUG "value": [ 06:17:30 DEBUG 1712726250, 06:17:30 DEBUG "1456.5666666666666" 06:17:30 DEBUG ] 06:17:30 DEBUG } 06:17:30 DEBUG ] 06:17:30 DEBUG } 06:17:30 DEBUG } 06:17:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726251" 06:17:31 INFO [http_cmd]: http status code OK 06:17:31 DEBUG --- status code --- 06:17:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:31 DEBUG --- http response --- 06:17:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:17:31 DEBUG ----- output ----- 06:17:31 DEBUG { 06:17:31 DEBUG "status": "success", 06:17:31 DEBUG "data": { 06:17:31 DEBUG "resultType": "vector", 06:17:31 DEBUG "result": [ 06:17:31 DEBUG { 06:17:31 DEBUG "metric": { 06:17:31 DEBUG "pod": "ds-idrepo-0" 06:17:31 DEBUG }, 06:17:31 DEBUG "value": [ 06:17:31 DEBUG 06:17:31 DEBUG 06:17:31 DEBUG 06:17:31 DEBUG [...] 06:17:31 DEBUG 06:17:31 DEBUG 06:17:31 DEBUG 06:17:31 DEBUG "pod": "ds-idrepo-1" 06:17:31 DEBUG }, 06:17:31 DEBUG "value": [ 06:17:31 DEBUG 1712726251, 06:17:31 DEBUG "0" 06:17:31 DEBUG ] 06:17:31 DEBUG } 06:17:31 DEBUG ] 06:17:31 DEBUG } 06:17:31 DEBUG } 06:17:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726251" 06:17:31 INFO [http_cmd]: http status code OK 06:17:31 DEBUG --- status code --- 06:17:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:31 DEBUG --- http response --- 06:17:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:17:31 DEBUG ----- output ----- 06:17:31 DEBUG { 06:17:31 DEBUG "status": "success", 06:17:31 DEBUG "data": { 06:17:31 DEBUG "resultType": "vector", 06:17:31 DEBUG "result": [ 06:17:31 DEBUG { 06:17:31 DEBUG "metric": { 06:17:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:17:31 DEBUG }, 06:17:31 DEBUG "value": [ 06:17:31 DEBUG 06:17:31 DEBUG 06:17:31 DEBUG 06:17:31 DEBUG [...] 06:17:31 DEBUG 06:17:31 DEBUG 06:17:31 DEBUG 06:17:31 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:17:31 DEBUG }, 06:17:31 DEBUG "value": [ 06:17:31 DEBUG 1712726251, 06:17:31 DEBUG "0" 06:17:31 DEBUG ] 06:17:31 DEBUG } 06:17:31 DEBUG ] 06:17:31 DEBUG } 06:17:31 DEBUG } 06:17:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712726251" 06:17:31 INFO [http_cmd]: http status code OK 06:17:31 DEBUG --- status code --- 06:17:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:31 DEBUG --- http response --- 06:17:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:17:31 DEBUG ----- output ----- 06:17:31 DEBUG { 06:17:31 DEBUG "status": "success", 06:17:31 DEBUG "data": { 06:17:31 DEBUG "resultType": "vector", 06:17:31 DEBUG "result": [ 06:17:31 DEBUG { 06:17:31 DEBUG "metric": { 06:17:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:17:31 DEBUG }, 06:17:31 DEBUG "value": [ 06:17:31 DEBUG 06:17:31 DEBUG 06:17:31 DEBUG 06:17:31 DEBUG [...] 06:17:31 DEBUG 06:17:31 DEBUG 06:17:31 DEBUG 06:17:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:17:31 DEBUG }, 06:17:31 DEBUG "value": [ 06:17:31 DEBUG 1712726251, 06:17:31 DEBUG "2488205312" 06:17:31 DEBUG ] 06:17:31 DEBUG } 06:17:31 DEBUG ] 06:17:31 DEBUG } 06:17:31 DEBUG } 06:17:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726251" 06:17:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726251" 06:17:31 INFO [http_cmd]: http status code OK 06:17:31 DEBUG --- status code --- 06:17:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:31 DEBUG --- http response --- 06:17:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:17:31 DEBUG ----- output ----- 06:17:31 DEBUG { 06:17:31 DEBUG "status": "success", 06:17:31 DEBUG "data": { 06:17:31 DEBUG "resultType": "vector", 06:17:31 DEBUG "result": [ 06:17:31 DEBUG { 06:17:31 DEBUG "metric": { 06:17:31 DEBUG "instance": "10.142.0.25:9100" 06:17:31 DEBUG }, 06:17:31 DEBUG "value": [ 06:17:31 DEBUG 06:17:31 DEBUG 06:17:31 DEBUG 06:17:31 DEBUG [...] 06:17:31 DEBUG 06:17:31 DEBUG 06:17:31 DEBUG 06:17:31 DEBUG "instance": "10.142.0.77:9100" 06:17:31 DEBUG }, 06:17:31 DEBUG "value": [ 06:17:31 DEBUG 1712726251, 06:17:31 DEBUG "42696549.36666667" 06:17:31 DEBUG ] 06:17:31 DEBUG } 06:17:31 DEBUG ] 06:17:31 DEBUG } 06:17:31 DEBUG } 06:17:31 INFO [http_cmd]: http status code OK 06:17:31 DEBUG --- status code --- 06:17:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:17:31 DEBUG --- http response --- 06:17:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:17:31 DEBUG ----- output ----- 06:17:31 DEBUG { 06:17:31 DEBUG "status": "success", 06:17:31 DEBUG "data": { 06:17:31 DEBUG "resultType": "vector", 06:17:31 DEBUG "result": [ 06:17:31 DEBUG { 06:17:31 DEBUG "metric": { 06:17:31 DEBUG "instance": "10.142.0.25:9100" 06:17:31 DEBUG }, 06:17:31 DEBUG "value": [ 06:17:31 DEBUG 06:17:31 DEBUG 06:17:31 DEBUG 06:17:31 DEBUG [...] 06:17:31 DEBUG 06:17:31 DEBUG 06:17:31 DEBUG 06:17:31 DEBUG "instance": "10.142.0.77:9100" 06:17:31 DEBUG }, 06:17:31 DEBUG "value": [ 06:17:31 DEBUG 1712726251, 06:17:31 DEBUG "0.04766666666666713" 06:17:31 DEBUG ] 06:17:31 DEBUG } 06:17:31 DEBUG ] 06:17:31 DEBUG } 06:17:31 DEBUG } 06:17:49 INFO 06:17:49 INFO [loop_until]: kubectl --namespace=xlou top pods 06:17:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:17:49 INFO [loop_until]: OK (rc = 0) 06:17:49 DEBUG --- stdout --- 06:17:49 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10975m 8980Mi am-654c4fcbcf-9scrj 9885m 8442Mi am-654c4fcbcf-v5qb5 8179m 8447Mi ds-cts-0 8266m 12993Mi ds-cts-1 8188m 12438Mi ds-cts-2 8019m 12449Mi ds-idrepo-0 10543m 5623Mi ds-idrepo-1 400m 5672Mi ds-idrepo-2 429m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 5m 2213Mi lodemon-b776b7d88-thrqf 2m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4325m 2289Mi 06:17:49 DEBUG --- stderr --- 06:17:49 DEBUG 06:17:57 INFO 06:17:57 INFO [loop_until]: kubectl --namespace=xlou top node 06:17:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:17:57 INFO [loop_until]: OK (rc = 0) 06:17:57 DEBUG --- stdout --- 06:17:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 98m 0% 2307Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9242m 58% 9318Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 77m 0% 3259Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13357m 84% 6673Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10870m 68% 9327Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11943m 75% 9800Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 78m 0% 855Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10516m 66% 6300Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8361m 52% 13600Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 8215m 51% 13222Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 485m 3% 6385Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8484m 53% 14083Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 468m 2% 6351Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4457m 28% 5816Mi 9% 06:17:57 DEBUG --- stderr --- 06:17:57 DEBUG 06:18:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712726307" 06:18:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726307" 06:18:27 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726307" 06:18:27 INFO [http_cmd]: http status code OK 06:18:27 ERROR [http_cmd]: ERROR 06:18:27 ERROR --- status code --- 06:18:27 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:18:27 ERROR --- http response --- 06:18:27 DEBUG --- status code --- 06:18:27 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:18:27 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:18:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:27 DEBUG --- http response --- 06:18:27 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:18:27 INFO [http_cmd]: http status code OK 06:18:27 DEBUG --- status code --- 06:18:27 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:27 DEBUG --- http response --- 06:18:27 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:18:27 DEBUG ----- output ----- 06:18:27 DEBUG { 06:18:27 DEBUG "status": "success", 06:18:27 DEBUG "data": { 06:18:27 DEBUG "resultType": "vector", 06:18:27 DEBUG "result": [ 06:18:27 DEBUG { 06:18:27 DEBUG "metric": { 06:18:27 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:18:27 DEBUG }, 06:18:27 DEBUG "value": [ 06:18:27 DEBUG 06:18:27 DEBUG 06:18:27 DEBUG 06:18:27 DEBUG [...] 06:18:27 DEBUG 06:18:27 DEBUG 06:18:27 DEBUG 06:18:27 DEBUG "pod": "am-654c4fcbcf-487cv" 06:18:27 DEBUG }, 06:18:27 DEBUG "value": [ 06:18:27 DEBUG 1712726307, 06:18:27 DEBUG "0" 06:18:27 DEBUG ] 06:18:27 DEBUG } 06:18:27 DEBUG ] 06:18:27 DEBUG } 06:18:27 DEBUG } 06:18:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726308" 06:18:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726308" 06:18:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726308" 06:18:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726308" 06:18:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726308" 06:18:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726308" 06:18:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726308" 06:18:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726308" 06:18:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726308" 06:18:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726308" 06:18:28 INFO [http_cmd]: http status code OK 06:18:28 DEBUG --- status code --- 06:18:28 INFO [http_cmd]: http status code OK 06:18:28 DEBUG --- status code --- 06:18:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:28 DEBUG --- http response --- 06:18:28 INFO [http_cmd]: http status code OK 06:18:28 DEBUG --- http response --- 06:18:28 INFO [http_cmd]: http status code OK 06:18:28 DEBUG --- status code --- 06:18:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:18:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:18:28 DEBUG --- status code --- 06:18:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:28 DEBUG --- http response --- 06:18:28 DEBUG ----- output ----- 06:18:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:28 DEBUG --- http response --- 06:18:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:18:28 DEBUG ----- output ----- 06:18:28 DEBUG { 06:18:28 DEBUG "status": "success", 06:18:28 DEBUG "data": { 06:18:28 DEBUG "resultType": "vector", 06:18:28 DEBUG "result": [ 06:18:28 DEBUG { 06:18:28 DEBUG "metric": { 06:18:28 DEBUG { 06:18:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:18:28 DEBUG ----- output ----- 06:18:28 DEBUG "instance": "10.142.0.25:9100" 06:18:28 INFO [http_cmd]: http status code OK 06:18:28 INFO [http_cmd]: http status code OK 06:18:28 INFO [http_cmd]: http status code OK 06:18:28 INFO [http_cmd]: http status code OK 06:18:28 DEBUG --- status code --- 06:18:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:28 DEBUG "status": "success", 06:18:28 DEBUG ----- output ----- 06:18:28 DEBUG { 06:18:28 DEBUG }, 06:18:28 DEBUG --- status code --- 06:18:28 DEBUG --- status code --- 06:18:28 DEBUG --- status code --- 06:18:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:28 DEBUG --- http response --- 06:18:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:18:28 DEBUG "data": { 06:18:28 DEBUG { 06:18:28 DEBUG "status": "success", 06:18:28 DEBUG "value": [ 06:18:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:28 INFO [http_cmd]: http status code OK 06:18:28 DEBUG --- http response --- 06:18:28 INFO [http_cmd]: http status code OK 06:18:28 DEBUG ----- output ----- 06:18:28 DEBUG "resultType": "vector", 06:18:28 DEBUG "status": "success", 06:18:28 DEBUG "data": { 06:18:28 DEBUG 06:18:28 DEBUG --- http response --- 06:18:28 DEBUG --- http response --- 06:18:28 DEBUG --- status code --- 06:18:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:28 DEBUG --- status code --- 06:18:28 DEBUG { 06:18:28 DEBUG "result": [ 06:18:28 DEBUG "data": { 06:18:28 DEBUG "resultType": "vector", 06:18:28 DEBUG 06:18:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:18:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:18:28 DEBUG --- http response --- 06:18:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:18:28 DEBUG "status": "success", 06:18:28 DEBUG { 06:18:28 DEBUG "resultType": "vector", 06:18:28 DEBUG "result": [ 06:18:28 DEBUG 06:18:28 DEBUG ----- output ----- 06:18:28 DEBUG ----- output ----- 06:18:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:18:28 DEBUG --- http response --- 06:18:28 DEBUG ----- output ----- 06:18:28 DEBUG "data": { 06:18:28 DEBUG "metric": { 06:18:28 DEBUG "result": [ 06:18:28 DEBUG { 06:18:28 DEBUG [...] 06:18:28 DEBUG { 06:18:28 DEBUG { 06:18:28 DEBUG ----- output ----- 06:18:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:18:28 DEBUG { 06:18:28 DEBUG "status": "success", 06:18:28 DEBUG "pod": "ds-idrepo-0" 06:18:28 DEBUG { 06:18:28 DEBUG "metric": { 06:18:28 DEBUG 06:18:28 DEBUG 06:18:28 DEBUG "status": "success", 06:18:28 DEBUG { 06:18:28 DEBUG ----- output ----- 06:18:28 DEBUG "data": { 06:18:28 DEBUG "resultType": "vector", 06:18:28 DEBUG }, 06:18:28 DEBUG "metric": { 06:18:28 DEBUG "instance": "10.142.0.25:9100" 06:18:28 DEBUG 06:18:28 DEBUG "status": "success", 06:18:28 DEBUG "data": { 06:18:28 DEBUG "status": "success", 06:18:28 DEBUG { 06:18:28 DEBUG "result": [ 06:18:28 DEBUG { 06:18:28 DEBUG "value": [ 06:18:28 DEBUG 06:18:28 DEBUG }, 06:18:28 DEBUG "instance": "10.142.0.77:9100" 06:18:28 DEBUG }, 06:18:28 DEBUG "resultType": "vector", 06:18:28 DEBUG "data": { 06:18:28 DEBUG "resultType": "vector", 06:18:28 DEBUG "result": [ 06:18:28 DEBUG { 06:18:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:18:28 DEBUG 06:18:28 DEBUG "value": [ 06:18:28 DEBUG 06:18:28 DEBUG "data": { 06:18:28 DEBUG "result": [ 06:18:28 DEBUG "status": "success", 06:18:28 DEBUG "data": { 06:18:28 DEBUG "resultType": "vector", 06:18:28 DEBUG "metric": { 06:18:28 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:18:28 DEBUG 06:18:28 DEBUG "value": [ 06:18:28 DEBUG 06:18:28 DEBUG "resultType": "vector", 06:18:28 DEBUG "result": [ 06:18:28 DEBUG { 06:18:28 DEBUG "metric": { 06:18:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:18:28 DEBUG }, 06:18:28 DEBUG "value": [ 06:18:28 DEBUG [...] 06:18:28 DEBUG 1712726308, 06:18:28 DEBUG 06:18:28 DEBUG { 06:18:28 DEBUG "metric": { 06:18:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:18:28 DEBUG }, 06:18:28 DEBUG "value": [ 06:18:28 DEBUG }, 06:18:28 DEBUG "value": [ 06:18:28 DEBUG 06:18:28 DEBUG "377568621568" 06:18:28 DEBUG [...] 06:18:28 DEBUG 06:18:28 DEBUG 06:18:28 DEBUG 06:18:28 DEBUG [...] 06:18:28 DEBUG 06:18:28 DEBUG 06:18:28 DEBUG 06:18:28 DEBUG 06:18:28 DEBUG 06:18:28 DEBUG [...] 06:18:28 DEBUG 06:18:28 DEBUG 06:18:28 DEBUG "resultType": "vector", 06:18:28 DEBUG "result": [ 06:18:28 DEBUG { 06:18:28 DEBUG "metric": { 06:18:28 DEBUG }, 06:18:28 DEBUG 06:18:28 DEBUG 06:18:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:18:28 DEBUG }, 06:18:28 DEBUG "pod": "ds-idrepo-0" 06:18:28 DEBUG }, 06:18:28 DEBUG "value": [ 06:18:28 DEBUG 06:18:28 DEBUG 06:18:28 DEBUG 06:18:28 DEBUG [...] 06:18:28 DEBUG 06:18:28 DEBUG 06:18:28 DEBUG "value": [ 06:18:28 DEBUG 06:18:28 DEBUG 06:18:28 DEBUG 06:18:28 DEBUG [...] 06:18:28 DEBUG 06:18:28 DEBUG "result": [ 06:18:28 DEBUG { 06:18:28 DEBUG "metric": { 06:18:28 DEBUG "instance": "10.142.0.25:9100" 06:18:28 DEBUG }, 06:18:28 DEBUG 06:18:28 DEBUG 06:18:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:18:28 DEBUG }, 06:18:28 DEBUG "value": [ 06:18:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:18:28 DEBUG }, 06:18:28 DEBUG "value": [ 06:18:28 DEBUG 1712726308, 06:18:28 DEBUG "259377" 06:18:28 DEBUG 1712726308, 06:18:28 DEBUG "1438.1" 06:18:28 DEBUG ] 06:18:28 DEBUG } 06:18:28 DEBUG 06:18:28 DEBUG ] 06:18:28 DEBUG } 06:18:28 DEBUG ] 06:18:28 DEBUG } 06:18:28 DEBUG } 06:18:28 DEBUG ] 06:18:28 DEBUG } 06:18:28 DEBUG "metric": { 06:18:28 DEBUG 06:18:28 DEBUG 06:18:28 DEBUG 06:18:28 DEBUG "value": [ 06:18:28 DEBUG 06:18:28 DEBUG 06:18:28 DEBUG 06:18:28 DEBUG [...] 06:18:28 DEBUG 06:18:28 DEBUG } 06:18:28 DEBUG 06:18:28 DEBUG "instance": "10.142.0.77:9100" 06:18:28 DEBUG 06:18:28 DEBUG "pod": "ds-idrepo-0" 06:18:28 DEBUG 06:18:28 DEBUG ] 06:18:28 DEBUG "value": [ 06:18:28 DEBUG 06:18:28 DEBUG "pod": "ds-cts-2" 06:18:28 DEBUG }, 06:18:28 DEBUG 06:18:28 DEBUG }, 06:18:28 DEBUG 06:18:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:18:28 DEBUG } 06:18:28 DEBUG ] 06:18:28 DEBUG "value": [ 06:18:28 DEBUG 1712726308, 06:18:28 DEBUG 06:18:28 DEBUG "value": [ 06:18:28 DEBUG }, 06:18:28 DEBUG "value": [ 06:18:28 DEBUG } 06:18:28 DEBUG } 06:18:28 DEBUG "0" 06:18:28 DEBUG ] 06:18:28 DEBUG } 06:18:28 DEBUG 06:18:28 DEBUG 1712726308, 06:18:28 DEBUG "1438.3666666666666" 06:18:28 DEBUG ] 06:18:28 DEBUG }, 06:18:28 DEBUG "value": [ 06:18:28 DEBUG 1712726308, 06:18:28 DEBUG "42468906.06666666" 06:18:28 DEBUG 06:18:28 DEBUG 1712726308, 06:18:28 DEBUG "pod": "ds-cts-2" 06:18:28 DEBUG } 06:18:28 DEBUG ] 06:18:28 DEBUG } 06:18:28 DEBUG ] 06:18:28 DEBUG } 06:18:28 DEBUG "0" 06:18:28 DEBUG }, 06:18:28 DEBUG ] 06:18:28 DEBUG } 06:18:28 DEBUG [...] 06:18:28 DEBUG 06:18:28 DEBUG 06:18:28 DEBUG ] 06:18:28 DEBUG } 06:18:28 DEBUG ] 06:18:28 DEBUG 06:18:28 DEBUG 06:18:28 DEBUG "instance": "10.142.0.77:9100" 06:18:28 DEBUG [...] 06:18:28 DEBUG } 06:18:28 DEBUG } 06:18:28 DEBUG "value": [ 06:18:28 DEBUG }, 06:18:28 DEBUG "value": [ 06:18:28 DEBUG 06:18:28 DEBUG } 06:18:28 DEBUG } 06:18:28 DEBUG ] 06:18:28 DEBUG 1712726308, 06:18:28 DEBUG 06:18:28 DEBUG 1712726308, 06:18:28 DEBUG "29.933333333333334" 06:18:28 DEBUG "0.06749999999999864" 06:18:28 DEBUG ] 06:18:28 DEBUG } 06:18:28 DEBUG ] 06:18:28 DEBUG } 06:18:28 DEBUG ] 06:18:28 DEBUG } 06:18:28 DEBUG } 06:18:28 DEBUG ] 06:18:28 DEBUG } 06:18:28 DEBUG 06:18:28 DEBUG } 06:18:28 DEBUG } 06:18:28 DEBUG "pod": "ds-idrepo-1" 06:18:28 DEBUG } 06:18:28 DEBUG }, 06:18:28 DEBUG "value": [ 06:18:28 DEBUG 1712726308, 06:18:28 DEBUG "0" 06:18:28 DEBUG ] 06:18:28 DEBUG } 06:18:28 DEBUG ] 06:18:28 DEBUG } 06:18:28 DEBUG } 06:18:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726309" 06:18:29 INFO [http_cmd]: http status code OK 06:18:29 DEBUG --- status code --- 06:18:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:29 DEBUG --- http response --- 06:18:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:18:29 DEBUG ----- output ----- 06:18:29 DEBUG { 06:18:29 DEBUG "status": "success", 06:18:29 DEBUG "data": { 06:18:29 DEBUG "resultType": "vector", 06:18:29 DEBUG "result": [ 06:18:29 DEBUG { 06:18:29 DEBUG "metric": { 06:18:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:18:29 DEBUG }, 06:18:29 DEBUG "value": [ 06:18:29 DEBUG 06:18:29 DEBUG 06:18:29 DEBUG 06:18:29 DEBUG [...] 06:18:29 DEBUG 06:18:29 DEBUG 06:18:29 DEBUG 06:18:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:18:29 DEBUG }, 06:18:29 DEBUG "value": [ 06:18:29 DEBUG 1712726309, 06:18:29 DEBUG "261953753.2690373" 06:18:29 DEBUG ] 06:18:29 DEBUG } 06:18:29 DEBUG ] 06:18:29 DEBUG } 06:18:29 DEBUG } 06:18:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712726309" 06:18:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712726309" 06:18:29 INFO [http_cmd]: http status code OK 06:18:29 DEBUG --- status code --- 06:18:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:29 DEBUG --- http response --- 06:18:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:18:29 DEBUG ----- output ----- 06:18:29 DEBUG { 06:18:29 DEBUG "status": "success", 06:18:29 DEBUG "data": { 06:18:29 DEBUG "resultType": "vector", 06:18:29 DEBUG "result": [ 06:18:29 DEBUG { 06:18:29 DEBUG "metric": { 06:18:29 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:18:29 DEBUG }, 06:18:29 DEBUG "value": [ 06:18:29 DEBUG 06:18:29 DEBUG 06:18:29 DEBUG 06:18:29 DEBUG [...] 06:18:29 DEBUG 06:18:29 DEBUG 06:18:29 DEBUG 06:18:29 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:18:29 DEBUG }, 06:18:29 DEBUG "value": [ 06:18:29 DEBUG 1712726309, 06:18:29 DEBUG "4.348928509720758" 06:18:29 DEBUG ] 06:18:29 DEBUG } 06:18:29 DEBUG ] 06:18:29 DEBUG } 06:18:29 DEBUG } 06:18:29 INFO [http_cmd]: http status code OK 06:18:29 DEBUG --- status code --- 06:18:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:29 DEBUG --- http response --- 06:18:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:18:29 DEBUG ----- output ----- 06:18:29 DEBUG { 06:18:29 DEBUG "status": "success", 06:18:29 DEBUG "data": { 06:18:29 DEBUG "resultType": "vector", 06:18:29 DEBUG "result": [ 06:18:29 DEBUG { 06:18:29 DEBUG "metric": { 06:18:29 DEBUG "pod": "ds-idrepo-0" 06:18:29 DEBUG }, 06:18:29 DEBUG "value": [ 06:18:29 DEBUG 06:18:29 DEBUG 06:18:29 DEBUG 06:18:29 DEBUG [...] 06:18:29 DEBUG 06:18:29 DEBUG 06:18:29 DEBUG 06:18:29 DEBUG "pod": "ds-cts-2" 06:18:29 DEBUG }, 06:18:29 DEBUG "value": [ 06:18:29 DEBUG 1712726309, 06:18:29 DEBUG "0" 06:18:29 DEBUG ] 06:18:29 DEBUG } 06:18:29 DEBUG ] 06:18:29 DEBUG } 06:18:29 DEBUG } 06:18:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726310" 06:18:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726310" 06:18:30 INFO [http_cmd]: http status code OK 06:18:30 INFO [http_cmd]: http status code OK 06:18:30 DEBUG --- status code --- 06:18:30 DEBUG --- status code --- 06:18:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:30 DEBUG --- http response --- 06:18:30 DEBUG --- http response --- 06:18:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:18:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:18:30 DEBUG ----- output ----- 06:18:30 DEBUG ----- output ----- 06:18:30 DEBUG { 06:18:30 DEBUG { 06:18:30 DEBUG "status": "success", 06:18:30 DEBUG "status": "success", 06:18:30 DEBUG "data": { 06:18:30 DEBUG "data": { 06:18:30 DEBUG "resultType": "vector", 06:18:30 DEBUG "resultType": "vector", 06:18:30 DEBUG "result": [ 06:18:30 DEBUG "result": [ 06:18:30 DEBUG { 06:18:30 DEBUG { 06:18:30 DEBUG "metric": { 06:18:30 DEBUG "metric": { 06:18:30 DEBUG "pod": "ds-idrepo-2" 06:18:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:18:30 DEBUG }, 06:18:30 DEBUG }, 06:18:30 DEBUG "value": [ 06:18:30 DEBUG 06:18:30 DEBUG "value": [ 06:18:30 DEBUG 06:18:30 DEBUG 06:18:30 DEBUG 06:18:30 DEBUG 06:18:30 DEBUG [...] 06:18:30 DEBUG 06:18:30 DEBUG 06:18:30 DEBUG [...] 06:18:30 DEBUG 06:18:30 DEBUG 06:18:30 DEBUG 06:18:30 DEBUG 06:18:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:18:30 DEBUG 06:18:30 DEBUG }, 06:18:30 DEBUG "pod": "ds-cts-2" 06:18:30 DEBUG "value": [ 06:18:30 DEBUG }, 06:18:30 DEBUG 1712726310, 06:18:30 DEBUG "8.348992369741374" 06:18:30 DEBUG ] 06:18:30 DEBUG "value": [ 06:18:30 DEBUG } 06:18:30 DEBUG ] 06:18:30 DEBUG 1712726310, 06:18:30 DEBUG } 06:18:30 DEBUG "0.00003333333333333336" 06:18:30 DEBUG } 06:18:30 DEBUG ] 06:18:30 DEBUG } 06:18:30 DEBUG ] 06:18:30 DEBUG } 06:18:30 DEBUG } 06:18:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726310" 06:18:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712726310" 06:18:30 INFO [http_cmd]: http status code OK 06:18:30 DEBUG --- status code --- 06:18:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:30 DEBUG --- http response --- 06:18:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:18:30 DEBUG ----- output ----- 06:18:30 DEBUG { 06:18:30 DEBUG "status": "success", 06:18:30 DEBUG "data": { 06:18:30 DEBUG "resultType": "vector", 06:18:30 DEBUG "result": [ 06:18:30 DEBUG { 06:18:30 DEBUG "metric": { 06:18:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:18:30 DEBUG }, 06:18:30 DEBUG "value": [ 06:18:30 DEBUG 06:18:30 DEBUG 06:18:30 DEBUG 06:18:30 DEBUG [...] 06:18:30 DEBUG 06:18:30 DEBUG 06:18:30 DEBUG 06:18:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:18:30 DEBUG }, 06:18:30 DEBUG "value": [ 06:18:30 DEBUG 1712726310, 06:18:30 DEBUG "1438.5333333333333" 06:18:30 DEBUG ] 06:18:30 DEBUG } 06:18:30 DEBUG ] 06:18:30 DEBUG } 06:18:30 DEBUG } 06:18:30 INFO [http_cmd]: http status code OK 06:18:30 DEBUG --- status code --- 06:18:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:30 DEBUG --- http response --- 06:18:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:18:30 DEBUG ----- output ----- 06:18:30 DEBUG { 06:18:30 DEBUG "status": "success", 06:18:30 DEBUG "data": { 06:18:30 DEBUG "resultType": "vector", 06:18:30 DEBUG "result": [ 06:18:30 DEBUG { 06:18:30 DEBUG "metric": { 06:18:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:18:30 DEBUG }, 06:18:30 DEBUG "value": [ 06:18:30 DEBUG 06:18:30 DEBUG 06:18:30 DEBUG 06:18:30 DEBUG [...] 06:18:30 DEBUG 06:18:30 DEBUG 06:18:30 DEBUG 06:18:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:18:30 DEBUG }, 06:18:30 DEBUG "value": [ 06:18:30 DEBUG 1712726310, 06:18:30 DEBUG "0.05" 06:18:30 DEBUG ] 06:18:30 DEBUG } 06:18:30 DEBUG ] 06:18:30 DEBUG } 06:18:30 DEBUG } 06:18:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726310" 06:18:30 INFO [http_cmd]: http status code OK 06:18:30 DEBUG --- status code --- 06:18:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:30 DEBUG --- http response --- 06:18:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:18:30 DEBUG ----- output ----- 06:18:30 DEBUG { 06:18:30 DEBUG "status": "success", 06:18:30 DEBUG "data": { 06:18:30 DEBUG "resultType": "vector", 06:18:30 DEBUG "result": [ 06:18:30 DEBUG { 06:18:30 DEBUG "metric": { 06:18:30 DEBUG "pod": "ds-idrepo-0" 06:18:30 DEBUG }, 06:18:30 DEBUG "value": [ 06:18:30 DEBUG 06:18:30 DEBUG 06:18:30 DEBUG 06:18:30 DEBUG [...] 06:18:30 DEBUG 06:18:30 DEBUG 06:18:30 DEBUG 06:18:30 DEBUG "pod": "ds-idrepo-1" 06:18:30 DEBUG }, 06:18:30 DEBUG "value": [ 06:18:30 DEBUG 1712726310, 06:18:30 DEBUG "0" 06:18:30 DEBUG ] 06:18:30 DEBUG } 06:18:30 DEBUG ] 06:18:30 DEBUG } 06:18:30 DEBUG } 06:18:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726310" 06:18:30 INFO [http_cmd]: http status code OK 06:18:30 DEBUG --- status code --- 06:18:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:30 DEBUG --- http response --- 06:18:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:18:30 DEBUG ----- output ----- 06:18:30 DEBUG { 06:18:30 DEBUG "status": "success", 06:18:30 DEBUG "data": { 06:18:30 DEBUG "resultType": "vector", 06:18:30 DEBUG "result": [ 06:18:30 DEBUG { 06:18:30 DEBUG "metric": { 06:18:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:18:30 DEBUG }, 06:18:30 DEBUG "value": [ 06:18:30 DEBUG 06:18:30 DEBUG 06:18:30 DEBUG 06:18:30 DEBUG [...] 06:18:30 DEBUG 06:18:30 DEBUG 06:18:30 DEBUG 06:18:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:18:30 DEBUG }, 06:18:30 DEBUG "value": [ 06:18:30 DEBUG 1712726310, 06:18:30 DEBUG "1434.8333333333333" 06:18:30 DEBUG ] 06:18:30 DEBUG } 06:18:30 DEBUG ] 06:18:30 DEBUG } 06:18:30 DEBUG } 06:18:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726311" 06:18:31 INFO [http_cmd]: http status code OK 06:18:31 DEBUG --- status code --- 06:18:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:31 DEBUG --- http response --- 06:18:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:18:31 DEBUG ----- output ----- 06:18:31 DEBUG { 06:18:31 DEBUG "status": "success", 06:18:31 DEBUG "data": { 06:18:31 DEBUG "resultType": "vector", 06:18:31 DEBUG "result": [ 06:18:31 DEBUG { 06:18:31 DEBUG "metric": { 06:18:31 DEBUG "pod": "ds-idrepo-0" 06:18:31 DEBUG }, 06:18:31 DEBUG "value": [ 06:18:31 DEBUG 06:18:31 DEBUG 06:18:31 DEBUG 06:18:31 DEBUG [...] 06:18:31 DEBUG 06:18:31 DEBUG 06:18:31 DEBUG 06:18:31 DEBUG "pod": "ds-idrepo-1" 06:18:31 DEBUG }, 06:18:31 DEBUG "value": [ 06:18:31 DEBUG 1712726311, 06:18:31 DEBUG "0" 06:18:31 DEBUG ] 06:18:31 DEBUG } 06:18:31 DEBUG ] 06:18:31 DEBUG } 06:18:31 DEBUG } 06:18:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726311" 06:18:31 INFO [http_cmd]: http status code OK 06:18:31 DEBUG --- status code --- 06:18:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:31 DEBUG --- http response --- 06:18:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:18:31 DEBUG ----- output ----- 06:18:31 DEBUG { 06:18:31 DEBUG "status": "success", 06:18:31 DEBUG "data": { 06:18:31 DEBUG "resultType": "vector", 06:18:31 DEBUG "result": [ 06:18:31 DEBUG { 06:18:31 DEBUG "metric": { 06:18:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:18:31 DEBUG }, 06:18:31 DEBUG "value": [ 06:18:31 DEBUG 06:18:31 DEBUG 06:18:31 DEBUG 06:18:31 DEBUG [...] 06:18:31 DEBUG 06:18:31 DEBUG 06:18:31 DEBUG 06:18:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:18:31 DEBUG }, 06:18:31 DEBUG "value": [ 06:18:31 DEBUG 1712726311, 06:18:31 DEBUG "0" 06:18:31 DEBUG ] 06:18:31 DEBUG } 06:18:31 DEBUG ] 06:18:31 DEBUG } 06:18:31 DEBUG } 06:18:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712726311" 06:18:31 INFO [http_cmd]: http status code OK 06:18:31 DEBUG --- status code --- 06:18:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:31 DEBUG --- http response --- 06:18:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:18:31 DEBUG ----- output ----- 06:18:31 DEBUG { 06:18:31 DEBUG "status": "success", 06:18:31 DEBUG "data": { 06:18:31 DEBUG "resultType": "vector", 06:18:31 DEBUG "result": [ 06:18:31 DEBUG { 06:18:31 DEBUG "metric": { 06:18:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:18:31 DEBUG }, 06:18:31 DEBUG "value": [ 06:18:31 DEBUG 06:18:31 DEBUG 06:18:31 DEBUG 06:18:31 DEBUG [...] 06:18:31 DEBUG 06:18:31 DEBUG 06:18:31 DEBUG 06:18:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:18:31 DEBUG }, 06:18:31 DEBUG "value": [ 06:18:31 DEBUG 1712726311, 06:18:31 DEBUG "2486476800" 06:18:31 DEBUG ] 06:18:31 DEBUG } 06:18:31 DEBUG ] 06:18:31 DEBUG } 06:18:31 DEBUG } 06:18:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726311" 06:18:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726311" 06:18:31 INFO [http_cmd]: http status code OK 06:18:31 DEBUG --- status code --- 06:18:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:31 DEBUG --- http response --- 06:18:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:18:31 DEBUG ----- output ----- 06:18:31 DEBUG { 06:18:31 DEBUG "status": "success", 06:18:31 DEBUG "data": { 06:18:31 DEBUG "resultType": "vector", 06:18:31 DEBUG "result": [ 06:18:31 DEBUG { 06:18:31 DEBUG "metric": { 06:18:31 DEBUG "instance": "10.142.0.25:9100" 06:18:31 DEBUG }, 06:18:31 DEBUG "value": [ 06:18:31 DEBUG 06:18:31 DEBUG 06:18:31 DEBUG 06:18:31 DEBUG [...] 06:18:31 DEBUG 06:18:31 DEBUG 06:18:31 DEBUG 06:18:31 DEBUG "instance": "10.142.0.77:9100" 06:18:31 DEBUG }, 06:18:31 DEBUG "value": [ 06:18:31 DEBUG 1712726311, 06:18:31 DEBUG "42470795.06666666" 06:18:31 DEBUG ] 06:18:31 DEBUG } 06:18:31 DEBUG ] 06:18:31 DEBUG } 06:18:31 DEBUG } 06:18:31 INFO [http_cmd]: http status code OK 06:18:31 DEBUG --- status code --- 06:18:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:18:31 DEBUG --- http response --- 06:18:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:18:31 DEBUG ----- output ----- 06:18:31 DEBUG { 06:18:31 DEBUG "status": "success", 06:18:31 DEBUG "data": { 06:18:31 DEBUG "resultType": "vector", 06:18:31 DEBUG "result": [ 06:18:31 DEBUG { 06:18:31 DEBUG "metric": { 06:18:31 DEBUG "instance": "10.142.0.25:9100" 06:18:31 DEBUG }, 06:18:31 DEBUG "value": [ 06:18:31 DEBUG 06:18:31 DEBUG 06:18:31 DEBUG 06:18:31 DEBUG [...] 06:18:31 DEBUG 06:18:31 DEBUG 06:18:31 DEBUG 06:18:31 DEBUG "instance": "10.142.0.77:9100" 06:18:31 DEBUG }, 06:18:31 DEBUG "value": [ 06:18:31 DEBUG 1712726311, 06:18:31 DEBUG "0.02333333333333414" 06:18:31 DEBUG ] 06:18:31 DEBUG } 06:18:31 DEBUG ] 06:18:31 DEBUG } 06:18:31 DEBUG } 06:18:49 INFO 06:18:49 INFO [loop_until]: kubectl --namespace=xlou top pods 06:18:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:18:49 INFO [loop_until]: OK (rc = 0) 06:18:49 DEBUG --- stdout --- 06:18:49 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 11133m 8979Mi am-654c4fcbcf-9scrj 10075m 8448Mi am-654c4fcbcf-v5qb5 8373m 8445Mi ds-cts-0 8535m 13393Mi ds-cts-1 8530m 13204Mi ds-cts-2 7557m 13331Mi ds-idrepo-0 10694m 5622Mi ds-idrepo-1 382m 5672Mi ds-idrepo-2 395m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 5m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4220m 2292Mi 06:18:49 DEBUG --- stderr --- 06:18:49 DEBUG 06:18:57 INFO 06:18:57 INFO [loop_until]: kubectl --namespace=xlou top node 06:18:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:18:57 INFO [loop_until]: OK (rc = 0) 06:18:57 DEBUG --- stdout --- 06:18:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 97m 0% 2305Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8973m 56% 9318Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 82m 0% 3258Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13583m 85% 6665Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10899m 68% 9329Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 12014m 75% 9796Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 75m 0% 854Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10859m 68% 6300Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8618m 54% 14071Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 7840m 49% 14017Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 450m 2% 6385Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8545m 53% 13729Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 418m 2% 6355Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4387m 27% 5908Mi 10% 06:18:57 DEBUG --- stderr --- 06:18:57 DEBUG 06:19:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712726368" 06:19:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726368" 06:19:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726368" 06:19:28 INFO [http_cmd]: http status code OK 06:19:28 DEBUG --- status code --- 06:19:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:28 DEBUG --- http response --- 06:19:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:19:28 DEBUG ----- output ----- 06:19:28 DEBUG { 06:19:28 DEBUG "status": "success", 06:19:28 DEBUG "data": { 06:19:28 DEBUG "resultType": "vector", 06:19:28 DEBUG "result": [ 06:19:28 DEBUG { 06:19:28 DEBUG "metric": { 06:19:28 ERROR [http_cmd]: ERROR 06:19:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:19:28 ERROR --- status code --- 06:19:28 DEBUG }, 06:19:28 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:19:28 INFO [http_cmd]: http status code OK 06:19:28 DEBUG "value": [ 06:19:28 ERROR --- http response --- 06:19:28 DEBUG --- status code --- 06:19:28 DEBUG 06:19:28 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:19:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:28 DEBUG 06:19:28 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:19:28 DEBUG --- http response --- 06:19:28 DEBUG 06:19:28 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:19:28 DEBUG [...] 06:19:28 DEBUG 06:19:28 DEBUG 06:19:28 DEBUG 06:19:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:19:28 DEBUG }, 06:19:28 DEBUG "value": [ 06:19:28 DEBUG 1712726368, 06:19:28 DEBUG "0" 06:19:28 DEBUG ] 06:19:28 DEBUG } 06:19:28 DEBUG ] 06:19:28 DEBUG } 06:19:28 DEBUG } 06:19:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726368" 06:19:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726368" 06:19:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726369" 06:19:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726369" 06:19:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726369" 06:19:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726369" 06:19:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726369" 06:19:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726369" 06:19:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726369" 06:19:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726369" 06:19:29 INFO [http_cmd]: http status code OK 06:19:29 DEBUG --- status code --- 06:19:29 INFO [http_cmd]: http status code OK 06:19:29 INFO [http_cmd]: http status code OK 06:19:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:29 DEBUG --- status code --- 06:19:29 INFO [http_cmd]: http status code OK 06:19:29 DEBUG --- status code --- 06:19:29 INFO [http_cmd]: http status code OK 06:19:29 INFO [http_cmd]: http status code OK 06:19:29 INFO [http_cmd]: http status code OK 06:19:29 INFO [http_cmd]: http status code OK 06:19:29 DEBUG --- status code --- 06:19:29 DEBUG --- http response --- 06:19:29 INFO [http_cmd]: http status code OK 06:19:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:29 DEBUG --- status code --- 06:19:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:29 DEBUG --- status code --- 06:19:29 DEBUG --- status code --- 06:19:29 DEBUG --- status code --- 06:19:29 INFO [http_cmd]: http status code OK 06:19:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:19:29 DEBUG --- status code --- 06:19:29 DEBUG --- http response --- 06:19:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:29 DEBUG --- http response --- 06:19:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:29 DEBUG --- status code --- 06:19:29 DEBUG --- http response --- 06:19:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:19:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:29 DEBUG --- http response --- 06:19:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:19:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:19:29 DEBUG ----- output ----- 06:19:29 DEBUG --- http response --- 06:19:29 DEBUG --- http response --- 06:19:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:29 DEBUG ----- output ----- 06:19:29 DEBUG ----- output ----- 06:19:29 DEBUG --- http response --- 06:19:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:19:29 DEBUG ----- output ----- 06:19:29 DEBUG { 06:19:29 DEBUG { 06:19:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:19:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:19:29 DEBUG --- http response --- 06:19:29 DEBUG { 06:19:29 DEBUG { 06:19:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:19:29 DEBUG ----- output ----- 06:19:29 DEBUG { 06:19:29 DEBUG "status": "success", 06:19:29 DEBUG "status": "success", 06:19:29 DEBUG ----- output ----- 06:19:29 DEBUG ----- output ----- 06:19:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:19:29 DEBUG "status": "success", 06:19:29 DEBUG "status": "success", 06:19:29 DEBUG ----- output ----- 06:19:29 DEBUG --- http response --- 06:19:29 DEBUG "status": "success", 06:19:29 DEBUG "data": { 06:19:29 DEBUG "data": { 06:19:29 DEBUG { 06:19:29 DEBUG { 06:19:29 DEBUG "status": "success", 06:19:29 DEBUG "data": { 06:19:29 DEBUG "resultType": "vector", 06:19:29 DEBUG "result": [ 06:19:29 DEBUG { 06:19:29 DEBUG "data": { 06:19:29 DEBUG "resultType": "vector", 06:19:29 DEBUG "resultType": "vector", 06:19:29 DEBUG "result": [ 06:19:29 DEBUG "data": { 06:19:29 DEBUG { 06:19:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:19:29 DEBUG "metric": { 06:19:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:19:29 DEBUG }, 06:19:29 DEBUG "resultType": "vector", 06:19:29 DEBUG "result": [ 06:19:29 DEBUG { 06:19:29 DEBUG { 06:19:29 DEBUG "resultType": "vector", 06:19:29 DEBUG "status": "success", 06:19:29 DEBUG ----- output ----- 06:19:29 DEBUG "value": [ 06:19:29 DEBUG ----- output ----- 06:19:29 DEBUG "data": { 06:19:29 DEBUG "result": [ 06:19:29 DEBUG "metric": { 06:19:29 DEBUG "status": "success", 06:19:29 DEBUG "metric": { 06:19:29 DEBUG "result": [ 06:19:29 DEBUG "data": { 06:19:29 DEBUG { 06:19:29 DEBUG 06:19:29 DEBUG { 06:19:29 DEBUG "resultType": "vector", 06:19:29 DEBUG { 06:19:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:19:29 DEBUG "data": { 06:19:29 DEBUG "instance": "10.142.0.25:9100" 06:19:29 DEBUG { 06:19:29 DEBUG "resultType": "vector", 06:19:29 DEBUG "status": "success", 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG "result": [ 06:19:29 DEBUG "metric": { 06:19:29 DEBUG }, 06:19:29 DEBUG "resultType": "vector", 06:19:29 DEBUG "result": [ 06:19:29 DEBUG "metric": { 06:19:29 DEBUG "result": [ 06:19:29 DEBUG "data": { 06:19:29 DEBUG [...] 06:19:29 DEBUG "status": "success", 06:19:29 DEBUG { 06:19:29 DEBUG "instance": "10.142.0.25:9100" 06:19:29 DEBUG "value": [ 06:19:29 DEBUG { 06:19:29 DEBUG }, 06:19:29 DEBUG "pod": "ds-idrepo-0" 06:19:29 DEBUG { 06:19:29 DEBUG "resultType": "vector", 06:19:29 DEBUG 06:19:29 DEBUG "data": { 06:19:29 DEBUG "metric": { 06:19:29 DEBUG }, 06:19:29 DEBUG 06:19:29 DEBUG "metric": { 06:19:29 DEBUG "value": [ 06:19:29 DEBUG }, 06:19:29 DEBUG "metric": { 06:19:29 DEBUG "result": [ 06:19:29 DEBUG 06:19:29 DEBUG "resultType": "vector", 06:19:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:19:29 DEBUG "value": [ 06:19:29 DEBUG 06:19:29 DEBUG "instance": "10.142.0.25:9100" 06:19:29 DEBUG 06:19:29 DEBUG "value": [ 06:19:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:19:29 DEBUG { 06:19:29 DEBUG 06:19:29 DEBUG "result": [ 06:19:29 DEBUG }, 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG }, 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG }, 06:19:29 DEBUG "metric": { 06:19:29 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:19:29 DEBUG { 06:19:29 DEBUG "value": [ 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG "value": [ 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG "value": [ 06:19:29 DEBUG "pod": "ds-idrepo-0" 06:19:29 DEBUG }, 06:19:29 DEBUG "metric": { 06:19:29 DEBUG 06:19:29 DEBUG [...] 06:19:29 DEBUG [...] 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG [...] 06:19:29 DEBUG }, 06:19:29 DEBUG "value": [ 06:19:29 DEBUG "pod": "ds-idrepo-0" 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG [...] 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG "value": [ 06:19:29 DEBUG 1712726369, 06:19:29 DEBUG "0" 06:19:29 DEBUG ] 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG [...] 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG }, 06:19:29 DEBUG "value": [ 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG [...] 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG "instance": "10.142.0.77:9100" 06:19:29 DEBUG }, 06:19:29 DEBUG }, 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG [...] 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG "value": [ 06:19:29 DEBUG "value": [ 06:19:29 DEBUG 1712726368, 06:19:29 DEBUG "1444.8" 06:19:29 DEBUG ] 06:19:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:19:29 DEBUG }, 06:19:29 DEBUG "value": [ 06:19:29 DEBUG 1712726369, 06:19:29 DEBUG "1445.2333333333333" 06:19:29 DEBUG 1712726369, 06:19:29 DEBUG "instance": "10.142.0.77:9100" 06:19:29 DEBUG }, 06:19:29 DEBUG "value": [ 06:19:29 DEBUG 1712726369, 06:19:29 DEBUG "pod": "ds-idrepo-1" 06:19:29 DEBUG ] 06:19:29 DEBUG } 06:19:29 DEBUG ] 06:19:29 DEBUG } 06:19:29 DEBUG "42641559" 06:19:29 DEBUG 06:19:29 DEBUG [...] 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG }, 06:19:29 DEBUG 06:19:29 DEBUG } 06:19:29 DEBUG } 06:19:29 DEBUG ] 06:19:29 DEBUG } 06:19:29 DEBUG } 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG "instance": "10.142.0.77:9100" 06:19:29 DEBUG "value": [ 06:19:29 DEBUG "pod": "ds-cts-2" 06:19:29 DEBUG 06:19:29 DEBUG } 06:19:29 DEBUG ] 06:19:29 DEBUG 06:19:29 DEBUG "pod": "ds-cts-2" 06:19:29 DEBUG }, 06:19:29 DEBUG 1712726369, 06:19:29 DEBUG }, 06:19:29 DEBUG [...] 06:19:29 DEBUG } 06:19:29 DEBUG } 06:19:29 DEBUG "value": [ 06:19:29 DEBUG ] 06:19:29 DEBUG "379787226112" 06:19:29 DEBUG "0" 06:19:29 DEBUG ] 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG } 06:19:29 DEBUG ] 06:19:29 DEBUG } 06:19:29 DEBUG "value": [ 06:19:29 DEBUG 1712726369, 06:19:29 DEBUG "0" 06:19:29 DEBUG ] 06:19:29 DEBUG } 06:19:29 DEBUG ] 06:19:29 DEBUG } 06:19:29 DEBUG } 06:19:29 DEBUG 1712726369, 06:19:29 DEBUG } 06:19:29 DEBUG ] 06:19:29 DEBUG } 06:19:29 DEBUG } 06:19:29 DEBUG "29.930340299303403" 06:19:29 DEBUG ] 06:19:29 DEBUG } 06:19:29 DEBUG ] 06:19:29 DEBUG ] 06:19:29 DEBUG } 06:19:29 DEBUG } 06:19:29 DEBUG ] 06:19:29 DEBUG } 06:19:29 DEBUG } 06:19:29 DEBUG }, 06:19:29 DEBUG "value": [ 06:19:29 DEBUG 1712726369, 06:19:29 DEBUG "0.15640000000000212" 06:19:29 DEBUG } 06:19:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:19:29 DEBUG }, 06:19:29 DEBUG } 06:19:29 DEBUG "value": [ 06:19:29 DEBUG ] 06:19:29 DEBUG 1712726368, 06:19:29 DEBUG "259697.5" 06:19:29 DEBUG ] 06:19:29 DEBUG } 06:19:29 DEBUG ] 06:19:29 DEBUG } 06:19:29 DEBUG } 06:19:29 DEBUG } 06:19:29 DEBUG ] 06:19:29 DEBUG } 06:19:29 DEBUG } 06:19:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726369" 06:19:29 INFO [http_cmd]: http status code OK 06:19:29 DEBUG --- status code --- 06:19:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:29 DEBUG --- http response --- 06:19:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:19:29 DEBUG ----- output ----- 06:19:29 DEBUG { 06:19:29 DEBUG "status": "success", 06:19:29 DEBUG "data": { 06:19:29 DEBUG "resultType": "vector", 06:19:29 DEBUG "result": [ 06:19:29 DEBUG { 06:19:29 DEBUG "metric": { 06:19:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:19:29 DEBUG }, 06:19:29 DEBUG "value": [ 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG [...] 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG 06:19:29 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:19:29 DEBUG }, 06:19:29 DEBUG "value": [ 06:19:29 DEBUG 1712726369, 06:19:29 DEBUG "104094.65962091954" 06:19:29 DEBUG ] 06:19:29 DEBUG } 06:19:29 DEBUG ] 06:19:29 DEBUG } 06:19:29 DEBUG } 06:19:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712726370" 06:19:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712726370" 06:19:30 INFO [http_cmd]: http status code OK 06:19:30 DEBUG --- status code --- 06:19:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:30 DEBUG --- http response --- 06:19:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:19:30 DEBUG ----- output ----- 06:19:30 DEBUG { 06:19:30 DEBUG "status": "success", 06:19:30 DEBUG "data": { 06:19:30 DEBUG "resultType": "vector", 06:19:30 DEBUG "result": [ 06:19:30 DEBUG { 06:19:30 DEBUG "metric": { 06:19:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:19:30 DEBUG }, 06:19:30 DEBUG "value": [ 06:19:30 DEBUG 06:19:30 DEBUG 06:19:30 DEBUG 06:19:30 DEBUG [...] 06:19:30 DEBUG 06:19:30 DEBUG 06:19:30 DEBUG 06:19:30 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:19:30 DEBUG }, 06:19:30 DEBUG "value": [ 06:19:30 DEBUG 1712726370, 06:19:30 DEBUG "4.23764113970868" 06:19:30 DEBUG ] 06:19:30 DEBUG } 06:19:30 DEBUG ] 06:19:30 DEBUG } 06:19:30 DEBUG } 06:19:30 INFO [http_cmd]: http status code OK 06:19:30 DEBUG --- status code --- 06:19:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:30 DEBUG --- http response --- 06:19:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:19:30 DEBUG ----- output ----- 06:19:30 DEBUG { 06:19:30 DEBUG "status": "success", 06:19:30 DEBUG "data": { 06:19:30 DEBUG "resultType": "vector", 06:19:30 DEBUG "result": [ 06:19:30 DEBUG { 06:19:30 DEBUG "metric": { 06:19:30 DEBUG "pod": "ds-idrepo-0" 06:19:30 DEBUG }, 06:19:30 DEBUG "value": [ 06:19:30 DEBUG 06:19:30 DEBUG 06:19:30 DEBUG 06:19:30 DEBUG [...] 06:19:30 DEBUG 06:19:30 DEBUG 06:19:30 DEBUG 06:19:30 DEBUG "pod": "ds-cts-2" 06:19:30 DEBUG }, 06:19:30 DEBUG "value": [ 06:19:30 DEBUG 1712726370, 06:19:30 DEBUG "0" 06:19:30 DEBUG ] 06:19:30 DEBUG } 06:19:30 DEBUG ] 06:19:30 DEBUG } 06:19:30 DEBUG } 06:19:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726370" 06:19:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726370" 06:19:30 INFO [http_cmd]: http status code OK 06:19:30 DEBUG --- status code --- 06:19:30 INFO [http_cmd]: http status code OK 06:19:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:30 DEBUG --- status code --- 06:19:30 DEBUG --- http response --- 06:19:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:19:30 DEBUG --- http response --- 06:19:30 DEBUG ----- output ----- 06:19:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:19:30 DEBUG { 06:19:30 DEBUG ----- output ----- 06:19:30 DEBUG "status": "success", 06:19:30 DEBUG "data": { 06:19:30 DEBUG { 06:19:30 DEBUG "resultType": "vector", 06:19:30 DEBUG "status": "success", 06:19:30 DEBUG "result": [ 06:19:30 DEBUG "data": { 06:19:30 DEBUG { 06:19:30 DEBUG "resultType": "vector", 06:19:30 DEBUG "metric": { 06:19:30 DEBUG "result": [ 06:19:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:19:30 DEBUG { 06:19:30 DEBUG }, 06:19:30 DEBUG "metric": { 06:19:30 DEBUG "value": [ 06:19:30 DEBUG "pod": "ds-idrepo-2" 06:19:30 DEBUG 06:19:30 DEBUG }, 06:19:30 DEBUG 06:19:30 DEBUG "value": [ 06:19:30 DEBUG 06:19:30 DEBUG 06:19:30 DEBUG [...] 06:19:30 DEBUG 06:19:30 DEBUG 06:19:30 DEBUG 06:19:30 DEBUG 06:19:30 DEBUG [...] 06:19:30 DEBUG 06:19:30 DEBUG 06:19:30 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:19:30 DEBUG 06:19:30 DEBUG }, 06:19:30 DEBUG 06:19:30 DEBUG "value": [ 06:19:30 DEBUG "pod": "ds-cts-2" 06:19:30 DEBUG 1712726370, 06:19:30 DEBUG }, 06:19:30 DEBUG "4.5289525730597475" 06:19:30 DEBUG "value": [ 06:19:30 DEBUG ] 06:19:30 DEBUG 1712726370, 06:19:30 DEBUG } 06:19:30 DEBUG "0.00003333000033329998" 06:19:30 DEBUG ] 06:19:30 DEBUG ] 06:19:30 DEBUG } 06:19:30 DEBUG } 06:19:30 DEBUG } 06:19:30 DEBUG ] 06:19:30 DEBUG } 06:19:30 DEBUG } 06:19:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726370" 06:19:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712726370" 06:19:30 INFO [http_cmd]: http status code OK 06:19:30 DEBUG --- status code --- 06:19:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:30 DEBUG --- http response --- 06:19:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:19:30 DEBUG ----- output ----- 06:19:30 DEBUG { 06:19:30 DEBUG "status": "success", 06:19:30 DEBUG "data": { 06:19:30 DEBUG "resultType": "vector", 06:19:30 DEBUG "result": [ 06:19:30 DEBUG { 06:19:30 DEBUG "metric": { 06:19:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:19:30 DEBUG }, 06:19:30 DEBUG "value": [ 06:19:30 DEBUG 06:19:30 DEBUG 06:19:30 DEBUG 06:19:30 DEBUG [...] 06:19:30 DEBUG 06:19:30 DEBUG 06:19:30 DEBUG 06:19:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:19:30 DEBUG }, 06:19:30 DEBUG "value": [ 06:19:30 DEBUG 1712726370, 06:19:30 DEBUG "1445.5666666666666" 06:19:30 DEBUG ] 06:19:30 DEBUG } 06:19:30 DEBUG ] 06:19:30 DEBUG } 06:19:30 DEBUG } 06:19:30 INFO [http_cmd]: http status code OK 06:19:30 DEBUG --- status code --- 06:19:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:30 DEBUG --- http response --- 06:19:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:19:30 DEBUG ----- output ----- 06:19:30 DEBUG { 06:19:30 DEBUG "status": "success", 06:19:30 DEBUG "data": { 06:19:30 DEBUG "resultType": "vector", 06:19:30 DEBUG "result": [ 06:19:30 DEBUG { 06:19:30 DEBUG "metric": { 06:19:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:19:30 DEBUG }, 06:19:30 DEBUG "value": [ 06:19:30 DEBUG 06:19:30 DEBUG 06:19:30 DEBUG 06:19:30 DEBUG [...] 06:19:30 DEBUG 06:19:30 DEBUG 06:19:30 DEBUG 06:19:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:19:30 DEBUG }, 06:19:30 DEBUG "value": [ 06:19:30 DEBUG 1712726370, 06:19:30 DEBUG "0.05" 06:19:30 DEBUG ] 06:19:30 DEBUG } 06:19:30 DEBUG ] 06:19:30 DEBUG } 06:19:30 DEBUG } 06:19:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726370" 06:19:30 INFO [http_cmd]: http status code OK 06:19:30 DEBUG --- status code --- 06:19:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:30 DEBUG --- http response --- 06:19:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:19:30 DEBUG ----- output ----- 06:19:30 DEBUG { 06:19:30 DEBUG "status": "success", 06:19:30 DEBUG "data": { 06:19:30 DEBUG "resultType": "vector", 06:19:30 DEBUG "result": [ 06:19:30 DEBUG { 06:19:30 DEBUG "metric": { 06:19:30 DEBUG "pod": "ds-idrepo-0" 06:19:30 DEBUG }, 06:19:30 DEBUG "value": [ 06:19:30 DEBUG 06:19:30 DEBUG 06:19:30 DEBUG 06:19:30 DEBUG [...] 06:19:30 DEBUG 06:19:30 DEBUG 06:19:30 DEBUG 06:19:30 DEBUG "pod": "ds-idrepo-1" 06:19:30 DEBUG }, 06:19:30 DEBUG "value": [ 06:19:30 DEBUG 1712726370, 06:19:30 DEBUG "0" 06:19:30 DEBUG ] 06:19:30 DEBUG } 06:19:30 DEBUG ] 06:19:30 DEBUG } 06:19:30 DEBUG } 06:19:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726371" 06:19:31 INFO [http_cmd]: http status code OK 06:19:31 DEBUG --- status code --- 06:19:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:31 DEBUG --- http response --- 06:19:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:19:31 DEBUG ----- output ----- 06:19:31 DEBUG { 06:19:31 DEBUG "status": "success", 06:19:31 DEBUG "data": { 06:19:31 DEBUG "resultType": "vector", 06:19:31 DEBUG "result": [ 06:19:31 DEBUG { 06:19:31 DEBUG "metric": { 06:19:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:19:31 DEBUG }, 06:19:31 DEBUG "value": [ 06:19:31 DEBUG 06:19:31 DEBUG 06:19:31 DEBUG 06:19:31 DEBUG [...] 06:19:31 DEBUG 06:19:31 DEBUG 06:19:31 DEBUG 06:19:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:19:31 DEBUG }, 06:19:31 DEBUG "value": [ 06:19:31 DEBUG 1712726371, 06:19:31 DEBUG "1438.2666666666667" 06:19:31 DEBUG ] 06:19:31 DEBUG } 06:19:31 DEBUG ] 06:19:31 DEBUG } 06:19:31 DEBUG } 06:19:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726371" 06:19:31 INFO [http_cmd]: http status code OK 06:19:31 DEBUG --- status code --- 06:19:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:31 DEBUG --- http response --- 06:19:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:19:31 DEBUG ----- output ----- 06:19:31 DEBUG { 06:19:31 DEBUG "status": "success", 06:19:31 DEBUG "data": { 06:19:31 DEBUG "resultType": "vector", 06:19:31 DEBUG "result": [ 06:19:31 DEBUG { 06:19:31 DEBUG "metric": { 06:19:31 DEBUG "pod": "ds-idrepo-0" 06:19:31 DEBUG }, 06:19:31 DEBUG "value": [ 06:19:31 DEBUG 06:19:31 DEBUG 06:19:31 DEBUG 06:19:31 DEBUG [...] 06:19:31 DEBUG 06:19:31 DEBUG 06:19:31 DEBUG 06:19:31 DEBUG "pod": "ds-idrepo-1" 06:19:31 DEBUG }, 06:19:31 DEBUG "value": [ 06:19:31 DEBUG 1712726371, 06:19:31 DEBUG "0" 06:19:31 DEBUG ] 06:19:31 DEBUG } 06:19:31 DEBUG ] 06:19:31 DEBUG } 06:19:31 DEBUG } 06:19:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726371" 06:19:31 INFO [http_cmd]: http status code OK 06:19:31 DEBUG --- status code --- 06:19:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:31 DEBUG --- http response --- 06:19:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:19:31 DEBUG ----- output ----- 06:19:31 DEBUG { 06:19:31 DEBUG "status": "success", 06:19:31 DEBUG "data": { 06:19:31 DEBUG "resultType": "vector", 06:19:31 DEBUG "result": [ 06:19:31 DEBUG { 06:19:31 DEBUG "metric": { 06:19:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:19:31 DEBUG }, 06:19:31 DEBUG "value": [ 06:19:31 DEBUG 06:19:31 DEBUG 06:19:31 DEBUG 06:19:31 DEBUG [...] 06:19:31 DEBUG 06:19:31 DEBUG 06:19:31 DEBUG 06:19:31 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:19:31 DEBUG }, 06:19:31 DEBUG "value": [ 06:19:31 DEBUG 1712726371, 06:19:31 DEBUG "0" 06:19:31 DEBUG ] 06:19:31 DEBUG } 06:19:31 DEBUG ] 06:19:31 DEBUG } 06:19:31 DEBUG } 06:19:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712726371" 06:19:31 INFO [http_cmd]: http status code OK 06:19:31 DEBUG --- status code --- 06:19:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:31 DEBUG --- http response --- 06:19:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:19:31 DEBUG ----- output ----- 06:19:31 DEBUG { 06:19:31 DEBUG "status": "success", 06:19:31 DEBUG "data": { 06:19:31 DEBUG "resultType": "vector", 06:19:31 DEBUG "result": [ 06:19:31 DEBUG { 06:19:31 DEBUG "metric": { 06:19:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:19:31 DEBUG }, 06:19:31 DEBUG "value": [ 06:19:31 DEBUG 06:19:31 DEBUG 06:19:31 DEBUG 06:19:31 DEBUG [...] 06:19:31 DEBUG 06:19:31 DEBUG 06:19:31 DEBUG 06:19:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:19:31 DEBUG }, 06:19:31 DEBUG "value": [ 06:19:31 DEBUG 1712726371, 06:19:31 DEBUG "2485944320" 06:19:31 DEBUG ] 06:19:31 DEBUG } 06:19:31 DEBUG ] 06:19:31 DEBUG } 06:19:31 DEBUG } 06:19:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726371" 06:19:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726371" 06:19:31 INFO [http_cmd]: http status code OK 06:19:31 DEBUG --- status code --- 06:19:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:31 DEBUG --- http response --- 06:19:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:19:31 DEBUG ----- output ----- 06:19:31 DEBUG { 06:19:31 DEBUG "status": "success", 06:19:31 DEBUG "data": { 06:19:31 DEBUG "resultType": "vector", 06:19:31 DEBUG "result": [ 06:19:31 DEBUG { 06:19:31 DEBUG "metric": { 06:19:31 DEBUG "instance": "10.142.0.25:9100" 06:19:31 DEBUG }, 06:19:31 DEBUG "value": [ 06:19:31 DEBUG 06:19:31 DEBUG 06:19:31 DEBUG 06:19:31 DEBUG [...] 06:19:31 DEBUG 06:19:31 DEBUG 06:19:31 DEBUG 06:19:31 DEBUG "instance": "10.142.0.77:9100" 06:19:31 DEBUG }, 06:19:31 DEBUG "value": [ 06:19:31 DEBUG 1712726371, 06:19:31 DEBUG "42645620.233333334" 06:19:31 DEBUG ] 06:19:31 DEBUG } 06:19:31 DEBUG ] 06:19:31 DEBUG } 06:19:31 DEBUG } 06:19:31 INFO [http_cmd]: http status code OK 06:19:31 DEBUG --- status code --- 06:19:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:19:31 DEBUG --- http response --- 06:19:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:19:31 DEBUG ----- output ----- 06:19:31 DEBUG { 06:19:31 DEBUG "status": "success", 06:19:31 DEBUG "data": { 06:19:31 DEBUG "resultType": "vector", 06:19:31 DEBUG "result": [ 06:19:31 DEBUG { 06:19:31 DEBUG "metric": { 06:19:31 DEBUG "instance": "10.142.0.25:9100" 06:19:31 DEBUG }, 06:19:31 DEBUG "value": [ 06:19:31 DEBUG 06:19:31 DEBUG 06:19:31 DEBUG 06:19:31 DEBUG [...] 06:19:31 DEBUG 06:19:31 DEBUG 06:19:31 DEBUG 06:19:31 DEBUG "instance": "10.142.0.77:9100" 06:19:31 DEBUG }, 06:19:31 DEBUG "value": [ 06:19:31 DEBUG 1712726371, 06:19:31 DEBUG "0.04600000000000044" 06:19:31 DEBUG ] 06:19:31 DEBUG } 06:19:31 DEBUG ] 06:19:31 DEBUG } 06:19:31 DEBUG } 06:19:49 INFO 06:19:49 INFO [loop_until]: kubectl --namespace=xlou top pods 06:19:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:19:49 INFO [loop_until]: OK (rc = 0) 06:19:49 DEBUG --- stdout --- 06:19:49 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 11001m 9008Mi am-654c4fcbcf-9scrj 10133m 8448Mi am-654c4fcbcf-v5qb5 8586m 8443Mi ds-cts-0 8175m 13391Mi ds-cts-1 8603m 12584Mi ds-cts-2 7660m 12741Mi ds-idrepo-0 10836m 5623Mi ds-idrepo-1 397m 5672Mi ds-idrepo-2 383m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 5m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4246m 2294Mi 06:19:49 DEBUG --- stderr --- 06:19:49 DEBUG 06:19:57 INFO 06:19:57 INFO [loop_until]: kubectl --namespace=xlou top node 06:19:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:19:57 INFO [loop_until]: OK (rc = 0) 06:19:57 DEBUG --- stdout --- 06:19:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 98m 0% 2305Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9294m 58% 9319Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 79m 0% 3253Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13655m 85% 6671Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 11033m 69% 9330Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11995m 75% 9812Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 74m 0% 854Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10835m 68% 6301Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8330m 52% 14081Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 7746m 48% 13405Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 456m 2% 6387Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8659m 54% 13299Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 464m 2% 6351Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4340m 27% 5906Mi 10% 06:19:57 DEBUG --- stderr --- 06:19:57 DEBUG 06:20:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712726428" 06:20:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726428" 06:20:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726428" 06:20:28 INFO [http_cmd]: http status code OK 06:20:28 DEBUG --- status code --- 06:20:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:28 DEBUG --- http response --- 06:20:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:20:28 DEBUG ----- output ----- 06:20:28 DEBUG { 06:20:28 DEBUG "status": "success", 06:20:28 DEBUG "data": { 06:20:28 DEBUG "resultType": "vector", 06:20:28 ERROR [http_cmd]: ERROR 06:20:28 DEBUG "result": [ 06:20:28 INFO [http_cmd]: http status code OK 06:20:28 DEBUG --- status code --- 06:20:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:28 DEBUG { 06:20:28 DEBUG --- http response --- 06:20:28 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:20:28 DEBUG "metric": { 06:20:28 ERROR --- status code --- 06:20:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:20:28 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:20:28 DEBUG }, 06:20:28 ERROR --- http response --- 06:20:28 DEBUG "value": [ 06:20:28 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:20:28 DEBUG 06:20:28 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:20:28 DEBUG 06:20:28 DEBUG 06:20:28 DEBUG [...] 06:20:28 DEBUG 06:20:28 DEBUG 06:20:28 DEBUG 06:20:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:20:28 DEBUG }, 06:20:28 DEBUG "value": [ 06:20:28 DEBUG 1712726428, 06:20:28 DEBUG "0" 06:20:28 DEBUG ] 06:20:28 DEBUG } 06:20:28 DEBUG ] 06:20:28 DEBUG } 06:20:28 DEBUG } 06:20:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726429" 06:20:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726429" 06:20:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726429" 06:20:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726429" 06:20:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726429" 06:20:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726429" 06:20:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726429" 06:20:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726429" 06:20:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726429" 06:20:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726429" 06:20:29 INFO [http_cmd]: http status code OK 06:20:29 INFO [http_cmd]: http status code OK 06:20:29 INFO [http_cmd]: http status code OK 06:20:29 INFO [http_cmd]: http status code OK 06:20:29 DEBUG --- status code --- 06:20:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:29 DEBUG --- http response --- 06:20:29 INFO [http_cmd]: http status code OK 06:20:29 DEBUG --- status code --- 06:20:29 DEBUG --- status code --- 06:20:29 DEBUG --- status code --- 06:20:29 INFO [http_cmd]: http status code OK 06:20:29 INFO [http_cmd]: http status code OK 06:20:29 DEBUG --- status code --- 06:20:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:20:29 DEBUG --- status code --- 06:20:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:29 DEBUG --- http response --- 06:20:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:29 INFO [http_cmd]: http status code OK 06:20:29 DEBUG --- status code --- 06:20:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:29 INFO [http_cmd]: http status code OK 06:20:29 DEBUG ----- output ----- 06:20:29 INFO [http_cmd]: http status code OK 06:20:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:20:29 DEBUG --- http response --- 06:20:29 DEBUG --- http response --- 06:20:29 DEBUG --- status code --- 06:20:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:29 DEBUG --- http response --- 06:20:29 DEBUG --- status code --- 06:20:29 DEBUG { 06:20:29 DEBUG --- status code --- 06:20:29 DEBUG --- http response --- 06:20:29 DEBUG ----- output ----- 06:20:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:20:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:20:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:29 DEBUG --- http response --- 06:20:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:20:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:29 DEBUG "status": "success", 06:20:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:20:29 DEBUG { 06:20:29 DEBUG ----- output ----- 06:20:29 DEBUG ----- output ----- 06:20:29 DEBUG { 06:20:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:20:29 DEBUG ----- output ----- 06:20:29 DEBUG --- http response --- 06:20:29 DEBUG "data": { 06:20:29 DEBUG --- http response --- 06:20:29 DEBUG ----- output ----- 06:20:29 DEBUG "status": "success", 06:20:29 DEBUG { 06:20:29 DEBUG --- http response --- 06:20:29 DEBUG "status": "success", 06:20:29 DEBUG { 06:20:29 DEBUG "status": "success", 06:20:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:20:29 DEBUG "resultType": "vector", 06:20:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:20:29 DEBUG { 06:20:29 DEBUG "data": { 06:20:29 DEBUG "status": "success", 06:20:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:20:29 DEBUG "data": { 06:20:29 DEBUG "resultType": "vector", 06:20:29 DEBUG "result": [ 06:20:29 DEBUG ----- output ----- 06:20:29 DEBUG { 06:20:29 DEBUG ----- output ----- 06:20:29 DEBUG "status": "success", 06:20:29 DEBUG "resultType": "vector", 06:20:29 DEBUG "data": { 06:20:29 DEBUG ----- output ----- 06:20:29 DEBUG ----- output ----- 06:20:29 DEBUG { 06:20:29 DEBUG "data": { 06:20:29 DEBUG "resultType": "vector", 06:20:29 DEBUG "status": "success", 06:20:29 DEBUG { 06:20:29 DEBUG "data": { 06:20:29 DEBUG "result": [ 06:20:29 DEBUG "resultType": "vector", 06:20:29 DEBUG { 06:20:29 DEBUG "status": "success", 06:20:29 DEBUG "status": "success", 06:20:29 DEBUG "result": [ 06:20:29 DEBUG { 06:20:29 DEBUG "data": { 06:20:29 DEBUG "status": "success", 06:20:29 DEBUG "resultType": "vector", 06:20:29 DEBUG { 06:20:29 DEBUG "result": [ 06:20:29 DEBUG { 06:20:29 DEBUG "data": { 06:20:29 DEBUG "data": { 06:20:29 DEBUG "metric": { 06:20:29 DEBUG "pod": "ds-idrepo-0" 06:20:29 DEBUG "resultType": "vector", 06:20:29 DEBUG "data": { 06:20:29 DEBUG "resultType": "vector", 06:20:29 DEBUG "metric": { 06:20:29 DEBUG { 06:20:29 DEBUG "metric": { 06:20:29 DEBUG "resultType": "vector", 06:20:29 DEBUG "resultType": "vector", 06:20:29 DEBUG }, 06:20:29 DEBUG "value": [ 06:20:29 DEBUG "result": [ 06:20:29 DEBUG "result": [ 06:20:29 DEBUG "result": [ 06:20:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:20:29 DEBUG "metric": { 06:20:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:20:29 DEBUG "result": [ 06:20:29 DEBUG "result": [ 06:20:29 DEBUG 06:20:29 DEBUG "result": [ 06:20:29 DEBUG { 06:20:29 DEBUG { 06:20:29 DEBUG { 06:20:29 DEBUG }, 06:20:29 DEBUG "instance": "10.142.0.25:9100" 06:20:29 DEBUG }, 06:20:29 DEBUG { 06:20:29 DEBUG { 06:20:29 DEBUG 06:20:29 DEBUG { 06:20:29 DEBUG "metric": { 06:20:29 DEBUG "metric": { 06:20:29 DEBUG "metric": { 06:20:29 DEBUG "value": [ 06:20:29 DEBUG }, 06:20:29 DEBUG "value": [ 06:20:29 DEBUG "metric": { 06:20:29 DEBUG "metric": { 06:20:29 DEBUG 06:20:29 DEBUG "metric": { 06:20:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:20:29 DEBUG "instance": "10.142.0.25:9100" 06:20:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:20:29 DEBUG 06:20:29 DEBUG "value": [ 06:20:29 DEBUG 06:20:29 DEBUG "instance": "10.142.0.25:9100" 06:20:29 DEBUG "pod": "ds-idrepo-0" 06:20:29 DEBUG [...] 06:20:29 DEBUG "pod": "ds-idrepo-0" 06:20:29 DEBUG }, 06:20:29 DEBUG }, 06:20:29 DEBUG "value": [ 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG }, 06:20:29 DEBUG }, 06:20:29 DEBUG 06:20:29 DEBUG "value": [ 06:20:29 DEBUG 06:20:29 DEBUG }, 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG "value": [ 06:20:29 DEBUG "value": [ 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG "value": [ 06:20:29 DEBUG 06:20:29 DEBUG [...] 06:20:29 DEBUG 06:20:29 DEBUG [...] 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG [...] 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG "pod": "ds-cts-2" 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG "pod": "ds-cts-2" 06:20:29 DEBUG [...] 06:20:29 DEBUG [...] 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG [...] 06:20:29 DEBUG 06:20:29 DEBUG }, 06:20:29 DEBUG }, 06:20:29 DEBUG 06:20:29 DEBUG }, 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG "value": [ 06:20:29 DEBUG 1712726429, 06:20:29 DEBUG 06:20:29 DEBUG [...] 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG }, 06:20:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:20:29 DEBUG 06:20:29 DEBUG [...] 06:20:29 DEBUG "value": [ 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG "value": [ 06:20:29 DEBUG }, 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG "0" 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG "instance": "10.142.0.77:9100" 06:20:29 DEBUG "instance": "10.142.0.77:9100" 06:20:29 DEBUG }, 06:20:29 DEBUG "value": [ 06:20:29 DEBUG 1712726429, 06:20:29 DEBUG "381814389760" 06:20:29 DEBUG ] 06:20:29 DEBUG [...] 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG }, 06:20:29 DEBUG "value": [ 06:20:29 DEBUG 1712726429, 06:20:29 DEBUG ] 06:20:29 DEBUG } 06:20:29 DEBUG ] 06:20:29 DEBUG } 06:20:29 DEBUG 06:20:29 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:20:29 DEBUG }, 06:20:29 DEBUG "value": [ 06:20:29 DEBUG "42748135.166666664" 06:20:29 DEBUG ] 06:20:29 DEBUG } 06:20:29 DEBUG 06:20:29 DEBUG "value": [ 06:20:29 DEBUG ] 06:20:29 DEBUG 1712726429, 06:20:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:20:29 DEBUG 1712726429, 06:20:29 DEBUG "value": [ 06:20:29 DEBUG "instance": "10.142.0.77:9100" 06:20:29 DEBUG } 06:20:29 DEBUG } 06:20:29 DEBUG 06:20:29 DEBUG 1712726429, 06:20:29 DEBUG } 06:20:29 DEBUG "0" 06:20:29 DEBUG }, 06:20:29 DEBUG "259402.5" 06:20:29 DEBUG ] 06:20:29 DEBUG }, 06:20:29 DEBUG ] 06:20:29 DEBUG "pod": "ds-idrepo-1" 06:20:29 DEBUG }, 06:20:29 DEBUG "value": [ 06:20:29 DEBUG ] 06:20:29 DEBUG "value": [ 06:20:29 DEBUG } 06:20:29 DEBUG ] 06:20:29 DEBUG "value": [ 06:20:29 DEBUG } 06:20:29 DEBUG } 06:20:29 DEBUG 1712726429, 06:20:29 DEBUG } 06:20:29 DEBUG } 06:20:29 DEBUG 1712726429, 06:20:29 DEBUG 1712726429, 06:20:29 DEBUG } 06:20:29 DEBUG 1712726429, 06:20:29 DEBUG "0" 06:20:29 DEBUG "1438.7" 06:20:29 DEBUG ] 06:20:29 DEBUG "1438.2666666666667" 06:20:29 DEBUG "29.966666666666665" 06:20:29 DEBUG ] 06:20:29 DEBUG "0.03996666666666612" 06:20:29 DEBUG ] 06:20:29 DEBUG ] 06:20:29 DEBUG } 06:20:29 DEBUG } 06:20:29 DEBUG } 06:20:29 DEBUG } 06:20:29 DEBUG ] 06:20:29 DEBUG } 06:20:29 DEBUG } 06:20:29 DEBUG ] 06:20:29 DEBUG ] 06:20:29 DEBUG } 06:20:29 DEBUG ] 06:20:29 DEBUG } 06:20:29 DEBUG ] 06:20:29 DEBUG } 06:20:29 DEBUG ] 06:20:29 DEBUG } 06:20:29 DEBUG } 06:20:29 DEBUG } 06:20:29 DEBUG } 06:20:29 DEBUG } 06:20:29 DEBUG } 06:20:29 DEBUG ] 06:20:29 DEBUG } 06:20:29 DEBUG } 06:20:29 DEBUG } 06:20:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726429" 06:20:29 INFO [http_cmd]: http status code OK 06:20:29 DEBUG --- status code --- 06:20:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:29 DEBUG --- http response --- 06:20:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:20:29 DEBUG ----- output ----- 06:20:29 DEBUG { 06:20:29 DEBUG "status": "success", 06:20:29 DEBUG "data": { 06:20:29 DEBUG "resultType": "vector", 06:20:29 DEBUG "result": [ 06:20:29 DEBUG { 06:20:29 DEBUG "metric": { 06:20:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:20:29 DEBUG }, 06:20:29 DEBUG "value": [ 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG [...] 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG 06:20:29 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:20:29 DEBUG }, 06:20:29 DEBUG "value": [ 06:20:29 DEBUG 1712726429, 06:20:29 DEBUG "0" 06:20:29 DEBUG ] 06:20:29 DEBUG } 06:20:29 DEBUG ] 06:20:29 DEBUG } 06:20:29 DEBUG } 06:20:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712726430" 06:20:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712726430" 06:20:30 INFO [http_cmd]: http status code OK 06:20:30 DEBUG --- status code --- 06:20:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:30 DEBUG --- http response --- 06:20:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:20:30 DEBUG ----- output ----- 06:20:30 DEBUG { 06:20:30 DEBUG "status": "success", 06:20:30 DEBUG "data": { 06:20:30 DEBUG "resultType": "vector", 06:20:30 DEBUG "result": [ 06:20:30 DEBUG { 06:20:30 DEBUG "metric": { 06:20:30 DEBUG "pod": "ds-idrepo-0" 06:20:30 DEBUG }, 06:20:30 DEBUG "value": [ 06:20:30 DEBUG 06:20:30 DEBUG 06:20:30 INFO [http_cmd]: http status code OK 06:20:30 DEBUG 06:20:30 DEBUG --- status code --- 06:20:30 DEBUG [...] 06:20:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:30 DEBUG 06:20:30 DEBUG --- http response --- 06:20:30 DEBUG 06:20:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:20:30 DEBUG ----- output ----- 06:20:30 DEBUG { 06:20:30 DEBUG "status": "success", 06:20:30 DEBUG "data": { 06:20:30 DEBUG "resultType": "vector", 06:20:30 DEBUG "result": [ 06:20:30 DEBUG { 06:20:30 DEBUG "metric": { 06:20:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:20:30 DEBUG }, 06:20:30 DEBUG "value": [ 06:20:30 DEBUG 06:20:30 DEBUG 06:20:30 DEBUG 06:20:30 DEBUG [...] 06:20:30 DEBUG 06:20:30 DEBUG 06:20:30 DEBUG 06:20:30 DEBUG "pod": "ds-cts-2" 06:20:30 DEBUG 06:20:30 DEBUG }, 06:20:30 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:20:30 DEBUG }, 06:20:30 DEBUG "value": [ 06:20:30 DEBUG 1712726430, 06:20:30 DEBUG "4.2504224792306395" 06:20:30 DEBUG ] 06:20:30 DEBUG } 06:20:30 DEBUG ] 06:20:30 DEBUG } 06:20:30 DEBUG } 06:20:30 DEBUG "value": [ 06:20:30 DEBUG 1712726430, 06:20:30 DEBUG "0" 06:20:30 DEBUG ] 06:20:30 DEBUG } 06:20:30 DEBUG ] 06:20:30 DEBUG } 06:20:30 DEBUG } 06:20:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726430" 06:20:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726430" 06:20:30 INFO [http_cmd]: http status code OK 06:20:30 INFO [http_cmd]: http status code OK 06:20:30 DEBUG --- status code --- 06:20:30 DEBUG --- status code --- 06:20:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:30 DEBUG --- http response --- 06:20:30 DEBUG --- http response --- 06:20:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:20:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:20:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726430" 06:20:30 DEBUG ----- output ----- 06:20:30 DEBUG ----- output ----- 06:20:30 DEBUG { 06:20:30 DEBUG "status": "success", 06:20:30 DEBUG "data": { 06:20:30 DEBUG "resultType": "vector", 06:20:30 DEBUG "result": [ 06:20:30 DEBUG { 06:20:30 DEBUG "metric": { 06:20:30 DEBUG "pod": "ds-idrepo-2" 06:20:30 DEBUG }, 06:20:30 DEBUG "value": [ 06:20:30 DEBUG 06:20:30 DEBUG 06:20:30 DEBUG 06:20:30 DEBUG [...] 06:20:30 DEBUG 06:20:30 DEBUG 06:20:30 DEBUG 06:20:30 DEBUG "pod": "ds-cts-2" 06:20:30 DEBUG }, 06:20:30 DEBUG "value": [ 06:20:30 DEBUG 1712726430, 06:20:30 DEBUG "0.00022263666666666666" 06:20:30 DEBUG ] 06:20:30 DEBUG } 06:20:30 DEBUG ] 06:20:30 DEBUG } 06:20:30 DEBUG } 06:20:30 DEBUG { 06:20:30 DEBUG "status": "success", 06:20:30 DEBUG "data": { 06:20:30 DEBUG "resultType": "vector", 06:20:30 DEBUG "result": [ 06:20:30 DEBUG { 06:20:30 DEBUG "metric": { 06:20:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:20:30 DEBUG }, 06:20:30 DEBUG "value": [ 06:20:30 DEBUG 06:20:30 DEBUG 06:20:30 DEBUG 06:20:30 DEBUG [...] 06:20:30 DEBUG 06:20:30 DEBUG 06:20:30 DEBUG 06:20:30 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:20:30 DEBUG }, 06:20:30 DEBUG "value": [ 06:20:30 DEBUG 1712726430, 06:20:30 DEBUG "0" 06:20:30 DEBUG ] 06:20:30 DEBUG } 06:20:30 DEBUG ] 06:20:30 DEBUG } 06:20:30 DEBUG } 06:20:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712726430" 06:20:30 INFO [http_cmd]: http status code OK 06:20:30 DEBUG --- status code --- 06:20:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:30 DEBUG --- http response --- 06:20:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:20:30 DEBUG ----- output ----- 06:20:30 DEBUG { 06:20:30 DEBUG "status": "success", 06:20:30 DEBUG "data": { 06:20:30 DEBUG "resultType": "vector", 06:20:30 DEBUG "result": [ 06:20:30 DEBUG { 06:20:30 DEBUG "metric": { 06:20:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:20:30 DEBUG }, 06:20:30 DEBUG "value": [ 06:20:30 DEBUG 06:20:30 DEBUG 06:20:30 DEBUG 06:20:30 DEBUG [...] 06:20:30 DEBUG 06:20:30 DEBUG 06:20:30 DEBUG 06:20:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:20:30 DEBUG }, 06:20:30 DEBUG "value": [ 06:20:30 DEBUG 1712726430, 06:20:30 DEBUG "1438.3" 06:20:30 DEBUG ] 06:20:30 DEBUG } 06:20:30 DEBUG ] 06:20:30 DEBUG } 06:20:30 DEBUG } 06:20:30 INFO [http_cmd]: http status code OK 06:20:30 DEBUG --- status code --- 06:20:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:30 DEBUG --- http response --- 06:20:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:20:30 DEBUG ----- output ----- 06:20:30 DEBUG { 06:20:30 DEBUG "status": "success", 06:20:30 DEBUG "data": { 06:20:30 DEBUG "resultType": "vector", 06:20:30 DEBUG "result": [ 06:20:30 DEBUG { 06:20:30 DEBUG "metric": { 06:20:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:20:30 DEBUG }, 06:20:30 DEBUG "value": [ 06:20:30 DEBUG 06:20:30 DEBUG 06:20:30 DEBUG 06:20:30 DEBUG [...] 06:20:30 DEBUG 06:20:30 DEBUG 06:20:30 DEBUG 06:20:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:20:30 DEBUG }, 06:20:30 DEBUG "value": [ 06:20:30 DEBUG 1712726430, 06:20:30 DEBUG "0.05" 06:20:30 DEBUG ] 06:20:30 DEBUG } 06:20:30 DEBUG ] 06:20:30 DEBUG } 06:20:30 DEBUG } 06:20:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726430" 06:20:30 INFO [http_cmd]: http status code OK 06:20:30 DEBUG --- status code --- 06:20:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:30 DEBUG --- http response --- 06:20:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:20:30 DEBUG ----- output ----- 06:20:30 DEBUG { 06:20:30 DEBUG "status": "success", 06:20:30 DEBUG "data": { 06:20:30 DEBUG "resultType": "vector", 06:20:30 DEBUG "result": [ 06:20:30 DEBUG { 06:20:30 DEBUG "metric": { 06:20:30 DEBUG "pod": "ds-idrepo-0" 06:20:30 DEBUG }, 06:20:30 DEBUG "value": [ 06:20:30 DEBUG 06:20:30 DEBUG 06:20:30 DEBUG 06:20:30 DEBUG [...] 06:20:30 DEBUG 06:20:30 DEBUG 06:20:30 DEBUG 06:20:30 DEBUG "pod": "ds-idrepo-1" 06:20:30 DEBUG }, 06:20:30 DEBUG "value": [ 06:20:30 DEBUG 1712726430, 06:20:30 DEBUG "0" 06:20:30 DEBUG ] 06:20:30 DEBUG } 06:20:30 DEBUG ] 06:20:30 DEBUG } 06:20:30 DEBUG } 06:20:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726431" 06:20:31 INFO [http_cmd]: http status code OK 06:20:31 DEBUG --- status code --- 06:20:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:31 DEBUG --- http response --- 06:20:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:20:31 DEBUG ----- output ----- 06:20:31 DEBUG { 06:20:31 DEBUG "status": "success", 06:20:31 DEBUG "data": { 06:20:31 DEBUG "resultType": "vector", 06:20:31 DEBUG "result": [ 06:20:31 DEBUG { 06:20:31 DEBUG "metric": { 06:20:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:20:31 DEBUG }, 06:20:31 DEBUG "value": [ 06:20:31 DEBUG 06:20:31 DEBUG 06:20:31 DEBUG 06:20:31 DEBUG [...] 06:20:31 DEBUG 06:20:31 DEBUG 06:20:31 DEBUG 06:20:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:20:31 DEBUG }, 06:20:31 DEBUG "value": [ 06:20:31 DEBUG 1712726431, 06:20:31 DEBUG "1439.5333333333333" 06:20:31 DEBUG ] 06:20:31 DEBUG } 06:20:31 DEBUG ] 06:20:31 DEBUG } 06:20:31 DEBUG } 06:20:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726431" 06:20:31 INFO [http_cmd]: http status code OK 06:20:31 DEBUG --- status code --- 06:20:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:31 DEBUG --- http response --- 06:20:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:20:31 DEBUG ----- output ----- 06:20:31 DEBUG { 06:20:31 DEBUG "status": "success", 06:20:31 DEBUG "data": { 06:20:31 DEBUG "resultType": "vector", 06:20:31 DEBUG "result": [ 06:20:31 DEBUG { 06:20:31 DEBUG "metric": { 06:20:31 DEBUG "pod": "ds-idrepo-0" 06:20:31 DEBUG }, 06:20:31 DEBUG "value": [ 06:20:31 DEBUG 06:20:31 DEBUG 06:20:31 DEBUG 06:20:31 DEBUG [...] 06:20:31 DEBUG 06:20:31 DEBUG 06:20:31 DEBUG 06:20:31 DEBUG "pod": "ds-idrepo-1" 06:20:31 DEBUG }, 06:20:31 DEBUG "value": [ 06:20:31 DEBUG 1712726431, 06:20:31 DEBUG "0" 06:20:31 DEBUG ] 06:20:31 DEBUG } 06:20:31 DEBUG ] 06:20:31 DEBUG } 06:20:31 DEBUG } 06:20:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726431" 06:20:31 INFO [http_cmd]: http status code OK 06:20:31 DEBUG --- status code --- 06:20:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:31 DEBUG --- http response --- 06:20:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:20:31 DEBUG ----- output ----- 06:20:31 DEBUG { 06:20:31 DEBUG "status": "success", 06:20:31 DEBUG "data": { 06:20:31 DEBUG "resultType": "vector", 06:20:31 DEBUG "result": [ 06:20:31 DEBUG { 06:20:31 DEBUG "metric": { 06:20:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:20:31 DEBUG }, 06:20:31 DEBUG "value": [ 06:20:31 DEBUG 06:20:31 DEBUG 06:20:31 DEBUG 06:20:31 DEBUG [...] 06:20:31 DEBUG 06:20:31 DEBUG 06:20:31 DEBUG 06:20:31 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:20:31 DEBUG }, 06:20:31 DEBUG "value": [ 06:20:31 DEBUG 1712726431, 06:20:31 DEBUG "0" 06:20:31 DEBUG ] 06:20:31 DEBUG } 06:20:31 DEBUG ] 06:20:31 DEBUG } 06:20:31 DEBUG } 06:20:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712726431" 06:20:31 INFO [http_cmd]: http status code OK 06:20:31 DEBUG --- status code --- 06:20:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:31 DEBUG --- http response --- 06:20:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:20:31 DEBUG ----- output ----- 06:20:31 DEBUG { 06:20:31 DEBUG "status": "success", 06:20:31 DEBUG "data": { 06:20:31 DEBUG "resultType": "vector", 06:20:31 DEBUG "result": [ 06:20:31 DEBUG { 06:20:31 DEBUG "metric": { 06:20:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:20:31 DEBUG }, 06:20:31 DEBUG "value": [ 06:20:31 DEBUG 06:20:31 DEBUG 06:20:31 DEBUG 06:20:31 DEBUG [...] 06:20:31 DEBUG 06:20:31 DEBUG 06:20:31 DEBUG 06:20:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:20:31 DEBUG }, 06:20:31 DEBUG "value": [ 06:20:31 DEBUG 1712726431, 06:20:31 DEBUG "2485145600" 06:20:31 DEBUG ] 06:20:31 DEBUG } 06:20:31 DEBUG ] 06:20:31 DEBUG } 06:20:31 DEBUG } 06:20:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726431" 06:20:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726431" 06:20:31 INFO [http_cmd]: http status code OK 06:20:31 DEBUG --- status code --- 06:20:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:31 DEBUG --- http response --- 06:20:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:20:31 DEBUG ----- output ----- 06:20:31 DEBUG { 06:20:31 DEBUG "status": "success", 06:20:31 DEBUG "data": { 06:20:31 DEBUG "resultType": "vector", 06:20:31 DEBUG "result": [ 06:20:31 DEBUG { 06:20:31 DEBUG "metric": { 06:20:31 DEBUG "instance": "10.142.0.25:9100" 06:20:31 DEBUG }, 06:20:31 DEBUG "value": [ 06:20:31 DEBUG 06:20:31 DEBUG 06:20:31 DEBUG 06:20:31 DEBUG [...] 06:20:31 DEBUG 06:20:31 DEBUG 06:20:31 DEBUG 06:20:31 DEBUG "instance": "10.142.0.77:9100" 06:20:31 DEBUG }, 06:20:31 DEBUG "value": [ 06:20:31 DEBUG 1712726431, 06:20:31 DEBUG "42749522.33333333" 06:20:31 DEBUG ] 06:20:31 DEBUG } 06:20:31 DEBUG ] 06:20:31 DEBUG } 06:20:31 DEBUG } 06:20:31 INFO [http_cmd]: http status code OK 06:20:31 DEBUG --- status code --- 06:20:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:20:31 DEBUG --- http response --- 06:20:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:20:31 DEBUG ----- output ----- 06:20:31 DEBUG { 06:20:31 DEBUG "status": "success", 06:20:31 DEBUG "data": { 06:20:31 DEBUG "resultType": "vector", 06:20:31 DEBUG "result": [ 06:20:31 DEBUG { 06:20:31 DEBUG "metric": { 06:20:31 DEBUG "instance": "10.142.0.25:9100" 06:20:31 DEBUG }, 06:20:31 DEBUG "value": [ 06:20:31 DEBUG 06:20:31 DEBUG 06:20:31 DEBUG 06:20:31 DEBUG [...] 06:20:31 DEBUG 06:20:31 DEBUG 06:20:31 DEBUG 06:20:31 DEBUG "instance": "10.142.0.77:9100" 06:20:31 DEBUG }, 06:20:31 DEBUG "value": [ 06:20:31 DEBUG 1712726431, 06:20:31 DEBUG "0.016666666666666073" 06:20:31 DEBUG ] 06:20:31 DEBUG } 06:20:31 DEBUG ] 06:20:31 DEBUG } 06:20:31 DEBUG } 06:20:49 INFO 06:20:49 INFO [loop_until]: kubectl --namespace=xlou top pods 06:20:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:20:49 INFO [loop_until]: OK (rc = 0) 06:20:49 DEBUG --- stdout --- 06:20:49 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 11024m 8983Mi am-654c4fcbcf-9scrj 10152m 8449Mi am-654c4fcbcf-v5qb5 8357m 8445Mi ds-cts-0 8428m 13040Mi ds-cts-1 8662m 13291Mi ds-cts-2 7710m 13394Mi ds-idrepo-0 10894m 5622Mi ds-idrepo-1 354m 5672Mi ds-idrepo-2 370m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 2m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4266m 2295Mi 06:20:49 DEBUG --- stderr --- 06:20:49 DEBUG 06:20:57 INFO 06:20:57 INFO [loop_until]: kubectl --namespace=xlou top node 06:20:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:20:57 INFO [loop_until]: OK (rc = 0) 06:20:57 DEBUG --- stdout --- 06:20:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 95m 0% 2305Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9300m 58% 9317Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 79m 0% 3256Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13260m 83% 6697Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10565m 66% 9333Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11900m 74% 9800Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 79m 0% 863Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10914m 68% 6303Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8638m 54% 13721Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 7563m 47% 14065Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 438m 2% 6387Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8759m 55% 14080Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 413m 2% 6351Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4403m 27% 5905Mi 10% 06:20:57 DEBUG --- stderr --- 06:20:57 DEBUG 06:21:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726488" 06:21:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712726488" 06:21:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726488" 06:21:28 INFO [http_cmd]: http status code OK 06:21:28 DEBUG --- status code --- 06:21:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:28 DEBUG --- http response --- 06:21:28 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:21:28 ERROR [http_cmd]: ERROR 06:21:28 ERROR --- status code --- 06:21:28 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:21:28 ERROR --- http response --- 06:21:28 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:21:28 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:21:28 INFO [http_cmd]: http status code OK 06:21:28 DEBUG --- status code --- 06:21:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:28 DEBUG --- http response --- 06:21:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:21:28 DEBUG ----- output ----- 06:21:28 DEBUG { 06:21:28 DEBUG "status": "success", 06:21:28 DEBUG "data": { 06:21:28 DEBUG "resultType": "vector", 06:21:28 DEBUG "result": [ 06:21:28 DEBUG { 06:21:28 DEBUG "metric": { 06:21:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:21:28 DEBUG }, 06:21:28 DEBUG "value": [ 06:21:28 DEBUG 06:21:28 DEBUG 06:21:28 DEBUG 06:21:28 DEBUG [...] 06:21:28 DEBUG 06:21:28 DEBUG 06:21:28 DEBUG 06:21:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:21:28 DEBUG }, 06:21:28 DEBUG "value": [ 06:21:28 DEBUG 1712726488, 06:21:28 DEBUG "0" 06:21:28 DEBUG ] 06:21:28 DEBUG } 06:21:28 DEBUG ] 06:21:28 DEBUG } 06:21:28 DEBUG } 06:21:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726489" 06:21:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726489" 06:21:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726489" 06:21:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726489" 06:21:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726489" 06:21:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726489" 06:21:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726489" 06:21:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726489" 06:21:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726489" 06:21:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726489" 06:21:29 INFO [http_cmd]: http status code OK 06:21:29 INFO [http_cmd]: http status code OK 06:21:29 DEBUG --- status code --- 06:21:29 INFO [http_cmd]: http status code OK 06:21:29 INFO [http_cmd]: http status code OK 06:21:29 DEBUG --- status code --- 06:21:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:29 DEBUG --- http response --- 06:21:29 INFO [http_cmd]: http status code OK 06:21:29 DEBUG --- status code --- 06:21:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:29 DEBUG --- http response --- 06:21:29 DEBUG --- status code --- 06:21:29 INFO [http_cmd]: http status code OK 06:21:29 INFO [http_cmd]: http status code OK 06:21:29 INFO [http_cmd]: http status code OK 06:21:29 DEBUG --- status code --- 06:21:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:29 DEBUG --- http response --- 06:21:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:21:29 DEBUG ----- output ----- 06:21:29 DEBUG { 06:21:29 DEBUG "status": "success", 06:21:29 DEBUG "data": { 06:21:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:29 DEBUG --- http response --- 06:21:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:21:29 DEBUG ----- output ----- 06:21:29 DEBUG { 06:21:29 DEBUG "status": "success", 06:21:29 DEBUG "data": { 06:21:29 DEBUG "resultType": "vector", 06:21:29 DEBUG "result": [ 06:21:29 DEBUG { 06:21:29 DEBUG "metric": { 06:21:29 DEBUG "pod": "ds-idrepo-0" 06:21:29 DEBUG }, 06:21:29 DEBUG "value": [ 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG [...] 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG "pod": "ds-idrepo-1" 06:21:29 DEBUG }, 06:21:29 INFO [http_cmd]: http status code OK 06:21:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:21:29 DEBUG ----- output ----- 06:21:29 DEBUG { 06:21:29 DEBUG "status": "success", 06:21:29 DEBUG "data": { 06:21:29 DEBUG "resultType": "vector", 06:21:29 DEBUG "value": [ 06:21:29 DEBUG 1712726489, 06:21:29 DEBUG "0" 06:21:29 DEBUG --- status code --- 06:21:29 DEBUG --- status code --- 06:21:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:29 DEBUG --- http response --- 06:21:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:29 DEBUG --- status code --- 06:21:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:29 DEBUG --- http response --- 06:21:29 DEBUG "resultType": "vector", 06:21:29 DEBUG "result": [ 06:21:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:29 DEBUG --- http response --- 06:21:29 INFO [http_cmd]: http status code OK 06:21:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:21:29 DEBUG --- http response --- 06:21:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:21:29 DEBUG ----- output ----- 06:21:29 DEBUG ] 06:21:29 DEBUG } 06:21:29 DEBUG ] 06:21:29 DEBUG { 06:21:29 DEBUG "metric": { 06:21:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:21:29 DEBUG ----- output ----- 06:21:29 DEBUG { 06:21:29 DEBUG "status": "success", 06:21:29 DEBUG "data": { 06:21:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:21:29 DEBUG ----- output ----- 06:21:29 DEBUG { 06:21:29 DEBUG "status": "success", 06:21:29 DEBUG "data": { 06:21:29 DEBUG ----- output ----- 06:21:29 DEBUG { 06:21:29 DEBUG "status": "success", 06:21:29 DEBUG "data": { 06:21:29 DEBUG "resultType": "vector", 06:21:29 DEBUG "result": [ 06:21:29 DEBUG { 06:21:29 DEBUG "metric": { 06:21:29 DEBUG "pod": "ds-idrepo-0" 06:21:29 DEBUG }, 06:21:29 DEBUG { 06:21:29 DEBUG "status": "success", 06:21:29 DEBUG "value": [ 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG [...] 06:21:29 DEBUG 06:21:29 DEBUG } 06:21:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:21:29 DEBUG "resultType": "vector", 06:21:29 DEBUG "data": { 06:21:29 DEBUG --- status code --- 06:21:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:29 DEBUG --- http response --- 06:21:29 DEBUG "result": [ 06:21:29 DEBUG { 06:21:29 DEBUG "metric": { 06:21:29 DEBUG } 06:21:29 DEBUG ----- output ----- 06:21:29 DEBUG "result": [ 06:21:29 DEBUG { 06:21:29 DEBUG "metric": { 06:21:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:21:29 DEBUG }, 06:21:29 DEBUG "value": [ 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG [...] 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:21:29 DEBUG }, 06:21:29 DEBUG "value": [ 06:21:29 DEBUG 1712726489, 06:21:29 DEBUG "1437.3" 06:21:29 DEBUG ] 06:21:29 DEBUG } 06:21:29 DEBUG ] 06:21:29 DEBUG } 06:21:29 DEBUG } 06:21:29 DEBUG "pod": "ds-idrepo-0" 06:21:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:21:29 DEBUG ----- output ----- 06:21:29 DEBUG { 06:21:29 DEBUG "status": "success", 06:21:29 DEBUG "data": { 06:21:29 DEBUG "resultType": "vector", 06:21:29 DEBUG "result": [ 06:21:29 DEBUG { 06:21:29 DEBUG "metric": { 06:21:29 DEBUG --- status code --- 06:21:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:29 DEBUG --- http response --- 06:21:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:21:29 DEBUG ----- output ----- 06:21:29 DEBUG { 06:21:29 DEBUG "resultType": "vector", 06:21:29 DEBUG "result": [ 06:21:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:21:29 DEBUG }, 06:21:29 DEBUG "value": [ 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG [...] 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:21:29 DEBUG }, 06:21:29 DEBUG "value": [ 06:21:29 DEBUG { 06:21:29 DEBUG "metric": { 06:21:29 DEBUG }, 06:21:29 DEBUG "value": [ 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG [...] 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG "pod": "ds-cts-2" 06:21:29 DEBUG }, 06:21:29 DEBUG "instance": "10.142.0.25:9100" 06:21:29 DEBUG 1712726489, 06:21:29 DEBUG "259008" 06:21:29 DEBUG ] 06:21:29 DEBUG } 06:21:29 DEBUG ] 06:21:29 DEBUG } 06:21:29 DEBUG } 06:21:29 DEBUG }, 06:21:29 DEBUG "resultType": "vector", 06:21:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:21:29 DEBUG }, 06:21:29 DEBUG "value": [ 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG "value": [ 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG [...] 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG "instance": "10.142.0.77:9100" 06:21:29 DEBUG }, 06:21:29 DEBUG "value": [ 06:21:29 DEBUG 1712726489, 06:21:29 DEBUG "383978562560" 06:21:29 DEBUG ] 06:21:29 DEBUG } 06:21:29 DEBUG ] 06:21:29 DEBUG } 06:21:29 DEBUG } 06:21:29 DEBUG "value": [ 06:21:29 DEBUG 1712726489, 06:21:29 DEBUG "29.930340299303403" 06:21:29 DEBUG ] 06:21:29 DEBUG } 06:21:29 DEBUG "result": [ 06:21:29 DEBUG 06:21:29 DEBUG [...] 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG { 06:21:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:21:29 DEBUG }, 06:21:29 DEBUG "value": [ 06:21:29 DEBUG 1712726489, 06:21:29 DEBUG "1437.8333333333333" 06:21:29 DEBUG "metric": { 06:21:29 DEBUG ] 06:21:29 DEBUG } 06:21:29 DEBUG ] 06:21:29 DEBUG } 06:21:29 DEBUG } 06:21:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:21:29 DEBUG { 06:21:29 DEBUG "status": "success", 06:21:29 DEBUG "data": { 06:21:29 DEBUG "resultType": "vector", 06:21:29 DEBUG }, 06:21:29 DEBUG "result": [ 06:21:29 DEBUG { 06:21:29 DEBUG "metric": { 06:21:29 DEBUG "instance": "10.142.0.25:9100" 06:21:29 DEBUG "value": [ 06:21:29 DEBUG }, 06:21:29 DEBUG "value": [ 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG [...] 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG "instance": "10.142.0.77:9100" 06:21:29 DEBUG }, 06:21:29 DEBUG "value": [ 06:21:29 DEBUG 06:21:29 DEBUG 1712726489, 06:21:29 DEBUG "0.15195359381250748" 06:21:29 DEBUG ] 06:21:29 DEBUG } 06:21:29 DEBUG [...] 06:21:29 DEBUG ] 06:21:29 DEBUG } 06:21:29 DEBUG } 06:21:29 DEBUG "status": "success", 06:21:29 DEBUG 06:21:29 DEBUG "data": { 06:21:29 DEBUG "resultType": "vector", 06:21:29 DEBUG "result": [ 06:21:29 DEBUG 06:21:29 DEBUG { 06:21:29 DEBUG "metric": { 06:21:29 DEBUG "instance": "10.142.0.25:9100" 06:21:29 DEBUG 06:21:29 DEBUG }, 06:21:29 DEBUG "value": [ 06:21:29 DEBUG 06:21:29 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG [...] 06:21:29 DEBUG }, 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG 06:21:29 DEBUG "instance": "10.142.0.77:9100" 06:21:29 DEBUG "value": [ 06:21:29 DEBUG }, 06:21:29 DEBUG "value": [ 06:21:29 DEBUG 1712726489, 06:21:29 DEBUG 1712726489, 06:21:29 DEBUG "42714860.44805974" 06:21:29 DEBUG ] 06:21:29 DEBUG } 06:21:29 DEBUG "0" 06:21:29 DEBUG ] 06:21:29 DEBUG } 06:21:29 DEBUG } 06:21:29 DEBUG ] 06:21:29 DEBUG } 06:21:29 DEBUG ] 06:21:29 DEBUG } 06:21:29 DEBUG ] 06:21:29 DEBUG } 06:21:29 DEBUG 06:21:29 DEBUG } 06:21:29 DEBUG 06:21:29 DEBUG } 06:21:29 DEBUG "pod": "ds-cts-2" 06:21:29 DEBUG }, 06:21:29 DEBUG "value": [ 06:21:29 DEBUG 1712726489, 06:21:29 DEBUG "0" 06:21:29 DEBUG ] 06:21:29 DEBUG } 06:21:29 DEBUG ] 06:21:29 DEBUG } 06:21:29 DEBUG } 06:21:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726490" 06:21:30 INFO [http_cmd]: http status code OK 06:21:30 DEBUG --- status code --- 06:21:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:30 DEBUG --- http response --- 06:21:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:21:30 DEBUG ----- output ----- 06:21:30 DEBUG { 06:21:30 DEBUG "status": "success", 06:21:30 DEBUG "data": { 06:21:30 DEBUG "resultType": "vector", 06:21:30 DEBUG "result": [ 06:21:30 DEBUG { 06:21:30 DEBUG "metric": { 06:21:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:21:30 DEBUG }, 06:21:30 DEBUG "value": [ 06:21:30 DEBUG 06:21:30 DEBUG 06:21:30 DEBUG 06:21:30 DEBUG [...] 06:21:30 DEBUG 06:21:30 DEBUG 06:21:30 DEBUG 06:21:30 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:21:30 DEBUG }, 06:21:30 DEBUG "value": [ 06:21:30 DEBUG 1712726490, 06:21:30 DEBUG "39580.01232205973" 06:21:30 DEBUG ] 06:21:30 DEBUG } 06:21:30 DEBUG ] 06:21:30 DEBUG } 06:21:30 DEBUG } 06:21:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712726490" 06:21:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712726490" 06:21:30 INFO [http_cmd]: http status code OK 06:21:30 DEBUG --- status code --- 06:21:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:30 DEBUG --- http response --- 06:21:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:21:30 DEBUG ----- output ----- 06:21:30 DEBUG { 06:21:30 INFO [http_cmd]: http status code OK 06:21:30 DEBUG "status": "success", 06:21:30 DEBUG --- status code --- 06:21:30 DEBUG "data": { 06:21:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:30 DEBUG "resultType": "vector", 06:21:30 DEBUG --- http response --- 06:21:30 DEBUG "result": [ 06:21:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:21:30 DEBUG { 06:21:30 DEBUG ----- output ----- 06:21:30 DEBUG "metric": { 06:21:30 DEBUG { 06:21:30 DEBUG "pod": "ds-idrepo-0" 06:21:30 DEBUG "status": "success", 06:21:30 DEBUG }, 06:21:30 DEBUG "data": { 06:21:30 DEBUG "value": [ 06:21:30 DEBUG "resultType": "vector", 06:21:30 DEBUG 06:21:30 DEBUG "result": [ 06:21:30 DEBUG 06:21:30 DEBUG { 06:21:30 DEBUG 06:21:30 DEBUG "metric": { 06:21:30 DEBUG [...] 06:21:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:21:30 DEBUG 06:21:30 DEBUG }, 06:21:30 DEBUG 06:21:30 DEBUG "value": [ 06:21:30 DEBUG 06:21:30 DEBUG 06:21:30 DEBUG "pod": "ds-cts-2" 06:21:30 DEBUG }, 06:21:30 DEBUG 06:21:30 DEBUG "value": [ 06:21:30 DEBUG 06:21:30 DEBUG 1712726490, 06:21:30 DEBUG [...] 06:21:30 DEBUG "0" 06:21:30 DEBUG 06:21:30 DEBUG ] 06:21:30 DEBUG } 06:21:30 DEBUG ] 06:21:30 DEBUG } 06:21:30 DEBUG 06:21:30 DEBUG } 06:21:30 DEBUG 06:21:30 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:21:30 DEBUG }, 06:21:30 DEBUG "value": [ 06:21:30 DEBUG 1712726490, 06:21:30 DEBUG "4.283806881465638" 06:21:30 DEBUG ] 06:21:30 DEBUG } 06:21:30 DEBUG ] 06:21:30 DEBUG } 06:21:30 DEBUG } 06:21:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726490" 06:21:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726490" 06:21:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726490" 06:21:30 INFO [http_cmd]: http status code OK 06:21:30 DEBUG --- status code --- 06:21:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:30 DEBUG --- http response --- 06:21:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:21:30 DEBUG ----- output ----- 06:21:30 DEBUG { 06:21:30 DEBUG "status": "success", 06:21:30 DEBUG "data": { 06:21:30 DEBUG "resultType": "vector", 06:21:30 DEBUG "result": [ 06:21:30 DEBUG { 06:21:30 DEBUG "metric": { 06:21:30 DEBUG "pod": "ds-idrepo-2" 06:21:30 DEBUG }, 06:21:30 DEBUG "value": [ 06:21:30 DEBUG 06:21:30 DEBUG 06:21:30 DEBUG 06:21:30 DEBUG [...] 06:21:30 DEBUG 06:21:30 DEBUG 06:21:30 DEBUG 06:21:30 DEBUG "pod": "ds-cts-2" 06:21:30 DEBUG }, 06:21:30 DEBUG "value": [ 06:21:30 DEBUG 1712726490, 06:21:30 DEBUG "0.0001228732682287327" 06:21:30 DEBUG ] 06:21:30 DEBUG } 06:21:30 DEBUG ] 06:21:30 DEBUG } 06:21:30 DEBUG } 06:21:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712726490" 06:21:30 INFO [http_cmd]: http status code OK 06:21:30 DEBUG --- status code --- 06:21:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:30 DEBUG --- http response --- 06:21:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:21:30 DEBUG ----- output ----- 06:21:30 DEBUG { 06:21:30 DEBUG "status": "success", 06:21:30 DEBUG "data": { 06:21:30 DEBUG "resultType": "vector", 06:21:30 DEBUG "result": [ 06:21:30 DEBUG { 06:21:30 DEBUG "metric": { 06:21:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:21:30 DEBUG }, 06:21:30 DEBUG "value": [ 06:21:30 DEBUG 06:21:30 DEBUG 06:21:30 DEBUG 06:21:30 DEBUG [...] 06:21:30 DEBUG 06:21:30 DEBUG 06:21:30 DEBUG 06:21:30 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:21:30 DEBUG }, 06:21:30 DEBUG "value": [ 06:21:30 DEBUG 1712726490, 06:21:30 DEBUG "4.246695104575429" 06:21:30 DEBUG ] 06:21:30 DEBUG } 06:21:30 DEBUG ] 06:21:30 DEBUG } 06:21:30 DEBUG } 06:21:30 INFO [http_cmd]: http status code OK 06:21:30 DEBUG --- status code --- 06:21:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:30 DEBUG --- http response --- 06:21:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:21:30 DEBUG ----- output ----- 06:21:30 DEBUG { 06:21:30 DEBUG "status": "success", 06:21:30 DEBUG "data": { 06:21:30 DEBUG "resultType": "vector", 06:21:30 DEBUG "result": [ 06:21:30 DEBUG { 06:21:30 DEBUG "metric": { 06:21:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:21:30 DEBUG }, 06:21:30 DEBUG "value": [ 06:21:30 DEBUG 06:21:30 DEBUG 06:21:30 DEBUG 06:21:30 DEBUG [...] 06:21:30 DEBUG 06:21:30 DEBUG 06:21:30 DEBUG 06:21:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:21:30 DEBUG }, 06:21:30 DEBUG "value": [ 06:21:30 DEBUG 1712726490, 06:21:30 DEBUG "1438.3" 06:21:30 DEBUG ] 06:21:30 DEBUG } 06:21:30 DEBUG ] 06:21:30 DEBUG } 06:21:30 DEBUG } 06:21:30 INFO [http_cmd]: http status code OK 06:21:30 DEBUG --- status code --- 06:21:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:30 DEBUG --- http response --- 06:21:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:21:30 DEBUG ----- output ----- 06:21:30 DEBUG { 06:21:30 DEBUG "status": "success", 06:21:30 DEBUG "data": { 06:21:30 DEBUG "resultType": "vector", 06:21:30 DEBUG "result": [ 06:21:30 DEBUG { 06:21:30 DEBUG "metric": { 06:21:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:21:30 DEBUG }, 06:21:30 DEBUG "value": [ 06:21:30 DEBUG 06:21:30 DEBUG 06:21:30 DEBUG 06:21:30 DEBUG [...] 06:21:30 DEBUG 06:21:30 DEBUG 06:21:30 DEBUG 06:21:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:21:30 DEBUG }, 06:21:30 DEBUG "value": [ 06:21:30 DEBUG 1712726490, 06:21:30 DEBUG "0.05" 06:21:30 DEBUG ] 06:21:30 DEBUG } 06:21:30 DEBUG ] 06:21:30 DEBUG } 06:21:30 DEBUG } 06:21:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726491" 06:21:31 INFO [http_cmd]: http status code OK 06:21:31 DEBUG --- status code --- 06:21:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:31 DEBUG --- http response --- 06:21:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:21:31 DEBUG ----- output ----- 06:21:31 DEBUG { 06:21:31 DEBUG "status": "success", 06:21:31 DEBUG "data": { 06:21:31 DEBUG "resultType": "vector", 06:21:31 DEBUG "result": [ 06:21:31 DEBUG { 06:21:31 DEBUG "metric": { 06:21:31 DEBUG "pod": "ds-idrepo-0" 06:21:31 DEBUG }, 06:21:31 DEBUG "value": [ 06:21:31 DEBUG 06:21:31 DEBUG 06:21:31 DEBUG 06:21:31 DEBUG [...] 06:21:31 DEBUG 06:21:31 DEBUG 06:21:31 DEBUG 06:21:31 DEBUG "pod": "ds-idrepo-1" 06:21:31 DEBUG }, 06:21:31 DEBUG "value": [ 06:21:31 DEBUG 1712726491, 06:21:31 DEBUG "0" 06:21:31 DEBUG ] 06:21:31 DEBUG } 06:21:31 DEBUG ] 06:21:31 DEBUG } 06:21:31 DEBUG } 06:21:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726491" 06:21:31 INFO [http_cmd]: http status code OK 06:21:31 DEBUG --- status code --- 06:21:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:31 DEBUG --- http response --- 06:21:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:21:31 DEBUG ----- output ----- 06:21:31 DEBUG { 06:21:31 DEBUG "status": "success", 06:21:31 DEBUG "data": { 06:21:31 DEBUG "resultType": "vector", 06:21:31 DEBUG "result": [ 06:21:31 DEBUG { 06:21:31 DEBUG "metric": { 06:21:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:21:31 DEBUG }, 06:21:31 DEBUG "value": [ 06:21:31 DEBUG 06:21:31 DEBUG 06:21:31 DEBUG 06:21:31 DEBUG [...] 06:21:31 DEBUG 06:21:31 DEBUG 06:21:31 DEBUG 06:21:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:21:31 DEBUG }, 06:21:31 DEBUG "value": [ 06:21:31 DEBUG 1712726491, 06:21:31 DEBUG "1444.8" 06:21:31 DEBUG ] 06:21:31 DEBUG } 06:21:31 DEBUG ] 06:21:31 DEBUG } 06:21:31 DEBUG } 06:21:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726491" 06:21:31 INFO [http_cmd]: http status code OK 06:21:31 DEBUG --- status code --- 06:21:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:31 DEBUG --- http response --- 06:21:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:21:31 DEBUG ----- output ----- 06:21:31 DEBUG { 06:21:31 DEBUG "status": "success", 06:21:31 DEBUG "data": { 06:21:31 DEBUG "resultType": "vector", 06:21:31 DEBUG "result": [ 06:21:31 DEBUG { 06:21:31 DEBUG "metric": { 06:21:31 DEBUG "pod": "ds-idrepo-0" 06:21:31 DEBUG }, 06:21:31 DEBUG "value": [ 06:21:31 DEBUG 06:21:31 DEBUG 06:21:31 DEBUG 06:21:31 DEBUG [...] 06:21:31 DEBUG 06:21:31 DEBUG 06:21:31 DEBUG 06:21:31 DEBUG "pod": "ds-idrepo-1" 06:21:31 DEBUG }, 06:21:31 DEBUG "value": [ 06:21:31 DEBUG 1712726491, 06:21:31 DEBUG "0" 06:21:31 DEBUG ] 06:21:31 DEBUG } 06:21:31 DEBUG ] 06:21:31 DEBUG } 06:21:31 DEBUG } 06:21:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726491" 06:21:31 INFO [http_cmd]: http status code OK 06:21:31 DEBUG --- status code --- 06:21:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:31 DEBUG --- http response --- 06:21:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:21:31 DEBUG ----- output ----- 06:21:31 DEBUG { 06:21:31 DEBUG "status": "success", 06:21:31 DEBUG "data": { 06:21:31 DEBUG "resultType": "vector", 06:21:31 DEBUG "result": [ 06:21:31 DEBUG { 06:21:31 DEBUG "metric": { 06:21:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:21:31 DEBUG }, 06:21:31 DEBUG "value": [ 06:21:31 DEBUG 06:21:31 DEBUG 06:21:31 DEBUG 06:21:31 DEBUG [...] 06:21:31 DEBUG 06:21:31 DEBUG 06:21:31 DEBUG 06:21:31 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:21:31 DEBUG }, 06:21:31 DEBUG "value": [ 06:21:31 DEBUG 1712726491, 06:21:31 DEBUG "0" 06:21:31 DEBUG ] 06:21:31 DEBUG } 06:21:31 DEBUG ] 06:21:31 DEBUG } 06:21:31 DEBUG } 06:21:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712726491" 06:21:31 INFO [http_cmd]: http status code OK 06:21:31 DEBUG --- status code --- 06:21:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:31 DEBUG --- http response --- 06:21:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:21:31 DEBUG ----- output ----- 06:21:31 DEBUG { 06:21:31 DEBUG "status": "success", 06:21:31 DEBUG "data": { 06:21:31 DEBUG "resultType": "vector", 06:21:31 DEBUG "result": [ 06:21:31 DEBUG { 06:21:31 DEBUG "metric": { 06:21:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:21:31 DEBUG }, 06:21:31 DEBUG "value": [ 06:21:31 DEBUG 06:21:31 DEBUG 06:21:31 DEBUG 06:21:31 DEBUG [...] 06:21:31 DEBUG 06:21:31 DEBUG 06:21:31 DEBUG 06:21:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:21:31 DEBUG }, 06:21:31 DEBUG "value": [ 06:21:31 DEBUG 1712726491, 06:21:31 DEBUG "2486312960" 06:21:31 DEBUG ] 06:21:31 DEBUG } 06:21:31 DEBUG ] 06:21:31 DEBUG } 06:21:31 DEBUG } 06:21:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726491" 06:21:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726491" 06:21:31 INFO [http_cmd]: http status code OK 06:21:31 DEBUG --- status code --- 06:21:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:31 DEBUG --- http response --- 06:21:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:21:31 DEBUG ----- output ----- 06:21:31 DEBUG { 06:21:31 DEBUG "status": "success", 06:21:31 DEBUG "data": { 06:21:31 DEBUG "resultType": "vector", 06:21:31 DEBUG "result": [ 06:21:31 DEBUG { 06:21:31 DEBUG "metric": { 06:21:31 DEBUG "instance": "10.142.0.25:9100" 06:21:31 DEBUG }, 06:21:31 DEBUG "value": [ 06:21:31 DEBUG 06:21:31 DEBUG 06:21:31 DEBUG 06:21:31 DEBUG [...] 06:21:31 DEBUG 06:21:31 DEBUG 06:21:31 DEBUG 06:21:31 DEBUG "instance": "10.142.0.77:9100" 06:21:31 DEBUG }, 06:21:31 DEBUG "value": [ 06:21:31 DEBUG 1712726491, 06:21:31 DEBUG "42716943.19242565" 06:21:31 DEBUG ] 06:21:31 DEBUG } 06:21:31 DEBUG ] 06:21:31 DEBUG } 06:21:31 DEBUG } 06:21:31 INFO [http_cmd]: http status code OK 06:21:31 DEBUG --- status code --- 06:21:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:21:31 DEBUG --- http response --- 06:21:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:21:31 DEBUG ----- output ----- 06:21:31 DEBUG { 06:21:31 DEBUG "status": "success", 06:21:31 DEBUG "data": { 06:21:31 DEBUG "resultType": "vector", 06:21:31 DEBUG "result": [ 06:21:31 DEBUG { 06:21:31 DEBUG "metric": { 06:21:31 DEBUG "instance": "10.142.0.25:9100" 06:21:31 DEBUG }, 06:21:31 DEBUG "value": [ 06:21:31 DEBUG 06:21:31 DEBUG 06:21:31 DEBUG 06:21:31 DEBUG [...] 06:21:31 DEBUG 06:21:31 DEBUG 06:21:31 DEBUG 06:21:31 DEBUG "instance": "10.142.0.77:9100" 06:21:31 DEBUG }, 06:21:31 DEBUG "value": [ 06:21:31 DEBUG 1712726491, 06:21:31 DEBUG "0.04967328977196978" 06:21:31 DEBUG ] 06:21:31 DEBUG } 06:21:31 DEBUG ] 06:21:31 DEBUG } 06:21:31 DEBUG } 06:21:49 INFO 06:21:50 INFO [loop_until]: kubectl --namespace=xlou top pods 06:21:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:21:50 INFO [loop_until]: OK (rc = 0) 06:21:50 DEBUG --- stdout --- 06:21:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10980m 9034Mi am-654c4fcbcf-9scrj 9822m 8452Mi am-654c4fcbcf-v5qb5 8505m 8445Mi ds-cts-0 8462m 13393Mi ds-cts-1 8556m 12729Mi ds-cts-2 7453m 12970Mi ds-idrepo-0 10750m 5622Mi ds-idrepo-1 345m 5672Mi ds-idrepo-2 363m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 5Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 2m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4285m 2299Mi 06:21:50 DEBUG --- stderr --- 06:21:50 DEBUG 06:21:57 INFO 06:21:57 INFO [loop_until]: kubectl --namespace=xlou top node 06:21:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:21:58 INFO [loop_until]: OK (rc = 0) 06:21:58 DEBUG --- stdout --- 06:21:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 94m 0% 2308Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9303m 58% 9318Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 78m 0% 3262Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13694m 86% 6667Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10867m 68% 9335Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11900m 74% 9831Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 75m 0% 852Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10827m 68% 6303Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8278m 52% 14079Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 7592m 47% 13581Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 428m 2% 6390Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8487m 53% 13479Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 409m 2% 6350Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4389m 27% 5907Mi 10% 06:21:58 DEBUG --- stderr --- 06:21:58 DEBUG 06:22:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726548" 06:22:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712726548" 06:22:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726548" 06:22:28 INFO [http_cmd]: http status code OK 06:22:28 ERROR [http_cmd]: ERROR 06:22:28 ERROR --- status code --- 06:22:28 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:22:28 DEBUG --- status code --- 06:22:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:28 DEBUG --- http response --- 06:22:28 ERROR --- http response --- 06:22:28 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:22:28 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:22:28 INFO [http_cmd]: http status code OK 06:22:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:22:28 DEBUG --- status code --- 06:22:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:28 DEBUG ----- output ----- 06:22:28 DEBUG { 06:22:28 DEBUG --- http response --- 06:22:28 DEBUG "status": "success", 06:22:28 DEBUG "data": { 06:22:28 DEBUG "resultType": "vector", 06:22:28 DEBUG "result": [ 06:22:28 DEBUG { 06:22:28 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:22:28 DEBUG "metric": { 06:22:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:22:28 DEBUG }, 06:22:28 DEBUG "value": [ 06:22:28 DEBUG 06:22:28 DEBUG 06:22:28 DEBUG 06:22:28 DEBUG [...] 06:22:28 DEBUG 06:22:28 DEBUG 06:22:28 DEBUG 06:22:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:22:28 DEBUG }, 06:22:28 DEBUG "value": [ 06:22:28 DEBUG 1712726548, 06:22:28 DEBUG "0" 06:22:28 DEBUG ] 06:22:28 DEBUG } 06:22:28 DEBUG ] 06:22:28 DEBUG } 06:22:28 DEBUG } 06:22:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726549" 06:22:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726549" 06:22:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726549" 06:22:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726549" 06:22:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726549" 06:22:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726549" 06:22:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726549" 06:22:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726549" 06:22:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726549" 06:22:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726549" 06:22:29 INFO [http_cmd]: http status code OK 06:22:29 DEBUG --- status code --- 06:22:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:29 DEBUG --- http response --- 06:22:29 INFO [http_cmd]: http status code OK 06:22:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:22:29 DEBUG --- status code --- 06:22:29 INFO [http_cmd]: http status code OK 06:22:29 INFO [http_cmd]: http status code OK 06:22:29 INFO [http_cmd]: http status code OK 06:22:29 DEBUG --- status code --- 06:22:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:29 DEBUG --- status code --- 06:22:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:29 DEBUG --- http response --- 06:22:29 DEBUG --- http response --- 06:22:29 INFO [http_cmd]: http status code OK 06:22:29 DEBUG --- status code --- 06:22:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:29 DEBUG --- http response --- 06:22:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:22:29 DEBUG ----- output ----- 06:22:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:22:29 INFO [http_cmd]: http status code OK 06:22:29 DEBUG --- http response --- 06:22:29 INFO [http_cmd]: http status code OK 06:22:29 DEBUG --- status code --- 06:22:29 DEBUG --- status code --- 06:22:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:29 DEBUG --- http response --- 06:22:29 DEBUG { 06:22:29 DEBUG ----- output ----- 06:22:29 DEBUG --- status code --- 06:22:29 INFO [http_cmd]: http status code OK 06:22:29 INFO [http_cmd]: http status code OK 06:22:29 DEBUG --- status code --- 06:22:29 DEBUG ----- output ----- 06:22:29 DEBUG { 06:22:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:22:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:22:29 DEBUG "status": "success", 06:22:29 DEBUG { 06:22:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:29 DEBUG --- status code --- 06:22:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:29 DEBUG --- http response --- 06:22:29 DEBUG "status": "success", 06:22:29 DEBUG "data": { 06:22:29 DEBUG ----- output ----- 06:22:29 DEBUG ----- output ----- 06:22:29 DEBUG "data": { 06:22:29 DEBUG "status": "success", 06:22:29 DEBUG --- http response --- 06:22:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:29 DEBUG --- http response --- 06:22:29 DEBUG "resultType": "vector", 06:22:29 DEBUG "result": [ 06:22:29 DEBUG { 06:22:29 DEBUG { 06:22:29 DEBUG "resultType": "vector", 06:22:29 DEBUG "data": { 06:22:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:22:29 DEBUG --- http response --- 06:22:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:22:29 DEBUG { 06:22:29 DEBUG "metric": { 06:22:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:22:29 DEBUG ----- output ----- 06:22:29 DEBUG "status": "success", 06:22:29 DEBUG "result": [ 06:22:29 DEBUG "resultType": "vector", 06:22:29 DEBUG ----- output ----- 06:22:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:22:29 DEBUG ----- output ----- 06:22:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:22:29 DEBUG }, 06:22:29 DEBUG { 06:22:29 DEBUG "status": "success", 06:22:29 DEBUG "data": { 06:22:29 DEBUG { 06:22:29 DEBUG "result": [ 06:22:29 DEBUG { 06:22:29 DEBUG "status": "success", 06:22:29 DEBUG { 06:22:29 DEBUG "value": [ 06:22:29 DEBUG 06:22:29 DEBUG "data": { 06:22:29 DEBUG "resultType": "vector", 06:22:29 DEBUG "resultType": "vector", 06:22:29 DEBUG "metric": { 06:22:29 DEBUG "pod": "ds-idrepo-0" 06:22:29 DEBUG }, 06:22:29 DEBUG "value": [ 06:22:29 DEBUG ----- output ----- 06:22:29 DEBUG { 06:22:29 DEBUG "status": "success", 06:22:29 DEBUG "result": [ 06:22:29 DEBUG "data": { 06:22:29 DEBUG "resultType": "vector", 06:22:29 DEBUG { 06:22:29 DEBUG 06:22:29 DEBUG 06:22:29 DEBUG "data": { 06:22:29 DEBUG "resultType": "vector", 06:22:29 DEBUG "result": [ 06:22:29 DEBUG { 06:22:29 DEBUG "metric": { 06:22:29 DEBUG "result": [ 06:22:29 DEBUG "result": [ 06:22:29 DEBUG "metric": { 06:22:29 DEBUG 06:22:29 DEBUG [...] 06:22:29 DEBUG { 06:22:29 DEBUG "metric": { 06:22:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:22:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:22:29 DEBUG }, 06:22:29 DEBUG { 06:22:29 DEBUG "metric": { 06:22:29 DEBUG "instance": "10.142.0.25:9100" 06:22:29 DEBUG 06:22:29 DEBUG 06:22:29 DEBUG }, 06:22:29 DEBUG "value": [ 06:22:29 DEBUG 06:22:29 DEBUG "value": [ 06:22:29 DEBUG 06:22:29 DEBUG }, 06:22:29 DEBUG "value": [ 06:22:29 DEBUG 06:22:29 DEBUG "pod": "ds-idrepo-1" 06:22:29 DEBUG { 06:22:29 DEBUG "metric": { 06:22:29 DEBUG "pod": "ds-idrepo-0" 06:22:29 DEBUG 06:22:29 DEBUG 06:22:29 DEBUG [...] 06:22:29 DEBUG 06:22:29 DEBUG 06:22:29 DEBUG 06:22:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:22:29 DEBUG }, 06:22:29 DEBUG "value": [ 06:22:29 DEBUG 1712726549, 06:22:29 DEBUG "1443.7" 06:22:29 DEBUG ] 06:22:29 DEBUG } 06:22:29 DEBUG "instance": "10.142.0.25:9100" 06:22:29 DEBUG }, 06:22:29 DEBUG "value": [ 06:22:29 DEBUG ] 06:22:29 DEBUG } 06:22:29 DEBUG } 06:22:29 DEBUG "status": "success", 06:22:29 DEBUG }, 06:22:29 DEBUG "value": [ 06:22:29 DEBUG 06:22:29 DEBUG 06:22:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:22:29 DEBUG ----- output ----- 06:22:29 DEBUG "status": "success", 06:22:29 DEBUG 06:22:29 DEBUG "data": { 06:22:29 DEBUG "resultType": "vector", 06:22:29 DEBUG 1712726549, 06:22:29 DEBUG 06:22:29 DEBUG [...] 06:22:29 DEBUG { 06:22:29 DEBUG "status": "success", 06:22:29 DEBUG "data": { 06:22:29 DEBUG 06:22:29 DEBUG "result": [ 06:22:29 DEBUG { 06:22:29 DEBUG "0" 06:22:29 DEBUG 06:22:29 DEBUG 06:22:29 DEBUG 06:22:29 DEBUG "data": { 06:22:29 DEBUG "resultType": "vector", 06:22:29 DEBUG [...] 06:22:29 DEBUG "metric": { 06:22:29 DEBUG "instance": "10.142.0.25:9100" 06:22:29 DEBUG ] 06:22:29 DEBUG 06:22:29 DEBUG [...] 06:22:29 DEBUG 06:22:29 DEBUG "resultType": "vector", 06:22:29 DEBUG "result": [ 06:22:29 DEBUG 06:22:29 DEBUG }, 06:22:29 DEBUG "value": [ 06:22:29 DEBUG 06:22:29 DEBUG 06:22:29 DEBUG 06:22:29 DEBUG 06:22:29 DEBUG "result": [ 06:22:29 DEBUG { 06:22:29 DEBUG 06:22:29 DEBUG 06:22:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:22:29 DEBUG }, 06:22:29 DEBUG 06:22:29 DEBUG 06:22:29 DEBUG [...] 06:22:29 DEBUG "metric": { 06:22:29 DEBUG { 06:22:29 DEBUG }, 06:22:29 DEBUG 06:22:29 DEBUG } 06:22:29 DEBUG "value": [ 06:22:29 DEBUG 06:22:29 DEBUG 06:22:29 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:22:29 DEBUG "pod": "ds-idrepo-0" 06:22:29 DEBUG "metric": { 06:22:29 DEBUG "value": [ 06:22:29 DEBUG 06:22:29 DEBUG ] 06:22:29 DEBUG 06:22:29 DEBUG "instance": "10.142.0.77:9100" 06:22:29 DEBUG 06:22:29 DEBUG }, 06:22:29 DEBUG }, 06:22:29 DEBUG "value": [ 06:22:29 DEBUG 1712726549, 06:22:29 DEBUG [...] 06:22:29 DEBUG } 06:22:29 DEBUG 06:22:29 DEBUG }, 06:22:29 DEBUG 06:22:29 DEBUG "value": [ 06:22:29 DEBUG 06:22:29 DEBUG 06:22:29 DEBUG "1443.4666666666667" 06:22:29 DEBUG 06:22:29 DEBUG } 06:22:29 DEBUG 06:22:29 DEBUG "value": [ 06:22:29 DEBUG "instance": "10.142.0.77:9100" 06:22:29 DEBUG 1712726549, 06:22:29 DEBUG 06:22:29 DEBUG [...] 06:22:29 DEBUG ] 06:22:29 DEBUG 06:22:29 DEBUG 06:22:29 DEBUG 1712726549, 06:22:29 DEBUG }, 06:22:29 DEBUG "0" 06:22:29 DEBUG 06:22:29 DEBUG 06:22:29 DEBUG } 06:22:29 DEBUG "instance": "10.142.0.77:9100" 06:22:29 DEBUG }, 06:22:29 DEBUG "385923652608" 06:22:29 DEBUG "value": [ 06:22:29 DEBUG ] 06:22:29 DEBUG 06:22:29 DEBUG "pod": "ds-cts-2" 06:22:29 DEBUG ] 06:22:29 DEBUG "value": [ 06:22:29 DEBUG 1712726549, 06:22:29 DEBUG ] 06:22:29 DEBUG 1712726549, 06:22:29 DEBUG } 06:22:29 DEBUG }, 06:22:29 DEBUG "value": [ 06:22:29 DEBUG } 06:22:29 DEBUG } 06:22:29 DEBUG [...] 06:22:29 DEBUG } 06:22:29 DEBUG "42446965.83333333" 06:22:29 DEBUG ] 06:22:29 DEBUG 1712726549, 06:22:29 DEBUG "29.53333333333333" 06:22:29 DEBUG 06:22:29 DEBUG 06:22:29 DEBUG ] 06:22:29 DEBUG ] 06:22:29 DEBUG } 06:22:29 DEBUG ] 06:22:29 DEBUG } 06:22:29 DEBUG 06:22:29 DEBUG "pod": "ds-cts-2" 06:22:29 DEBUG } 06:22:29 DEBUG } 06:22:29 DEBUG } 06:22:29 DEBUG ] 06:22:29 DEBUG } 06:22:29 DEBUG }, 06:22:29 DEBUG "value": [ 06:22:29 DEBUG } 06:22:29 DEBUG ] 06:22:29 DEBUG } 06:22:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:22:29 DEBUG 1712726549, 06:22:29 DEBUG "0" 06:22:29 DEBUG } 06:22:29 DEBUG } 06:22:29 DEBUG ] 06:22:29 DEBUG } 06:22:29 DEBUG }, 06:22:29 DEBUG ] 06:22:29 DEBUG } 06:22:29 DEBUG "value": [ 06:22:29 DEBUG } 06:22:29 DEBUG "0.163366666666665" 06:22:29 DEBUG 06:22:29 DEBUG ] 06:22:29 DEBUG 06:22:29 DEBUG } 06:22:29 DEBUG 06:22:29 DEBUG ] 06:22:29 DEBUG [...] 06:22:29 DEBUG } 06:22:29 DEBUG 06:22:29 DEBUG } 06:22:29 DEBUG 06:22:29 DEBUG 06:22:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:22:29 DEBUG }, 06:22:29 DEBUG "value": [ 06:22:29 DEBUG 1712726549, 06:22:29 DEBUG "258510.5" 06:22:29 DEBUG ] 06:22:29 DEBUG } 06:22:29 DEBUG ] 06:22:29 DEBUG } 06:22:29 DEBUG } 06:22:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726550" 06:22:30 INFO [http_cmd]: http status code OK 06:22:30 DEBUG --- status code --- 06:22:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:30 DEBUG --- http response --- 06:22:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:22:30 DEBUG ----- output ----- 06:22:30 DEBUG { 06:22:30 DEBUG "status": "success", 06:22:30 DEBUG "data": { 06:22:30 DEBUG "resultType": "vector", 06:22:30 DEBUG "result": [ 06:22:30 DEBUG { 06:22:30 DEBUG "metric": { 06:22:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:22:30 DEBUG }, 06:22:30 DEBUG "value": [ 06:22:30 DEBUG 06:22:30 DEBUG 06:22:30 DEBUG 06:22:30 DEBUG [...] 06:22:30 DEBUG 06:22:30 DEBUG 06:22:30 DEBUG 06:22:30 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:22:30 DEBUG }, 06:22:30 DEBUG "value": [ 06:22:30 DEBUG 1712726550, 06:22:30 DEBUG "56000004.10749104" 06:22:30 DEBUG ] 06:22:30 DEBUG } 06:22:30 DEBUG ] 06:22:30 DEBUG } 06:22:30 DEBUG } 06:22:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712726550" 06:22:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712726550" 06:22:30 INFO [http_cmd]: http status code OK 06:22:30 DEBUG --- status code --- 06:22:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:30 DEBUG --- http response --- 06:22:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:22:30 DEBUG ----- output ----- 06:22:30 DEBUG { 06:22:30 DEBUG "status": "success", 06:22:30 DEBUG "data": { 06:22:30 DEBUG "resultType": "vector", 06:22:30 DEBUG "result": [ 06:22:30 DEBUG { 06:22:30 DEBUG "metric": { 06:22:30 DEBUG "pod": "ds-idrepo-0" 06:22:30 DEBUG }, 06:22:30 DEBUG "value": [ 06:22:30 DEBUG 06:22:30 DEBUG 06:22:30 DEBUG 06:22:30 INFO [http_cmd]: http status code OK 06:22:30 DEBUG [...] 06:22:30 DEBUG --- status code --- 06:22:30 DEBUG 06:22:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:30 DEBUG 06:22:30 DEBUG --- http response --- 06:22:30 DEBUG 06:22:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:22:30 DEBUG ----- output ----- 06:22:30 DEBUG { 06:22:30 DEBUG "pod": "ds-cts-2" 06:22:30 DEBUG }, 06:22:30 DEBUG "value": [ 06:22:30 DEBUG "status": "success", 06:22:30 DEBUG 1712726550, 06:22:30 DEBUG "data": { 06:22:30 DEBUG "0" 06:22:30 DEBUG "resultType": "vector", 06:22:30 DEBUG ] 06:22:30 DEBUG "result": [ 06:22:30 DEBUG } 06:22:30 DEBUG { 06:22:30 DEBUG ] 06:22:30 DEBUG "metric": { 06:22:30 DEBUG } 06:22:30 DEBUG } 06:22:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:22:30 DEBUG }, 06:22:30 DEBUG "value": [ 06:22:30 DEBUG 06:22:30 DEBUG 06:22:30 DEBUG 06:22:30 DEBUG [...] 06:22:30 DEBUG 06:22:30 DEBUG 06:22:30 DEBUG 06:22:30 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:22:30 DEBUG }, 06:22:30 DEBUG "value": [ 06:22:30 DEBUG 1712726550, 06:22:30 DEBUG "4.350763390215288" 06:22:30 DEBUG ] 06:22:30 DEBUG } 06:22:30 DEBUG ] 06:22:30 DEBUG } 06:22:30 DEBUG } 06:22:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726550" 06:22:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726550" 06:22:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726550" 06:22:30 INFO [http_cmd]: http status code OK 06:22:30 DEBUG --- status code --- 06:22:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:30 DEBUG --- http response --- 06:22:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:22:30 DEBUG ----- output ----- 06:22:30 DEBUG { 06:22:30 DEBUG "status": "success", 06:22:30 DEBUG "data": { 06:22:30 DEBUG "resultType": "vector", 06:22:30 DEBUG "result": [ 06:22:30 DEBUG { 06:22:30 DEBUG "metric": { 06:22:30 DEBUG "pod": "ds-idrepo-2" 06:22:30 DEBUG }, 06:22:30 DEBUG "value": [ 06:22:30 DEBUG 06:22:30 DEBUG 06:22:30 DEBUG 06:22:30 DEBUG [...] 06:22:30 DEBUG 06:22:30 DEBUG 06:22:30 DEBUG 06:22:30 DEBUG "pod": "ds-cts-2" 06:22:30 DEBUG }, 06:22:30 DEBUG "value": [ 06:22:30 DEBUG 1712726550, 06:22:30 DEBUG "0.00019353222222222223" 06:22:30 DEBUG ] 06:22:30 DEBUG } 06:22:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712726550" 06:22:30 DEBUG ] 06:22:30 DEBUG } 06:22:30 DEBUG } 06:22:30 INFO [http_cmd]: http status code OK 06:22:30 INFO [http_cmd]: http status code OK 06:22:30 DEBUG --- status code --- 06:22:30 DEBUG --- status code --- 06:22:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:30 DEBUG --- http response --- 06:22:30 DEBUG --- http response --- 06:22:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:22:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:22:30 DEBUG ----- output ----- 06:22:30 DEBUG ----- output ----- 06:22:30 DEBUG { 06:22:30 DEBUG { 06:22:30 DEBUG "status": "success", 06:22:30 DEBUG "data": { 06:22:30 DEBUG "status": "success", 06:22:30 DEBUG "data": { 06:22:30 DEBUG "resultType": "vector", 06:22:30 DEBUG "result": [ 06:22:30 DEBUG { 06:22:30 DEBUG "metric": { 06:22:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:22:30 DEBUG }, 06:22:30 DEBUG "value": [ 06:22:30 DEBUG 06:22:30 DEBUG 06:22:30 DEBUG 06:22:30 DEBUG [...] 06:22:30 DEBUG 06:22:30 DEBUG 06:22:30 DEBUG 06:22:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:22:30 DEBUG }, 06:22:30 DEBUG "value": [ 06:22:30 DEBUG 1712726550, 06:22:30 DEBUG "1443.3" 06:22:30 DEBUG ] 06:22:30 DEBUG "resultType": "vector", 06:22:30 DEBUG } 06:22:30 INFO [http_cmd]: http status code OK 06:22:30 DEBUG "result": [ 06:22:30 DEBUG ] 06:22:30 DEBUG --- status code --- 06:22:30 DEBUG { 06:22:30 DEBUG } 06:22:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:30 DEBUG --- http response --- 06:22:30 DEBUG } 06:22:30 DEBUG "metric": { 06:22:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:22:30 DEBUG ----- output ----- 06:22:30 DEBUG { 06:22:30 DEBUG "status": "success", 06:22:30 DEBUG "data": { 06:22:30 DEBUG "resultType": "vector", 06:22:30 DEBUG "result": [ 06:22:30 DEBUG { 06:22:30 DEBUG "metric": { 06:22:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:22:30 DEBUG }, 06:22:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:22:30 DEBUG }, 06:22:30 DEBUG "value": [ 06:22:30 DEBUG 06:22:30 DEBUG "value": [ 06:22:30 DEBUG 06:22:30 DEBUG 06:22:30 DEBUG 06:22:30 DEBUG 06:22:30 DEBUG [...] 06:22:30 DEBUG 06:22:30 DEBUG 06:22:30 DEBUG 06:22:30 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:22:30 DEBUG 06:22:30 DEBUG [...] 06:22:30 DEBUG 06:22:30 DEBUG 06:22:30 DEBUG 06:22:30 DEBUG }, 06:22:30 DEBUG "value": [ 06:22:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:22:30 DEBUG }, 06:22:30 DEBUG 1712726550, 06:22:30 DEBUG "0.006613199663688538" 06:22:30 DEBUG "value": [ 06:22:30 DEBUG 1712726550, 06:22:30 DEBUG ] 06:22:30 DEBUG } 06:22:30 DEBUG "0.051" 06:22:30 DEBUG ] 06:22:30 DEBUG ] 06:22:30 DEBUG } 06:22:30 DEBUG } 06:22:30 DEBUG ] 06:22:30 DEBUG } 06:22:30 DEBUG } 06:22:30 DEBUG } 06:22:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726551" 06:22:31 INFO [http_cmd]: http status code OK 06:22:31 DEBUG --- status code --- 06:22:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:31 DEBUG --- http response --- 06:22:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:22:31 DEBUG ----- output ----- 06:22:31 DEBUG { 06:22:31 DEBUG "status": "success", 06:22:31 DEBUG "data": { 06:22:31 DEBUG "resultType": "vector", 06:22:31 DEBUG "result": [ 06:22:31 DEBUG { 06:22:31 DEBUG "metric": { 06:22:31 DEBUG "pod": "ds-idrepo-0" 06:22:31 DEBUG }, 06:22:31 DEBUG "value": [ 06:22:31 DEBUG 06:22:31 DEBUG 06:22:31 DEBUG 06:22:31 DEBUG [...] 06:22:31 DEBUG 06:22:31 DEBUG 06:22:31 DEBUG 06:22:31 DEBUG "pod": "ds-idrepo-1" 06:22:31 DEBUG }, 06:22:31 DEBUG "value": [ 06:22:31 DEBUG 1712726551, 06:22:31 DEBUG "0" 06:22:31 DEBUG ] 06:22:31 DEBUG } 06:22:31 DEBUG ] 06:22:31 DEBUG } 06:22:31 DEBUG } 06:22:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726551" 06:22:31 INFO [http_cmd]: http status code OK 06:22:31 DEBUG --- status code --- 06:22:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:31 DEBUG --- http response --- 06:22:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:22:31 DEBUG ----- output ----- 06:22:31 DEBUG { 06:22:31 DEBUG "status": "success", 06:22:31 DEBUG "data": { 06:22:31 DEBUG "resultType": "vector", 06:22:31 DEBUG "result": [ 06:22:31 DEBUG { 06:22:31 DEBUG "metric": { 06:22:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:22:31 DEBUG }, 06:22:31 DEBUG "value": [ 06:22:31 DEBUG 06:22:31 DEBUG 06:22:31 DEBUG 06:22:31 DEBUG [...] 06:22:31 DEBUG 06:22:31 DEBUG 06:22:31 DEBUG 06:22:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:22:31 DEBUG }, 06:22:31 DEBUG "value": [ 06:22:31 DEBUG 1712726551, 06:22:31 DEBUG "1451.4333333333334" 06:22:31 DEBUG ] 06:22:31 DEBUG } 06:22:31 DEBUG ] 06:22:31 DEBUG } 06:22:31 DEBUG } 06:22:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726551" 06:22:31 INFO [http_cmd]: http status code OK 06:22:31 DEBUG --- status code --- 06:22:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:31 DEBUG --- http response --- 06:22:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:22:31 DEBUG ----- output ----- 06:22:31 DEBUG { 06:22:31 DEBUG "status": "success", 06:22:31 DEBUG "data": { 06:22:31 DEBUG "resultType": "vector", 06:22:31 DEBUG "result": [ 06:22:31 DEBUG { 06:22:31 DEBUG "metric": { 06:22:31 DEBUG "pod": "ds-idrepo-0" 06:22:31 DEBUG }, 06:22:31 DEBUG "value": [ 06:22:31 DEBUG 06:22:31 DEBUG 06:22:31 DEBUG 06:22:31 DEBUG [...] 06:22:31 DEBUG 06:22:31 DEBUG 06:22:31 DEBUG 06:22:31 DEBUG "pod": "ds-idrepo-1" 06:22:31 DEBUG }, 06:22:31 DEBUG "value": [ 06:22:31 DEBUG 1712726551, 06:22:31 DEBUG "0" 06:22:31 DEBUG ] 06:22:31 DEBUG } 06:22:31 DEBUG ] 06:22:31 DEBUG } 06:22:31 DEBUG } 06:22:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726551" 06:22:31 INFO [http_cmd]: http status code OK 06:22:31 DEBUG --- status code --- 06:22:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:31 DEBUG --- http response --- 06:22:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:22:31 DEBUG ----- output ----- 06:22:31 DEBUG { 06:22:31 DEBUG "status": "success", 06:22:31 DEBUG "data": { 06:22:31 DEBUG "resultType": "vector", 06:22:31 DEBUG "result": [ 06:22:31 DEBUG { 06:22:31 DEBUG "metric": { 06:22:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:22:31 DEBUG }, 06:22:31 DEBUG "value": [ 06:22:31 DEBUG 06:22:31 DEBUG 06:22:31 DEBUG 06:22:31 DEBUG [...] 06:22:31 DEBUG 06:22:31 DEBUG 06:22:31 DEBUG 06:22:31 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:22:31 DEBUG }, 06:22:31 DEBUG "value": [ 06:22:31 DEBUG 1712726551, 06:22:31 DEBUG "0" 06:22:31 DEBUG ] 06:22:31 DEBUG } 06:22:31 DEBUG ] 06:22:31 DEBUG } 06:22:31 DEBUG } 06:22:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712726551" 06:22:31 INFO [http_cmd]: http status code OK 06:22:31 DEBUG --- status code --- 06:22:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:31 DEBUG --- http response --- 06:22:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:22:31 DEBUG ----- output ----- 06:22:31 DEBUG { 06:22:31 DEBUG "status": "success", 06:22:31 DEBUG "data": { 06:22:31 DEBUG "resultType": "vector", 06:22:31 DEBUG "result": [ 06:22:31 DEBUG { 06:22:31 DEBUG "metric": { 06:22:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:22:31 DEBUG }, 06:22:31 DEBUG "value": [ 06:22:31 DEBUG 06:22:31 DEBUG 06:22:31 DEBUG 06:22:31 DEBUG [...] 06:22:31 DEBUG 06:22:31 DEBUG 06:22:31 DEBUG 06:22:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:22:31 DEBUG }, 06:22:31 DEBUG "value": [ 06:22:31 DEBUG 1712726551, 06:22:31 DEBUG "2488565760" 06:22:31 DEBUG ] 06:22:31 DEBUG } 06:22:31 DEBUG ] 06:22:31 DEBUG } 06:22:31 DEBUG } 06:22:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726551" 06:22:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726551" 06:22:31 INFO [http_cmd]: http status code OK 06:22:31 DEBUG --- status code --- 06:22:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:31 DEBUG --- http response --- 06:22:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:22:31 DEBUG ----- output ----- 06:22:31 DEBUG { 06:22:31 DEBUG "status": "success", 06:22:31 DEBUG "data": { 06:22:31 DEBUG "resultType": "vector", 06:22:31 DEBUG "result": [ 06:22:31 DEBUG { 06:22:31 DEBUG "metric": { 06:22:31 DEBUG "instance": "10.142.0.25:9100" 06:22:31 DEBUG }, 06:22:31 DEBUG "value": [ 06:22:31 DEBUG 06:22:31 DEBUG 06:22:31 DEBUG 06:22:31 DEBUG [...] 06:22:31 DEBUG 06:22:31 DEBUG 06:22:31 DEBUG 06:22:31 DEBUG "instance": "10.142.0.77:9100" 06:22:31 DEBUG }, 06:22:31 DEBUG "value": [ 06:22:31 DEBUG 1712726551, 06:22:31 DEBUG "42450471.166666664" 06:22:31 DEBUG ] 06:22:31 DEBUG } 06:22:31 DEBUG ] 06:22:31 DEBUG } 06:22:31 DEBUG } 06:22:31 INFO [http_cmd]: http status code OK 06:22:31 DEBUG --- status code --- 06:22:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:22:31 DEBUG --- http response --- 06:22:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:22:31 DEBUG ----- output ----- 06:22:31 DEBUG { 06:22:31 DEBUG "status": "success", 06:22:31 DEBUG "data": { 06:22:31 DEBUG "resultType": "vector", 06:22:31 DEBUG "result": [ 06:22:31 DEBUG { 06:22:31 DEBUG "metric": { 06:22:31 DEBUG "instance": "10.142.0.25:9100" 06:22:31 DEBUG }, 06:22:31 DEBUG "value": [ 06:22:31 DEBUG 06:22:31 DEBUG 06:22:31 DEBUG 06:22:31 DEBUG [...] 06:22:31 DEBUG 06:22:31 DEBUG 06:22:31 DEBUG 06:22:31 DEBUG "instance": "10.142.0.77:9100" 06:22:31 DEBUG }, 06:22:31 DEBUG "value": [ 06:22:31 DEBUG 1712726551, 06:22:31 DEBUG "0.048666666666666095" 06:22:31 DEBUG ] 06:22:31 DEBUG } 06:22:31 DEBUG ] 06:22:31 DEBUG } 06:22:31 DEBUG } 06:22:50 INFO 06:22:50 INFO [loop_until]: kubectl --namespace=xlou top pods 06:22:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:22:50 INFO [loop_until]: OK (rc = 0) 06:22:50 DEBUG --- stdout --- 06:22:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10385m 9002Mi am-654c4fcbcf-9scrj 10260m 8455Mi am-654c4fcbcf-v5qb5 8469m 8444Mi ds-cts-0 8255m 12565Mi ds-cts-1 8618m 13391Mi ds-cts-2 7516m 13393Mi ds-idrepo-0 10874m 5622Mi ds-idrepo-1 340m 5672Mi ds-idrepo-2 377m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 5Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 5m 2213Mi lodemon-b776b7d88-thrqf 10m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4135m 2351Mi 06:22:50 DEBUG --- stderr --- 06:22:50 DEBUG 06:22:58 INFO 06:22:58 INFO [loop_until]: kubectl --namespace=xlou top node 06:22:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:22:58 INFO [loop_until]: OK (rc = 0) 06:22:58 DEBUG --- stdout --- 06:22:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 95m 0% 2306Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8903m 56% 9320Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 84m 0% 3257Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13743m 86% 6678Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10903m 68% 9337Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11879m 74% 9815Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 74m 0% 854Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10878m 68% 6301Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8565m 53% 13237Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 7635m 48% 13147Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 481m 3% 6390Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8625m 54% 14087Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 397m 2% 6355Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4363m 27% 5955Mi 10% 06:22:58 DEBUG --- stderr --- 06:22:58 DEBUG 06:23:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712726608" 06:23:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726608" 06:23:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726608" 06:23:28 INFO [http_cmd]: http status code OK 06:23:28 INFO [http_cmd]: http status code OK 06:23:28 DEBUG --- status code --- 06:23:28 DEBUG --- status code --- 06:23:28 ERROR [http_cmd]: ERROR 06:23:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:28 DEBUG --- http response --- 06:23:28 DEBUG --- http response --- 06:23:28 ERROR --- status code --- 06:23:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:23:28 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:23:28 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:23:28 DEBUG ----- output ----- 06:23:28 ERROR --- http response --- 06:23:28 DEBUG { 06:23:28 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:23:28 DEBUG "status": "success", 06:23:28 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:23:28 DEBUG "data": { 06:23:28 DEBUG "resultType": "vector", 06:23:28 DEBUG "result": [ 06:23:28 DEBUG { 06:23:28 DEBUG "metric": { 06:23:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:23:28 DEBUG }, 06:23:28 DEBUG "value": [ 06:23:28 DEBUG 06:23:28 DEBUG 06:23:28 DEBUG 06:23:28 DEBUG [...] 06:23:28 DEBUG 06:23:28 DEBUG 06:23:28 DEBUG 06:23:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:23:28 DEBUG }, 06:23:28 DEBUG "value": [ 06:23:28 DEBUG 1712726608, 06:23:28 DEBUG "0" 06:23:28 DEBUG ] 06:23:28 DEBUG } 06:23:28 DEBUG ] 06:23:28 DEBUG } 06:23:28 DEBUG } 06:23:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726609" 06:23:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726609" 06:23:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726609" 06:23:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726609" 06:23:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726609" 06:23:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726609" 06:23:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726609" 06:23:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726609" 06:23:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726609" 06:23:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726609" 06:23:29 INFO [http_cmd]: http status code OK 06:23:29 INFO [http_cmd]: http status code OK 06:23:29 INFO [http_cmd]: http status code OK 06:23:29 INFO [http_cmd]: http status code OK 06:23:29 INFO [http_cmd]: http status code OK 06:23:29 DEBUG --- status code --- 06:23:29 INFO [http_cmd]: http status code OK 06:23:29 DEBUG --- status code --- 06:23:29 INFO [http_cmd]: http status code OK 06:23:29 INFO [http_cmd]: http status code OK 06:23:29 DEBUG --- status code --- 06:23:29 DEBUG --- status code --- 06:23:29 DEBUG --- status code --- 06:23:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:29 DEBUG --- http response --- 06:23:29 DEBUG --- status code --- 06:23:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:29 DEBUG --- status code --- 06:23:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:29 DEBUG --- http response --- 06:23:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:23:29 DEBUG ----- output ----- 06:23:29 DEBUG { 06:23:29 DEBUG "status": "success", 06:23:29 DEBUG "data": { 06:23:29 DEBUG "resultType": "vector", 06:23:29 DEBUG "result": [ 06:23:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:23:29 DEBUG --- http response --- 06:23:29 DEBUG --- http response --- 06:23:29 DEBUG { 06:23:29 DEBUG "metric": { 06:23:29 DEBUG "pod": "ds-idrepo-0" 06:23:29 DEBUG }, 06:23:29 DEBUG ----- output ----- 06:23:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:29 DEBUG --- http response --- 06:23:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:23:29 DEBUG "value": [ 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG { 06:23:29 INFO [http_cmd]: http status code OK 06:23:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:23:29 DEBUG ----- output ----- 06:23:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:23:29 DEBUG [...] 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG "status": "success", 06:23:29 DEBUG --- status code --- 06:23:29 DEBUG ----- output ----- 06:23:29 DEBUG { 06:23:29 DEBUG --- http response --- 06:23:29 DEBUG ----- output ----- 06:23:29 DEBUG "pod": "ds-cts-2" 06:23:29 DEBUG }, 06:23:29 INFO [http_cmd]: http status code OK 06:23:29 DEBUG --- status code --- 06:23:29 DEBUG "data": { 06:23:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:29 DEBUG { 06:23:29 DEBUG "status": "success", 06:23:29 DEBUG { 06:23:29 DEBUG "status": "success", 06:23:29 DEBUG "data": { 06:23:29 DEBUG "resultType": "vector", 06:23:29 DEBUG "result": [ 06:23:29 DEBUG { 06:23:29 DEBUG "metric": { 06:23:29 DEBUG --- http response --- 06:23:29 DEBUG "status": "success", 06:23:29 DEBUG "data": { 06:23:29 DEBUG "instance": "10.142.0.25:9100" 06:23:29 DEBUG }, 06:23:29 DEBUG "value": [ 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:23:29 DEBUG "data": { 06:23:29 DEBUG "resultType": "vector", 06:23:29 DEBUG 06:23:29 DEBUG [...] 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG ----- output ----- 06:23:29 DEBUG "resultType": "vector", 06:23:29 DEBUG "result": [ 06:23:29 DEBUG "instance": "10.142.0.77:9100" 06:23:29 DEBUG }, 06:23:29 DEBUG "value": [ 06:23:29 DEBUG 1712726609, 06:23:29 DEBUG "387927962624" 06:23:29 DEBUG { 06:23:29 DEBUG "result": [ 06:23:29 DEBUG { 06:23:29 DEBUG ] 06:23:29 DEBUG } 06:23:29 DEBUG ] 06:23:29 DEBUG } 06:23:29 DEBUG } 06:23:29 DEBUG "status": "success", 06:23:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:29 DEBUG "metric": { 06:23:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:29 DEBUG --- http response --- 06:23:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:23:29 DEBUG ----- output ----- 06:23:29 DEBUG "data": { 06:23:29 DEBUG --- status code --- 06:23:29 DEBUG "resultType": "vector", 06:23:29 DEBUG "result": [ 06:23:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:23:29 DEBUG { 06:23:29 DEBUG "status": "success", 06:23:29 DEBUG "data": { 06:23:29 DEBUG "resultType": "vector", 06:23:29 DEBUG "resultType": "vector", 06:23:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:29 DEBUG { 06:23:29 DEBUG "metric": { 06:23:29 DEBUG }, 06:23:29 DEBUG "result": [ 06:23:29 DEBUG { 06:23:29 DEBUG "metric": { 06:23:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:23:29 DEBUG "result": [ 06:23:29 DEBUG --- http response --- 06:23:29 DEBUG "instance": "10.142.0.25:9100" 06:23:29 DEBUG }, 06:23:29 DEBUG "value": [ 06:23:29 DEBUG }, 06:23:29 DEBUG "value": [ 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG { 06:23:29 DEBUG "metric": { 06:23:29 DEBUG "value": [ 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG [...] 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:23:29 DEBUG }, 06:23:29 DEBUG 06:23:29 DEBUG "instance": "10.142.0.77:9100" 06:23:29 DEBUG }, 06:23:29 DEBUG "value": [ 06:23:29 DEBUG 1712726609, 06:23:29 DEBUG "0.0415333333333353" 06:23:29 DEBUG "value": [ 06:23:29 DEBUG 06:23:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:23:29 DEBUG ----- output ----- 06:23:29 DEBUG { 06:23:29 DEBUG "status": "success", 06:23:29 DEBUG "data": { 06:23:29 DEBUG "resultType": "vector", 06:23:29 DEBUG "result": [ 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG [...] 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:23:29 DEBUG 06:23:29 DEBUG }, 06:23:29 DEBUG "value": [ 06:23:29 DEBUG 1712726609, 06:23:29 DEBUG "1445.7333333333333" 06:23:29 DEBUG ] 06:23:29 DEBUG } 06:23:29 DEBUG ] 06:23:29 DEBUG } 06:23:29 DEBUG } 06:23:29 DEBUG [...] 06:23:29 DEBUG --- http response --- 06:23:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:23:29 DEBUG ----- output ----- 06:23:29 DEBUG { 06:23:29 DEBUG "status": "success", 06:23:29 DEBUG "data": { 06:23:29 DEBUG "resultType": "vector", 06:23:29 DEBUG "result": [ 06:23:29 DEBUG { 06:23:29 DEBUG "metric": { 06:23:29 DEBUG "pod": "ds-idrepo-0" 06:23:29 DEBUG }, 06:23:29 DEBUG "value": [ 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG [...] 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG "pod": "ds-cts-2" 06:23:29 DEBUG }, 06:23:29 DEBUG "value": [ 06:23:29 DEBUG 1712726609, 06:23:29 DEBUG "0" 06:23:29 DEBUG ] 06:23:29 DEBUG } 06:23:29 DEBUG ] 06:23:29 DEBUG } 06:23:29 DEBUG } 06:23:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:23:29 DEBUG ----- output ----- 06:23:29 DEBUG { 06:23:29 DEBUG "status": "success", 06:23:29 DEBUG "data": { 06:23:29 DEBUG "resultType": "vector", 06:23:29 DEBUG "result": [ 06:23:29 DEBUG { 06:23:29 DEBUG "metric": { 06:23:29 DEBUG "instance": "10.142.0.25:9100" 06:23:29 DEBUG }, 06:23:29 DEBUG "value": [ 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG [...] 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG "instance": "10.142.0.77:9100" 06:23:29 DEBUG }, 06:23:29 DEBUG "value": [ 06:23:29 DEBUG 1712726609, 06:23:29 DEBUG "42761183.266666666" 06:23:29 DEBUG ] 06:23:29 DEBUG } 06:23:29 DEBUG ] 06:23:29 DEBUG } 06:23:29 DEBUG } 06:23:29 DEBUG { 06:23:29 DEBUG "metric": { 06:23:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:23:29 DEBUG }, 06:23:29 DEBUG "value": [ 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG [...] 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:23:29 DEBUG }, 06:23:29 DEBUG "value": [ 06:23:29 DEBUG 1712726609, 06:23:29 DEBUG "1445.7666666666667" 06:23:29 DEBUG ] 06:23:29 DEBUG } 06:23:29 DEBUG ] 06:23:29 DEBUG } 06:23:29 DEBUG } 06:23:29 DEBUG "value": [ 06:23:29 DEBUG 1712726609, 06:23:29 DEBUG "29.833333333333332" 06:23:29 DEBUG ] 06:23:29 DEBUG } 06:23:29 DEBUG ] 06:23:29 DEBUG } 06:23:29 DEBUG } 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:23:29 DEBUG }, 06:23:29 DEBUG "value": [ 06:23:29 DEBUG 1712726609, 06:23:29 DEBUG "258471" 06:23:29 DEBUG ] 06:23:29 DEBUG } 06:23:29 DEBUG ] 06:23:29 DEBUG } 06:23:29 DEBUG } 06:23:29 DEBUG 06:23:29 DEBUG [...] 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:23:29 DEBUG }, 06:23:29 DEBUG "value": [ 06:23:29 DEBUG 1712726609, 06:23:29 DEBUG "0" 06:23:29 DEBUG ] 06:23:29 DEBUG } 06:23:29 DEBUG ] 06:23:29 DEBUG } 06:23:29 DEBUG } 06:23:29 DEBUG { 06:23:29 DEBUG "metric": { 06:23:29 DEBUG "pod": "ds-idrepo-0" 06:23:29 DEBUG }, 06:23:29 DEBUG "value": [ 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG [...] 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG 06:23:29 DEBUG "pod": "ds-idrepo-1" 06:23:29 DEBUG }, 06:23:29 DEBUG "value": [ 06:23:29 DEBUG 1712726609, 06:23:29 DEBUG "0" 06:23:29 DEBUG ] 06:23:29 DEBUG } 06:23:29 DEBUG ] 06:23:29 DEBUG } 06:23:29 DEBUG } 06:23:29 DEBUG ] 06:23:29 DEBUG } 06:23:29 DEBUG ] 06:23:29 DEBUG } 06:23:29 DEBUG } 06:23:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726610" 06:23:30 INFO [http_cmd]: http status code OK 06:23:30 DEBUG --- status code --- 06:23:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:30 DEBUG --- http response --- 06:23:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:23:30 DEBUG ----- output ----- 06:23:30 DEBUG { 06:23:30 DEBUG "status": "success", 06:23:30 DEBUG "data": { 06:23:30 DEBUG "resultType": "vector", 06:23:30 DEBUG "result": [ 06:23:30 DEBUG { 06:23:30 DEBUG "metric": { 06:23:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:23:30 DEBUG }, 06:23:30 DEBUG "value": [ 06:23:30 DEBUG 06:23:30 DEBUG 06:23:30 DEBUG 06:23:30 DEBUG [...] 06:23:30 DEBUG 06:23:30 DEBUG 06:23:30 DEBUG 06:23:30 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:23:30 DEBUG }, 06:23:30 DEBUG "value": [ 06:23:30 DEBUG 1712726610, 06:23:30 DEBUG "111972.18232396022" 06:23:30 DEBUG ] 06:23:30 DEBUG } 06:23:30 DEBUG ] 06:23:30 DEBUG } 06:23:30 DEBUG } 06:23:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712726610" 06:23:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712726610" 06:23:30 INFO [http_cmd]: http status code OK 06:23:30 DEBUG --- status code --- 06:23:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:30 DEBUG --- http response --- 06:23:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:23:30 DEBUG ----- output ----- 06:23:30 DEBUG { 06:23:30 DEBUG "status": "success", 06:23:30 DEBUG "data": { 06:23:30 DEBUG "resultType": "vector", 06:23:30 DEBUG "result": [ 06:23:30 DEBUG { 06:23:30 DEBUG "metric": { 06:23:30 DEBUG "pod": "ds-idrepo-0" 06:23:30 DEBUG }, 06:23:30 DEBUG "value": [ 06:23:30 DEBUG 06:23:30 DEBUG 06:23:30 DEBUG 06:23:30 DEBUG [...] 06:23:30 DEBUG 06:23:30 DEBUG 06:23:30 DEBUG 06:23:30 DEBUG "pod": "ds-cts-2" 06:23:30 DEBUG }, 06:23:30 DEBUG "value": [ 06:23:30 DEBUG 1712726610, 06:23:30 DEBUG "0" 06:23:30 DEBUG ] 06:23:30 DEBUG } 06:23:30 DEBUG ] 06:23:30 DEBUG } 06:23:30 DEBUG } 06:23:30 INFO [http_cmd]: http status code OK 06:23:30 DEBUG --- status code --- 06:23:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:30 DEBUG --- http response --- 06:23:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:23:30 DEBUG ----- output ----- 06:23:30 DEBUG { 06:23:30 DEBUG "status": "success", 06:23:30 DEBUG "data": { 06:23:30 DEBUG "resultType": "vector", 06:23:30 DEBUG "result": [ 06:23:30 DEBUG { 06:23:30 DEBUG "metric": { 06:23:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:23:30 DEBUG }, 06:23:30 DEBUG "value": [ 06:23:30 DEBUG 06:23:30 DEBUG 06:23:30 DEBUG 06:23:30 DEBUG [...] 06:23:30 DEBUG 06:23:30 DEBUG 06:23:30 DEBUG 06:23:30 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:23:30 DEBUG }, 06:23:30 DEBUG "value": [ 06:23:30 DEBUG 1712726610, 06:23:30 DEBUG "4.227907543475477" 06:23:30 DEBUG ] 06:23:30 DEBUG } 06:23:30 DEBUG ] 06:23:30 DEBUG } 06:23:30 DEBUG } 06:23:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726610" 06:23:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726610" 06:23:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712726610" 06:23:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726610" 06:23:30 INFO [http_cmd]: http status code OK 06:23:30 DEBUG --- status code --- 06:23:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:30 DEBUG --- http response --- 06:23:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:23:30 DEBUG ----- output ----- 06:23:30 DEBUG { 06:23:30 DEBUG "status": "success", 06:23:30 DEBUG "data": { 06:23:30 DEBUG "resultType": "vector", 06:23:30 DEBUG "result": [ 06:23:30 DEBUG { 06:23:30 DEBUG "metric": { 06:23:30 DEBUG "pod": "ds-idrepo-2" 06:23:30 DEBUG }, 06:23:30 DEBUG "value": [ 06:23:30 DEBUG 06:23:30 DEBUG 06:23:30 DEBUG 06:23:30 DEBUG [...] 06:23:30 DEBUG 06:23:30 DEBUG 06:23:30 DEBUG 06:23:30 DEBUG "pod": "ds-cts-2" 06:23:30 DEBUG }, 06:23:30 DEBUG "value": [ 06:23:30 DEBUG 1712726610, 06:23:30 DEBUG "0.00017686555555555556" 06:23:30 DEBUG ] 06:23:30 DEBUG } 06:23:30 DEBUG ] 06:23:30 DEBUG } 06:23:30 DEBUG } 06:23:30 INFO [http_cmd]: http status code OK 06:23:30 DEBUG --- status code --- 06:23:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:30 DEBUG --- http response --- 06:23:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:23:30 DEBUG ----- output ----- 06:23:30 DEBUG { 06:23:30 DEBUG "status": "success", 06:23:30 DEBUG "data": { 06:23:30 DEBUG "resultType": "vector", 06:23:30 DEBUG "result": [ 06:23:30 DEBUG { 06:23:30 DEBUG "metric": { 06:23:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:23:30 DEBUG }, 06:23:30 DEBUG "value": [ 06:23:30 DEBUG 06:23:30 DEBUG 06:23:30 DEBUG 06:23:30 DEBUG [...] 06:23:30 DEBUG 06:23:30 DEBUG 06:23:30 DEBUG 06:23:30 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:23:30 DEBUG }, 06:23:30 DEBUG "value": [ 06:23:30 DEBUG 1712726610, 06:23:30 DEBUG "6.247946479814225" 06:23:30 DEBUG ] 06:23:30 DEBUG } 06:23:30 DEBUG ] 06:23:30 DEBUG } 06:23:30 DEBUG } 06:23:30 INFO [http_cmd]: http status code OK 06:23:30 DEBUG --- status code --- 06:23:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:30 DEBUG --- http response --- 06:23:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:23:30 DEBUG ----- output ----- 06:23:30 DEBUG { 06:23:30 DEBUG "status": "success", 06:23:30 DEBUG "data": { 06:23:30 DEBUG "resultType": "vector", 06:23:30 DEBUG "result": [ 06:23:30 DEBUG { 06:23:30 DEBUG "metric": { 06:23:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:23:30 DEBUG }, 06:23:30 DEBUG "value": [ 06:23:30 DEBUG 06:23:30 DEBUG 06:23:30 DEBUG 06:23:30 DEBUG [...] 06:23:30 DEBUG 06:23:30 DEBUG 06:23:30 DEBUG 06:23:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:23:30 DEBUG }, 06:23:30 DEBUG "value": [ 06:23:30 DEBUG 1712726610, 06:23:30 DEBUG "0.051" 06:23:30 DEBUG ] 06:23:30 DEBUG } 06:23:30 DEBUG ] 06:23:30 DEBUG } 06:23:30 DEBUG } 06:23:30 INFO [http_cmd]: http status code OK 06:23:30 DEBUG --- status code --- 06:23:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:30 DEBUG --- http response --- 06:23:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:23:30 DEBUG ----- output ----- 06:23:30 DEBUG { 06:23:30 DEBUG "status": "success", 06:23:30 DEBUG "data": { 06:23:30 DEBUG "resultType": "vector", 06:23:30 DEBUG "result": [ 06:23:30 DEBUG { 06:23:30 DEBUG "metric": { 06:23:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:23:30 DEBUG }, 06:23:30 DEBUG "value": [ 06:23:30 DEBUG 06:23:30 DEBUG 06:23:30 DEBUG 06:23:30 DEBUG [...] 06:23:30 DEBUG 06:23:30 DEBUG 06:23:30 DEBUG 06:23:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:23:30 DEBUG }, 06:23:30 DEBUG "value": [ 06:23:30 DEBUG 1712726610, 06:23:30 DEBUG "1445.6666666666667" 06:23:30 DEBUG ] 06:23:30 DEBUG } 06:23:30 DEBUG ] 06:23:30 DEBUG } 06:23:30 DEBUG } 06:23:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726611" 06:23:31 INFO [http_cmd]: http status code OK 06:23:31 DEBUG --- status code --- 06:23:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:31 DEBUG --- http response --- 06:23:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:23:31 DEBUG ----- output ----- 06:23:31 DEBUG { 06:23:31 DEBUG "status": "success", 06:23:31 DEBUG "data": { 06:23:31 DEBUG "resultType": "vector", 06:23:31 DEBUG "result": [ 06:23:31 DEBUG { 06:23:31 DEBUG "metric": { 06:23:31 DEBUG "pod": "ds-idrepo-0" 06:23:31 DEBUG }, 06:23:31 DEBUG "value": [ 06:23:31 DEBUG 06:23:31 DEBUG 06:23:31 DEBUG 06:23:31 DEBUG [...] 06:23:31 DEBUG 06:23:31 DEBUG 06:23:31 DEBUG 06:23:31 DEBUG "pod": "ds-idrepo-1" 06:23:31 DEBUG }, 06:23:31 DEBUG "value": [ 06:23:31 DEBUG 1712726611, 06:23:31 DEBUG "0" 06:23:31 DEBUG ] 06:23:31 DEBUG } 06:23:31 DEBUG ] 06:23:31 DEBUG } 06:23:31 DEBUG } 06:23:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726611" 06:23:31 INFO [http_cmd]: http status code OK 06:23:31 DEBUG --- status code --- 06:23:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:31 DEBUG --- http response --- 06:23:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:23:31 DEBUG ----- output ----- 06:23:31 DEBUG { 06:23:31 DEBUG "status": "success", 06:23:31 DEBUG "data": { 06:23:31 DEBUG "resultType": "vector", 06:23:31 DEBUG "result": [ 06:23:31 DEBUG { 06:23:31 DEBUG "metric": { 06:23:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:23:31 DEBUG }, 06:23:31 DEBUG "value": [ 06:23:31 DEBUG 06:23:31 DEBUG 06:23:31 DEBUG 06:23:31 DEBUG [...] 06:23:31 DEBUG 06:23:31 DEBUG 06:23:31 DEBUG 06:23:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:23:31 DEBUG }, 06:23:31 DEBUG "value": [ 06:23:31 DEBUG 1712726611, 06:23:31 DEBUG "1438.2" 06:23:31 DEBUG ] 06:23:31 DEBUG } 06:23:31 DEBUG ] 06:23:31 DEBUG } 06:23:31 DEBUG } 06:23:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726611" 06:23:31 INFO [http_cmd]: http status code OK 06:23:31 DEBUG --- status code --- 06:23:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:31 DEBUG --- http response --- 06:23:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:23:31 DEBUG ----- output ----- 06:23:31 DEBUG { 06:23:31 DEBUG "status": "success", 06:23:31 DEBUG "data": { 06:23:31 DEBUG "resultType": "vector", 06:23:31 DEBUG "result": [ 06:23:31 DEBUG { 06:23:31 DEBUG "metric": { 06:23:31 DEBUG "pod": "ds-idrepo-0" 06:23:31 DEBUG }, 06:23:31 DEBUG "value": [ 06:23:31 DEBUG 06:23:31 DEBUG 06:23:31 DEBUG 06:23:31 DEBUG [...] 06:23:31 DEBUG 06:23:31 DEBUG 06:23:31 DEBUG 06:23:31 DEBUG "pod": "ds-idrepo-1" 06:23:31 DEBUG }, 06:23:31 DEBUG "value": [ 06:23:31 DEBUG 1712726611, 06:23:31 DEBUG "0" 06:23:31 DEBUG ] 06:23:31 DEBUG } 06:23:31 DEBUG ] 06:23:31 DEBUG } 06:23:31 DEBUG } 06:23:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726611" 06:23:31 INFO [http_cmd]: http status code OK 06:23:31 DEBUG --- status code --- 06:23:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:31 DEBUG --- http response --- 06:23:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:23:31 DEBUG ----- output ----- 06:23:31 DEBUG { 06:23:31 DEBUG "status": "success", 06:23:31 DEBUG "data": { 06:23:31 DEBUG "resultType": "vector", 06:23:31 DEBUG "result": [ 06:23:31 DEBUG { 06:23:31 DEBUG "metric": { 06:23:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:23:31 DEBUG }, 06:23:31 DEBUG "value": [ 06:23:31 DEBUG 06:23:31 DEBUG 06:23:31 DEBUG 06:23:31 DEBUG [...] 06:23:31 DEBUG 06:23:31 DEBUG 06:23:31 DEBUG 06:23:31 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:23:31 DEBUG }, 06:23:31 DEBUG "value": [ 06:23:31 DEBUG 1712726611, 06:23:31 DEBUG "0" 06:23:31 DEBUG ] 06:23:31 DEBUG } 06:23:31 DEBUG ] 06:23:31 DEBUG } 06:23:31 DEBUG } 06:23:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712726611" 06:23:31 INFO [http_cmd]: http status code OK 06:23:31 DEBUG --- status code --- 06:23:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:31 DEBUG --- http response --- 06:23:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:23:31 DEBUG ----- output ----- 06:23:31 DEBUG { 06:23:31 DEBUG "status": "success", 06:23:31 DEBUG "data": { 06:23:31 DEBUG "resultType": "vector", 06:23:31 DEBUG "result": [ 06:23:31 DEBUG { 06:23:31 DEBUG "metric": { 06:23:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:23:31 DEBUG }, 06:23:31 DEBUG "value": [ 06:23:31 DEBUG 06:23:31 DEBUG 06:23:31 DEBUG 06:23:31 DEBUG [...] 06:23:31 DEBUG 06:23:31 DEBUG 06:23:31 DEBUG 06:23:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:23:31 DEBUG }, 06:23:31 DEBUG "value": [ 06:23:31 DEBUG 1712726611, 06:23:31 DEBUG "2486181888" 06:23:31 DEBUG ] 06:23:31 DEBUG } 06:23:31 DEBUG ] 06:23:31 DEBUG } 06:23:31 DEBUG } 06:23:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726611" 06:23:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726611" 06:23:31 INFO [http_cmd]: http status code OK 06:23:31 DEBUG --- status code --- 06:23:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:31 DEBUG --- http response --- 06:23:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:23:31 DEBUG ----- output ----- 06:23:31 DEBUG { 06:23:31 DEBUG "status": "success", 06:23:31 DEBUG "data": { 06:23:31 DEBUG "resultType": "vector", 06:23:31 DEBUG "result": [ 06:23:31 DEBUG { 06:23:31 DEBUG "metric": { 06:23:31 DEBUG "instance": "10.142.0.25:9100" 06:23:31 DEBUG }, 06:23:31 DEBUG "value": [ 06:23:31 DEBUG 06:23:31 DEBUG 06:23:31 DEBUG 06:23:31 DEBUG [...] 06:23:31 DEBUG 06:23:31 DEBUG 06:23:31 DEBUG 06:23:31 DEBUG "instance": "10.142.0.77:9100" 06:23:31 DEBUG }, 06:23:31 DEBUG "value": [ 06:23:31 DEBUG 1712726611, 06:23:31 DEBUG "42763101.03333333" 06:23:31 DEBUG ] 06:23:31 DEBUG } 06:23:31 DEBUG ] 06:23:31 DEBUG } 06:23:31 DEBUG } 06:23:31 INFO [http_cmd]: http status code OK 06:23:31 DEBUG --- status code --- 06:23:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:23:31 DEBUG --- http response --- 06:23:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:23:31 DEBUG ----- output ----- 06:23:31 DEBUG { 06:23:31 DEBUG "status": "success", 06:23:31 DEBUG "data": { 06:23:31 DEBUG "resultType": "vector", 06:23:31 DEBUG "result": [ 06:23:31 DEBUG { 06:23:31 DEBUG "metric": { 06:23:31 DEBUG "instance": "10.142.0.25:9100" 06:23:31 DEBUG }, 06:23:31 DEBUG "value": [ 06:23:31 DEBUG 06:23:31 DEBUG 06:23:31 DEBUG 06:23:31 DEBUG [...] 06:23:31 DEBUG 06:23:31 DEBUG 06:23:31 DEBUG 06:23:31 DEBUG "instance": "10.142.0.77:9100" 06:23:31 DEBUG }, 06:23:31 DEBUG "value": [ 06:23:31 DEBUG 1712726611, 06:23:31 DEBUG "0.019999999999999334" 06:23:31 DEBUG ] 06:23:31 DEBUG } 06:23:31 DEBUG ] 06:23:31 DEBUG } 06:23:31 DEBUG } 06:23:50 INFO 06:23:50 INFO [loop_until]: kubectl --namespace=xlou top pods 06:23:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:23:50 INFO [loop_until]: OK (rc = 0) 06:23:50 DEBUG --- stdout --- 06:23:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10761m 8983Mi am-654c4fcbcf-9scrj 9871m 8454Mi am-654c4fcbcf-v5qb5 8438m 8447Mi ds-cts-0 8506m 13121Mi ds-cts-1 8354m 12828Mi ds-cts-2 7391m 13046Mi ds-idrepo-0 10642m 5622Mi ds-idrepo-1 369m 5672Mi ds-idrepo-2 391m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 6m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4167m 2355Mi 06:23:50 DEBUG --- stderr --- 06:23:50 DEBUG 06:23:58 INFO 06:23:58 INFO [loop_until]: kubectl --namespace=xlou top node 06:23:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:23:58 INFO [loop_until]: OK (rc = 0) 06:23:58 DEBUG --- stdout --- 06:23:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 96m 0% 2304Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9265m 58% 9331Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 76m 0% 3257Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13386m 84% 6695Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 11024m 69% 9332Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11571m 72% 9802Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 74m 0% 856Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10906m 68% 6303Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8523m 53% 13835Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 7611m 47% 13873Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 432m 2% 6389Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8540m 53% 13670Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 426m 2% 6353Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4369m 27% 5961Mi 10% 06:23:58 DEBUG --- stderr --- 06:23:58 DEBUG 06:24:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726668" 06:24:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712726668" 06:24:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726668" 06:24:28 ERROR [http_cmd]: ERROR 06:24:28 ERROR --- status code --- 06:24:28 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:24:28 ERROR --- http response --- 06:24:28 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:24:28 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:24:28 INFO [http_cmd]: http status code OK 06:24:28 DEBUG --- status code --- 06:24:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:28 DEBUG --- http response --- 06:24:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:24:28 DEBUG ----- output ----- 06:24:28 DEBUG { 06:24:28 DEBUG "status": "success", 06:24:28 DEBUG "data": { 06:24:28 DEBUG "resultType": "vector", 06:24:28 DEBUG "result": [ 06:24:28 DEBUG { 06:24:28 DEBUG "metric": { 06:24:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:24:28 DEBUG }, 06:24:28 DEBUG "value": [ 06:24:28 DEBUG 06:24:28 DEBUG 06:24:28 DEBUG 06:24:28 DEBUG [...] 06:24:28 DEBUG 06:24:28 DEBUG 06:24:28 DEBUG 06:24:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:24:28 DEBUG }, 06:24:28 DEBUG "value": [ 06:24:28 DEBUG 1712726668, 06:24:28 DEBUG "0" 06:24:28 DEBUG ] 06:24:28 DEBUG } 06:24:28 DEBUG ] 06:24:28 DEBUG } 06:24:28 DEBUG } 06:24:28 INFO [http_cmd]: http status code OK 06:24:28 DEBUG --- status code --- 06:24:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:28 DEBUG --- http response --- 06:24:28 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:24:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726669" 06:24:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726669" 06:24:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726669" 06:24:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726669" 06:24:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726669" 06:24:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726669" 06:24:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726669" 06:24:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726669" 06:24:29 INFO [http_cmd]: http status code OK 06:24:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726669" 06:24:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726669" 06:24:29 DEBUG --- status code --- 06:24:29 INFO [http_cmd]: http status code OK 06:24:29 INFO [http_cmd]: http status code OK 06:24:29 DEBUG --- status code --- 06:24:29 INFO [http_cmd]: http status code OK 06:24:29 DEBUG --- status code --- 06:24:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:29 DEBUG --- http response --- 06:24:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:29 DEBUG --- http response --- 06:24:29 INFO [http_cmd]: http status code OK 06:24:29 DEBUG --- status code --- 06:24:29 INFO [http_cmd]: http status code OK 06:24:29 INFO [http_cmd]: http status code OK 06:24:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:24:29 DEBUG ----- output ----- 06:24:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:29 DEBUG --- http response --- 06:24:29 DEBUG --- status code --- 06:24:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:29 DEBUG --- status code --- 06:24:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:29 INFO [http_cmd]: http status code OK 06:24:29 DEBUG --- status code --- 06:24:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:24:29 DEBUG ----- output ----- 06:24:29 DEBUG { 06:24:29 INFO [http_cmd]: http status code OK 06:24:29 DEBUG --- http response --- 06:24:29 DEBUG --- status code --- 06:24:29 DEBUG --- http response --- 06:24:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:29 DEBUG { 06:24:29 DEBUG "status": "success", 06:24:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:24:29 DEBUG --- status code --- 06:24:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:29 DEBUG --- http response --- 06:24:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:29 INFO [http_cmd]: http status code OK 06:24:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:24:29 DEBUG --- http response --- 06:24:29 DEBUG "data": { 06:24:29 DEBUG "status": "success", 06:24:29 DEBUG ----- output ----- 06:24:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:24:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:24:29 DEBUG --- http response --- 06:24:29 DEBUG --- status code --- 06:24:29 DEBUG ----- output ----- 06:24:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:24:29 DEBUG "resultType": "vector", 06:24:29 DEBUG "data": { 06:24:29 DEBUG { 06:24:29 DEBUG --- http response --- 06:24:29 DEBUG ----- output ----- 06:24:29 DEBUG ----- output ----- 06:24:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:24:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:29 DEBUG { 06:24:29 DEBUG ----- output ----- 06:24:29 DEBUG "result": [ 06:24:29 DEBUG "resultType": "vector", 06:24:29 DEBUG "status": "success", 06:24:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:24:29 DEBUG { 06:24:29 DEBUG { 06:24:29 DEBUG "status": "success", 06:24:29 DEBUG --- http response --- 06:24:29 DEBUG "status": "success", 06:24:29 DEBUG { 06:24:29 DEBUG { 06:24:29 DEBUG "result": [ 06:24:29 DEBUG "data": { 06:24:29 DEBUG ----- output ----- 06:24:29 DEBUG "status": "success", 06:24:29 DEBUG ----- output ----- 06:24:29 DEBUG "data": { 06:24:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:24:29 DEBUG "data": { 06:24:29 DEBUG "status": "success", 06:24:29 DEBUG "metric": { 06:24:29 DEBUG { 06:24:29 DEBUG "resultType": "vector", 06:24:29 DEBUG "result": [ 06:24:29 DEBUG { 06:24:29 DEBUG "metric": { 06:24:29 DEBUG "resultType": "vector", 06:24:29 DEBUG "result": [ 06:24:29 DEBUG "resultType": "vector", 06:24:29 DEBUG "data": { 06:24:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:24:29 DEBUG "metric": { 06:24:29 DEBUG { 06:24:29 DEBUG "instance": "10.142.0.25:9100" 06:24:29 DEBUG { 06:24:29 DEBUG "data": { 06:24:29 DEBUG ----- output ----- 06:24:29 DEBUG { 06:24:29 DEBUG "result": [ 06:24:29 DEBUG "resultType": "vector", 06:24:29 DEBUG }, 06:24:29 DEBUG "pod": "ds-idrepo-0" 06:24:29 DEBUG "status": "success", 06:24:29 DEBUG }, 06:24:29 DEBUG "status": "success", 06:24:29 DEBUG "resultType": "vector", 06:24:29 DEBUG { 06:24:29 DEBUG "status": "success", 06:24:29 DEBUG "data": { 06:24:29 DEBUG "result": [ 06:24:29 DEBUG "value": [ 06:24:29 DEBUG }, 06:24:29 DEBUG "data": { 06:24:29 DEBUG "resultType": "vector", 06:24:29 DEBUG "result": [ 06:24:29 DEBUG { 06:24:29 DEBUG "metric": { 06:24:29 DEBUG "resultType": "vector", 06:24:29 DEBUG "result": [ 06:24:29 DEBUG { 06:24:29 DEBUG 06:24:29 DEBUG "value": [ 06:24:29 DEBUG "value": [ 06:24:29 DEBUG 06:24:29 DEBUG 06:24:29 DEBUG 06:24:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:24:29 DEBUG { 06:24:29 DEBUG { 06:24:29 DEBUG "metric": { 06:24:29 DEBUG 06:24:29 DEBUG 06:24:29 DEBUG "metric": { 06:24:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:24:29 DEBUG }, 06:24:29 DEBUG "value": [ 06:24:29 DEBUG }, 06:24:29 DEBUG [...] 06:24:29 DEBUG "metric": { 06:24:29 DEBUG "pod": "ds-idrepo-0" 06:24:29 DEBUG }, 06:24:29 DEBUG 06:24:29 DEBUG 06:24:29 DEBUG 06:24:29 DEBUG 06:24:29 DEBUG [...] 06:24:29 DEBUG "value": [ 06:24:29 DEBUG 06:24:29 DEBUG 06:24:29 DEBUG "pod": "ds-idrepo-0" 06:24:29 DEBUG 06:24:29 DEBUG 06:24:29 DEBUG [...] 06:24:29 DEBUG 06:24:29 DEBUG 06:24:29 DEBUG "data": { 06:24:29 DEBUG 06:24:29 DEBUG 06:24:29 DEBUG "value": [ 06:24:29 DEBUG }, 06:24:29 DEBUG [...] 06:24:29 DEBUG 06:24:29 DEBUG 06:24:29 DEBUG 06:24:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:24:29 DEBUG }, 06:24:29 DEBUG "value": [ 06:24:29 DEBUG 06:24:29 DEBUG 06:24:29 DEBUG "value": [ 06:24:29 DEBUG 1712726669, 06:24:29 DEBUG "1440.1226544012266" 06:24:29 DEBUG ] 06:24:29 DEBUG } 06:24:29 DEBUG ] 06:24:29 DEBUG "resultType": "vector", 06:24:29 DEBUG 06:24:29 DEBUG [...] 06:24:29 DEBUG 06:24:29 DEBUG 06:24:29 DEBUG } 06:24:29 DEBUG } 06:24:29 DEBUG "metric": { 06:24:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:24:29 DEBUG }, 06:24:29 DEBUG "value": [ 06:24:29 DEBUG 06:24:29 DEBUG 06:24:29 DEBUG 06:24:29 DEBUG 06:24:29 DEBUG 06:24:29 DEBUG 06:24:29 DEBUG 06:24:29 DEBUG 06:24:29 DEBUG "result": [ 06:24:29 DEBUG { 06:24:29 DEBUG 06:24:29 DEBUG [...] 06:24:29 DEBUG "result": [ 06:24:29 DEBUG { 06:24:29 DEBUG [...] 06:24:29 DEBUG 06:24:29 DEBUG "pod": "ds-cts-2" 06:24:29 DEBUG "metric": { 06:24:29 DEBUG "instance": "10.142.0.25:9100" 06:24:29 DEBUG 06:24:29 DEBUG 06:24:29 DEBUG "metric": { 06:24:29 DEBUG "instance": "10.142.0.25:9100" 06:24:29 DEBUG 06:24:29 DEBUG 06:24:29 DEBUG }, 06:24:29 DEBUG }, 06:24:29 DEBUG "value": [ 06:24:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:24:29 DEBUG 06:24:29 DEBUG }, 06:24:29 DEBUG "value": [ 06:24:29 DEBUG 06:24:29 DEBUG 06:24:29 DEBUG "value": [ 06:24:29 DEBUG 06:24:29 DEBUG "instance": "10.142.0.77:9100" 06:24:29 DEBUG }, 06:24:29 DEBUG 06:24:29 DEBUG 06:24:29 DEBUG 06:24:29 DEBUG [...] 06:24:29 DEBUG 06:24:29 DEBUG 1712726669, 06:24:29 DEBUG "0" 06:24:29 DEBUG }, 06:24:29 DEBUG "value": [ 06:24:29 DEBUG "pod": "ds-cts-2" 06:24:29 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:24:29 DEBUG 06:24:29 DEBUG 06:24:29 DEBUG 06:24:29 DEBUG 06:24:29 DEBUG ] 06:24:29 DEBUG 1712726669, 06:24:29 DEBUG "value": [ 06:24:29 DEBUG 1712726669, 06:24:29 DEBUG }, 06:24:29 DEBUG [...] 06:24:29 DEBUG 06:24:29 DEBUG 06:24:29 DEBUG "pod": "ds-idrepo-1" 06:24:29 DEBUG }, 06:24:29 DEBUG "42749480.93333334" 06:24:29 DEBUG }, 06:24:29 DEBUG "value": [ 06:24:29 DEBUG "value": [ 06:24:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:24:29 DEBUG 06:24:29 DEBUG 06:24:29 DEBUG } 06:24:29 DEBUG "value": [ 06:24:29 DEBUG ] 06:24:29 DEBUG "1440.05599440056" 06:24:29 DEBUG ] 06:24:29 DEBUG } 06:24:29 DEBUG ] 06:24:29 DEBUG } 06:24:29 DEBUG [...] 06:24:29 DEBUG ] 06:24:29 DEBUG } 06:24:29 DEBUG } 06:24:29 DEBUG 1712726669, 06:24:29 DEBUG "0" 06:24:29 DEBUG ] 06:24:29 DEBUG } 06:24:29 DEBUG ] 06:24:29 DEBUG 06:24:29 DEBUG } 06:24:29 DEBUG 1712726669, 06:24:29 DEBUG ] 06:24:29 DEBUG 06:24:29 DEBUG "instance": "10.142.0.77:9100" 06:24:29 DEBUG }, 06:24:29 DEBUG "value": [ 06:24:29 DEBUG 1712726669, 06:24:29 DEBUG 06:24:29 DEBUG "0" 06:24:29 DEBUG } 06:24:29 DEBUG "0.1551666666666699" 06:24:29 DEBUG ] 06:24:29 DEBUG } 06:24:29 DEBUG ] 06:24:29 DEBUG } 06:24:29 DEBUG 06:24:29 DEBUG ] 06:24:29 DEBUG } 06:24:29 DEBUG } 06:24:29 DEBUG } 06:24:29 DEBUG } 06:24:29 DEBUG 1712726669, 06:24:29 DEBUG "29.4" 06:24:29 DEBUG "instance": "10.142.0.77:9100" 06:24:29 DEBUG }, 06:24:29 DEBUG }, 06:24:29 DEBUG "value": [ 06:24:29 DEBUG 1712726669, 06:24:29 DEBUG "258612.5" 06:24:29 DEBUG "value": [ 06:24:29 DEBUG 1712726669, 06:24:29 DEBUG "390098816000" 06:24:29 DEBUG ] 06:24:29 DEBUG } 06:24:29 DEBUG ] 06:24:29 DEBUG } 06:24:29 DEBUG } 06:24:29 DEBUG ] 06:24:29 DEBUG } 06:24:29 DEBUG ] 06:24:29 DEBUG } 06:24:29 DEBUG } 06:24:29 DEBUG } 06:24:29 DEBUG ] 06:24:29 DEBUG } 06:24:29 DEBUG ] 06:24:29 DEBUG ] 06:24:29 DEBUG } 06:24:29 DEBUG } 06:24:29 DEBUG } 06:24:29 DEBUG } 06:24:29 DEBUG } 06:24:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726670" 06:24:30 INFO [http_cmd]: http status code OK 06:24:30 DEBUG --- status code --- 06:24:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:30 DEBUG --- http response --- 06:24:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:24:30 DEBUG ----- output ----- 06:24:30 DEBUG { 06:24:30 DEBUG "status": "success", 06:24:30 DEBUG "data": { 06:24:30 DEBUG "resultType": "vector", 06:24:30 DEBUG "result": [ 06:24:30 DEBUG { 06:24:30 DEBUG "metric": { 06:24:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:24:30 DEBUG }, 06:24:30 DEBUG "value": [ 06:24:30 DEBUG 06:24:30 DEBUG 06:24:30 DEBUG 06:24:30 DEBUG [...] 06:24:30 DEBUG 06:24:30 DEBUG 06:24:30 DEBUG 06:24:30 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:24:30 DEBUG }, 06:24:30 DEBUG "value": [ 06:24:30 DEBUG 1712726670, 06:24:30 DEBUG "67710215.40943089" 06:24:30 DEBUG ] 06:24:30 DEBUG } 06:24:30 DEBUG ] 06:24:30 DEBUG } 06:24:30 DEBUG } 06:24:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712726670" 06:24:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712726670" 06:24:30 INFO [http_cmd]: http status code OK 06:24:30 DEBUG --- status code --- 06:24:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:30 DEBUG --- http response --- 06:24:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:24:30 DEBUG ----- output ----- 06:24:30 DEBUG { 06:24:30 DEBUG "status": "success", 06:24:30 DEBUG "data": { 06:24:30 DEBUG "resultType": "vector", 06:24:30 DEBUG "result": [ 06:24:30 DEBUG { 06:24:30 DEBUG "metric": { 06:24:30 DEBUG "pod": "ds-idrepo-0" 06:24:30 DEBUG }, 06:24:30 DEBUG "value": [ 06:24:30 DEBUG 06:24:30 DEBUG 06:24:30 DEBUG 06:24:30 DEBUG [...] 06:24:30 DEBUG 06:24:30 DEBUG 06:24:30 DEBUG 06:24:30 DEBUG "pod": "ds-cts-2" 06:24:30 DEBUG }, 06:24:30 DEBUG "value": [ 06:24:30 DEBUG 1712726670, 06:24:30 DEBUG "0" 06:24:30 DEBUG ] 06:24:30 DEBUG } 06:24:30 DEBUG ] 06:24:30 DEBUG } 06:24:30 DEBUG } 06:24:30 INFO [http_cmd]: http status code OK 06:24:30 DEBUG --- status code --- 06:24:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:30 DEBUG --- http response --- 06:24:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:24:30 DEBUG ----- output ----- 06:24:30 DEBUG { 06:24:30 DEBUG "status": "success", 06:24:30 DEBUG "data": { 06:24:30 DEBUG "resultType": "vector", 06:24:30 DEBUG "result": [ 06:24:30 DEBUG { 06:24:30 DEBUG "metric": { 06:24:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:24:30 DEBUG }, 06:24:30 DEBUG "value": [ 06:24:30 DEBUG 06:24:30 DEBUG 06:24:30 DEBUG 06:24:30 DEBUG [...] 06:24:30 DEBUG 06:24:30 DEBUG 06:24:30 DEBUG 06:24:30 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:24:30 DEBUG }, 06:24:30 DEBUG "value": [ 06:24:30 DEBUG 1712726670, 06:24:30 DEBUG "4.266293737626149" 06:24:30 DEBUG ] 06:24:30 DEBUG } 06:24:30 DEBUG ] 06:24:30 DEBUG } 06:24:30 DEBUG } 06:24:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726670" 06:24:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726670" 06:24:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712726670" 06:24:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726670" 06:24:30 INFO [http_cmd]: http status code OK 06:24:30 DEBUG --- status code --- 06:24:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:30 DEBUG --- http response --- 06:24:30 INFO [http_cmd]: http status code OK 06:24:30 INFO [http_cmd]: http status code OK 06:24:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:24:30 INFO [http_cmd]: http status code OK 06:24:30 DEBUG --- status code --- 06:24:30 DEBUG --- status code --- 06:24:30 DEBUG ----- output ----- 06:24:30 DEBUG --- status code --- 06:24:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:30 DEBUG { 06:24:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:30 DEBUG --- http response --- 06:24:30 DEBUG --- http response --- 06:24:30 DEBUG "status": "success", 06:24:30 DEBUG --- http response --- 06:24:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:24:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:24:30 DEBUG "data": { 06:24:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:24:30 DEBUG ----- output ----- 06:24:30 DEBUG { 06:24:30 DEBUG "status": "success", 06:24:30 DEBUG ----- output ----- 06:24:30 DEBUG ----- output ----- 06:24:30 DEBUG { 06:24:30 DEBUG "status": "success", 06:24:30 DEBUG "data": { 06:24:30 DEBUG { 06:24:30 DEBUG "resultType": "vector", 06:24:30 DEBUG "result": [ 06:24:30 DEBUG { 06:24:30 DEBUG "status": "success", 06:24:30 DEBUG "metric": { 06:24:30 DEBUG "data": { 06:24:30 DEBUG "resultType": "vector", 06:24:30 DEBUG "data": { 06:24:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:24:30 DEBUG "result": [ 06:24:30 DEBUG { 06:24:30 DEBUG "metric": { 06:24:30 DEBUG }, 06:24:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:24:30 DEBUG }, 06:24:30 DEBUG "resultType": "vector", 06:24:30 DEBUG "value": [ 06:24:30 DEBUG "resultType": "vector", 06:24:30 DEBUG "value": [ 06:24:30 DEBUG "result": [ 06:24:30 DEBUG 06:24:30 DEBUG "result": [ 06:24:30 DEBUG { 06:24:30 DEBUG "metric": { 06:24:30 DEBUG 06:24:30 DEBUG { 06:24:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:24:30 DEBUG }, 06:24:30 DEBUG 06:24:30 DEBUG "metric": { 06:24:30 DEBUG "value": [ 06:24:30 DEBUG 06:24:30 DEBUG [...] 06:24:30 DEBUG "pod": "ds-idrepo-2" 06:24:30 DEBUG 06:24:30 DEBUG 06:24:30 DEBUG 06:24:30 DEBUG }, 06:24:30 DEBUG [...] 06:24:30 DEBUG 06:24:30 DEBUG 06:24:30 DEBUG "value": [ 06:24:30 DEBUG 06:24:30 DEBUG 06:24:30 DEBUG 06:24:30 DEBUG 06:24:30 DEBUG 06:24:30 DEBUG 06:24:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:24:30 DEBUG }, 06:24:30 DEBUG 06:24:30 DEBUG [...] 06:24:30 DEBUG "value": [ 06:24:30 DEBUG 1712726670, 06:24:30 DEBUG "0.051" 06:24:30 DEBUG 06:24:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:24:30 DEBUG ] 06:24:30 DEBUG } 06:24:30 DEBUG 06:24:30 DEBUG }, 06:24:30 DEBUG ] 06:24:30 DEBUG } 06:24:30 DEBUG 06:24:30 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:24:30 DEBUG }, 06:24:30 DEBUG 06:24:30 DEBUG 06:24:30 DEBUG "value": [ 06:24:30 DEBUG "value": [ 06:24:30 DEBUG [...] 06:24:30 DEBUG 06:24:30 DEBUG 1712726670, 06:24:30 DEBUG 1712726670, 06:24:30 DEBUG 06:24:30 DEBUG 06:24:30 DEBUG "1.7097876345472773" 06:24:30 DEBUG ] 06:24:30 DEBUG } 06:24:30 DEBUG } 06:24:30 DEBUG "1439.9226743992267" 06:24:30 DEBUG ] 06:24:30 DEBUG ] 06:24:30 DEBUG "pod": "ds-cts-2" 06:24:30 DEBUG }, 06:24:30 DEBUG } 06:24:30 DEBUG } 06:24:30 DEBUG "value": [ 06:24:30 DEBUG } 06:24:30 DEBUG ] 06:24:30 DEBUG } 06:24:30 DEBUG 1712726670, 06:24:30 DEBUG } 06:24:30 DEBUG "0.00004577111111111111" 06:24:30 DEBUG ] 06:24:30 DEBUG } 06:24:30 DEBUG ] 06:24:30 DEBUG } 06:24:30 DEBUG } 06:24:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726671" 06:24:31 INFO [http_cmd]: http status code OK 06:24:31 DEBUG --- status code --- 06:24:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:31 DEBUG --- http response --- 06:24:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:24:31 DEBUG ----- output ----- 06:24:31 DEBUG { 06:24:31 DEBUG "status": "success", 06:24:31 DEBUG "data": { 06:24:31 DEBUG "resultType": "vector", 06:24:31 DEBUG "result": [ 06:24:31 DEBUG { 06:24:31 DEBUG "metric": { 06:24:31 DEBUG "pod": "ds-idrepo-0" 06:24:31 DEBUG }, 06:24:31 DEBUG "value": [ 06:24:31 DEBUG 06:24:31 DEBUG 06:24:31 DEBUG 06:24:31 DEBUG [...] 06:24:31 DEBUG 06:24:31 DEBUG 06:24:31 DEBUG 06:24:31 DEBUG "pod": "ds-idrepo-1" 06:24:31 DEBUG }, 06:24:31 DEBUG "value": [ 06:24:31 DEBUG 1712726671, 06:24:31 DEBUG "0" 06:24:31 DEBUG ] 06:24:31 DEBUG } 06:24:31 DEBUG ] 06:24:31 DEBUG } 06:24:31 DEBUG } 06:24:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726671" 06:24:31 INFO [http_cmd]: http status code OK 06:24:31 DEBUG --- status code --- 06:24:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:31 DEBUG --- http response --- 06:24:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:24:31 DEBUG ----- output ----- 06:24:31 DEBUG { 06:24:31 DEBUG "status": "success", 06:24:31 DEBUG "data": { 06:24:31 DEBUG "resultType": "vector", 06:24:31 DEBUG "result": [ 06:24:31 DEBUG { 06:24:31 DEBUG "metric": { 06:24:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:24:31 DEBUG }, 06:24:31 DEBUG "value": [ 06:24:31 DEBUG 06:24:31 DEBUG 06:24:31 DEBUG 06:24:31 DEBUG [...] 06:24:31 DEBUG 06:24:31 DEBUG 06:24:31 DEBUG 06:24:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:24:31 DEBUG }, 06:24:31 DEBUG "value": [ 06:24:31 DEBUG 1712726671, 06:24:31 DEBUG "1435.7564243575644" 06:24:31 DEBUG ] 06:24:31 DEBUG } 06:24:31 DEBUG ] 06:24:31 DEBUG } 06:24:31 DEBUG } 06:24:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726671" 06:24:31 INFO [http_cmd]: http status code OK 06:24:31 DEBUG --- status code --- 06:24:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:31 DEBUG --- http response --- 06:24:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:24:31 DEBUG ----- output ----- 06:24:31 DEBUG { 06:24:31 DEBUG "status": "success", 06:24:31 DEBUG "data": { 06:24:31 DEBUG "resultType": "vector", 06:24:31 DEBUG "result": [ 06:24:31 DEBUG { 06:24:31 DEBUG "metric": { 06:24:31 DEBUG "pod": "ds-idrepo-0" 06:24:31 DEBUG }, 06:24:31 DEBUG "value": [ 06:24:31 DEBUG 06:24:31 DEBUG 06:24:31 DEBUG 06:24:31 DEBUG [...] 06:24:31 DEBUG 06:24:31 DEBUG 06:24:31 DEBUG 06:24:31 DEBUG "pod": "ds-idrepo-1" 06:24:31 DEBUG }, 06:24:31 DEBUG "value": [ 06:24:31 DEBUG 1712726671, 06:24:31 DEBUG "0" 06:24:31 DEBUG ] 06:24:31 DEBUG } 06:24:31 DEBUG ] 06:24:31 DEBUG } 06:24:31 DEBUG } 06:24:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726671" 06:24:31 INFO [http_cmd]: http status code OK 06:24:31 DEBUG --- status code --- 06:24:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:31 DEBUG --- http response --- 06:24:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:24:31 DEBUG ----- output ----- 06:24:31 DEBUG { 06:24:31 DEBUG "status": "success", 06:24:31 DEBUG "data": { 06:24:31 DEBUG "resultType": "vector", 06:24:31 DEBUG "result": [ 06:24:31 DEBUG { 06:24:31 DEBUG "metric": { 06:24:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:24:31 DEBUG }, 06:24:31 DEBUG "value": [ 06:24:31 DEBUG 06:24:31 DEBUG 06:24:31 DEBUG 06:24:31 DEBUG [...] 06:24:31 DEBUG 06:24:31 DEBUG 06:24:31 DEBUG 06:24:31 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:24:31 DEBUG }, 06:24:31 DEBUG "value": [ 06:24:31 DEBUG 1712726671, 06:24:31 DEBUG "0" 06:24:31 DEBUG ] 06:24:31 DEBUG } 06:24:31 DEBUG ] 06:24:31 DEBUG } 06:24:31 DEBUG } 06:24:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712726671" 06:24:31 INFO [http_cmd]: http status code OK 06:24:31 DEBUG --- status code --- 06:24:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:31 DEBUG --- http response --- 06:24:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:24:31 DEBUG ----- output ----- 06:24:31 DEBUG { 06:24:31 DEBUG "status": "success", 06:24:31 DEBUG "data": { 06:24:31 DEBUG "resultType": "vector", 06:24:31 DEBUG "result": [ 06:24:31 DEBUG { 06:24:31 DEBUG "metric": { 06:24:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:24:31 DEBUG }, 06:24:31 DEBUG "value": [ 06:24:31 DEBUG 06:24:31 DEBUG 06:24:31 DEBUG 06:24:31 DEBUG [...] 06:24:31 DEBUG 06:24:31 DEBUG 06:24:31 DEBUG 06:24:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:24:31 DEBUG }, 06:24:31 DEBUG "value": [ 06:24:31 DEBUG 1712726671, 06:24:31 DEBUG "2486001664" 06:24:31 DEBUG ] 06:24:31 DEBUG } 06:24:31 DEBUG ] 06:24:31 DEBUG } 06:24:31 DEBUG } 06:24:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726671" 06:24:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726671" 06:24:31 INFO [http_cmd]: http status code OK 06:24:31 DEBUG --- status code --- 06:24:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:31 DEBUG --- http response --- 06:24:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:24:31 DEBUG ----- output ----- 06:24:31 DEBUG { 06:24:31 DEBUG "status": "success", 06:24:31 DEBUG "data": { 06:24:31 DEBUG "resultType": "vector", 06:24:31 DEBUG "result": [ 06:24:31 DEBUG { 06:24:31 DEBUG "metric": { 06:24:31 DEBUG "instance": "10.142.0.25:9100" 06:24:31 DEBUG }, 06:24:31 DEBUG "value": [ 06:24:31 DEBUG 06:24:31 DEBUG 06:24:31 DEBUG 06:24:31 DEBUG [...] 06:24:31 DEBUG 06:24:31 DEBUG 06:24:31 DEBUG 06:24:31 DEBUG "instance": "10.142.0.77:9100" 06:24:31 DEBUG }, 06:24:31 DEBUG "value": [ 06:24:31 DEBUG 1712726671, 06:24:31 DEBUG "42753379.63333333" 06:24:31 DEBUG ] 06:24:31 DEBUG } 06:24:31 DEBUG ] 06:24:31 DEBUG } 06:24:31 DEBUG } 06:24:31 INFO [http_cmd]: http status code OK 06:24:31 DEBUG --- status code --- 06:24:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:24:31 DEBUG --- http response --- 06:24:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:24:31 DEBUG ----- output ----- 06:24:31 DEBUG { 06:24:31 DEBUG "status": "success", 06:24:31 DEBUG "data": { 06:24:31 DEBUG "resultType": "vector", 06:24:31 DEBUG "result": [ 06:24:31 DEBUG { 06:24:31 DEBUG "metric": { 06:24:31 DEBUG "instance": "10.142.0.25:9100" 06:24:31 DEBUG }, 06:24:31 DEBUG "value": [ 06:24:31 DEBUG 06:24:31 DEBUG 06:24:31 DEBUG 06:24:31 DEBUG [...] 06:24:31 DEBUG 06:24:31 DEBUG 06:24:31 DEBUG 06:24:31 DEBUG "instance": "10.142.0.77:9100" 06:24:31 DEBUG }, 06:24:31 DEBUG "value": [ 06:24:31 DEBUG 1712726671, 06:24:31 DEBUG "0.054333333333333296" 06:24:31 DEBUG ] 06:24:31 DEBUG } 06:24:31 DEBUG ] 06:24:31 DEBUG } 06:24:31 DEBUG } 06:24:50 INFO 06:24:50 INFO [loop_until]: kubectl --namespace=xlou top pods 06:24:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:24:50 INFO [loop_until]: OK (rc = 0) 06:24:50 DEBUG --- stdout --- 06:24:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10805m 8982Mi am-654c4fcbcf-9scrj 9600m 8450Mi am-654c4fcbcf-v5qb5 8475m 8450Mi ds-cts-0 8256m 13394Mi ds-cts-1 8679m 12477Mi ds-cts-2 7607m 12662Mi ds-idrepo-0 10589m 5623Mi ds-idrepo-1 400m 5672Mi ds-idrepo-2 449m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 5m 2213Mi lodemon-b776b7d88-thrqf 9m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4300m 2302Mi 06:24:50 DEBUG --- stderr --- 06:24:50 DEBUG 06:24:58 INFO 06:24:58 INFO [loop_until]: kubectl --namespace=xlou top node 06:24:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:24:58 INFO [loop_until]: OK (rc = 0) 06:24:58 DEBUG --- stdout --- 06:24:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 91m 0% 2306Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9215m 57% 9321Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 76m 0% 3260Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13450m 84% 6697Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10857m 68% 9333Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11926m 75% 9805Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 77m 0% 856Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10874m 68% 6303Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8303m 52% 14068Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 7707m 48% 13322Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 483m 3% 6391Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8638m 54% 13239Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 466m 2% 6351Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4258m 26% 5905Mi 10% 06:24:58 DEBUG --- stderr --- 06:24:58 DEBUG 06:25:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712726728" 06:25:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726728" 06:25:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726728" 06:25:28 ERROR [http_cmd]: ERROR 06:25:28 INFO [http_cmd]: http status code OK 06:25:28 ERROR --- status code --- 06:25:28 DEBUG --- status code --- 06:25:28 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:25:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:28 ERROR --- http response --- 06:25:28 DEBUG --- http response --- 06:25:28 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:25:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:25:28 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:25:28 DEBUG ----- output ----- 06:25:28 DEBUG { 06:25:28 DEBUG "status": "success", 06:25:28 DEBUG "data": { 06:25:28 DEBUG "resultType": "vector", 06:25:28 DEBUG "result": [ 06:25:28 DEBUG { 06:25:28 DEBUG "metric": { 06:25:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:25:28 DEBUG }, 06:25:28 DEBUG "value": [ 06:25:28 DEBUG 06:25:28 DEBUG 06:25:28 DEBUG 06:25:28 DEBUG [...] 06:25:28 DEBUG 06:25:28 DEBUG 06:25:28 DEBUG 06:25:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:25:28 DEBUG }, 06:25:28 DEBUG "value": [ 06:25:28 DEBUG 1712726728, 06:25:28 INFO [http_cmd]: http status code OK 06:25:28 DEBUG "0" 06:25:28 DEBUG --- status code --- 06:25:28 DEBUG ] 06:25:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:28 DEBUG } 06:25:28 DEBUG --- http response --- 06:25:28 DEBUG ] 06:25:28 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:25:28 DEBUG } 06:25:28 DEBUG } 06:25:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726729" 06:25:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726729" 06:25:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726729" 06:25:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726729" 06:25:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726729" 06:25:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726729" 06:25:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726729" 06:25:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726729" 06:25:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726729" 06:25:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726729" 06:25:29 INFO [http_cmd]: http status code OK 06:25:29 DEBUG --- status code --- 06:25:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:29 DEBUG --- http response --- 06:25:29 INFO [http_cmd]: http status code OK 06:25:29 INFO [http_cmd]: http status code OK 06:25:29 INFO [http_cmd]: http status code OK 06:25:29 DEBUG --- status code --- 06:25:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:29 DEBUG --- http response --- 06:25:29 INFO [http_cmd]: http status code OK 06:25:29 DEBUG --- status code --- 06:25:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:29 DEBUG --- http response --- 06:25:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:25:29 DEBUG ----- output ----- 06:25:29 DEBUG { 06:25:29 DEBUG "status": "success", 06:25:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:25:29 DEBUG "data": { 06:25:29 DEBUG "resultType": "vector", 06:25:29 DEBUG "result": [ 06:25:29 DEBUG { 06:25:29 DEBUG "metric": { 06:25:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:25:29 DEBUG }, 06:25:29 DEBUG ----- output ----- 06:25:29 DEBUG "value": [ 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG [...] 06:25:29 DEBUG --- status code --- 06:25:29 INFO [http_cmd]: http status code OK 06:25:29 DEBUG --- status code --- 06:25:29 DEBUG { 06:25:29 INFO [http_cmd]: http status code OK 06:25:29 DEBUG --- status code --- 06:25:29 INFO [http_cmd]: http status code OK 06:25:29 DEBUG --- status code --- 06:25:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:29 DEBUG --- http response --- 06:25:29 INFO [http_cmd]: http status code OK 06:25:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:25:29 DEBUG "status": "success", 06:25:29 DEBUG "data": { 06:25:29 INFO [http_cmd]: http status code OK 06:25:29 DEBUG --- status code --- 06:25:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:29 DEBUG --- http response --- 06:25:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:25:29 DEBUG --- status code --- 06:25:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:29 DEBUG ----- output ----- 06:25:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:29 DEBUG "resultType": "vector", 06:25:29 DEBUG --- status code --- 06:25:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:29 DEBUG ----- output ----- 06:25:29 DEBUG { 06:25:29 DEBUG "status": "success", 06:25:29 DEBUG --- http response --- 06:25:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:25:29 DEBUG ----- output ----- 06:25:29 DEBUG --- http response --- 06:25:29 DEBUG "result": [ 06:25:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:29 DEBUG --- http response --- 06:25:29 DEBUG "data": { 06:25:29 DEBUG "resultType": "vector", 06:25:29 DEBUG "result": [ 06:25:29 DEBUG { 06:25:29 DEBUG "metric": { 06:25:29 DEBUG "pod": "ds-idrepo-0" 06:25:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:25:29 DEBUG ----- output ----- 06:25:29 DEBUG { 06:25:29 DEBUG "status": "success", 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:25:29 DEBUG }, 06:25:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:25:29 DEBUG --- http response --- 06:25:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:25:29 DEBUG "data": { 06:25:29 DEBUG "resultType": "vector", 06:25:29 DEBUG --- http response --- 06:25:29 DEBUG { 06:25:29 DEBUG "value": [ 06:25:29 DEBUG 1712726729, 06:25:29 DEBUG "258931" 06:25:29 DEBUG ] 06:25:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:25:29 DEBUG ----- output ----- 06:25:29 DEBUG { 06:25:29 DEBUG "status": "success", 06:25:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:25:29 DEBUG "status": "success", 06:25:29 DEBUG } 06:25:29 DEBUG ----- output ----- 06:25:29 DEBUG { 06:25:29 DEBUG "status": "success", 06:25:29 DEBUG ----- output ----- 06:25:29 DEBUG { 06:25:29 DEBUG "status": "success", 06:25:29 DEBUG "data": { 06:25:29 DEBUG ----- output ----- 06:25:29 DEBUG "data": { 06:25:29 DEBUG ] 06:25:29 DEBUG } 06:25:29 DEBUG } 06:25:29 DEBUG }, 06:25:29 DEBUG "resultType": "vector", 06:25:29 DEBUG "data": { 06:25:29 DEBUG "result": [ 06:25:29 DEBUG { 06:25:29 DEBUG { 06:25:29 DEBUG "resultType": "vector", 06:25:29 DEBUG "data": { 06:25:29 DEBUG { 06:25:29 DEBUG "value": [ 06:25:29 DEBUG "result": [ 06:25:29 DEBUG "resultType": "vector", 06:25:29 DEBUG { 06:25:29 DEBUG "metric": { 06:25:29 DEBUG "status": "success", 06:25:29 DEBUG "data": { 06:25:29 DEBUG "resultType": "vector", 06:25:29 DEBUG "result": [ 06:25:29 DEBUG 06:25:29 DEBUG { 06:25:29 DEBUG "result": [ 06:25:29 DEBUG { 06:25:29 DEBUG "metric": { 06:25:29 DEBUG "resultType": "vector", 06:25:29 DEBUG "result": [ 06:25:29 DEBUG { 06:25:29 DEBUG "status": "success", 06:25:29 DEBUG 06:25:29 DEBUG "metric": { 06:25:29 DEBUG "instance": "10.142.0.25:9100" 06:25:29 DEBUG "metric": { 06:25:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:25:29 DEBUG { 06:25:29 DEBUG "metric": { 06:25:29 DEBUG "pod": "ds-idrepo-0" 06:25:29 DEBUG "data": { 06:25:29 DEBUG "resultType": "vector", 06:25:29 DEBUG 06:25:29 DEBUG }, 06:25:29 DEBUG "value": [ 06:25:29 DEBUG }, 06:25:29 DEBUG "instance": "10.142.0.25:9100" 06:25:29 DEBUG }, 06:25:29 DEBUG "value": [ 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG "result": [ 06:25:29 DEBUG { 06:25:29 DEBUG "metric": { 06:25:29 DEBUG "value": [ 06:25:29 DEBUG "pod": "ds-idrepo-0" 06:25:29 DEBUG [...] 06:25:29 DEBUG 06:25:29 DEBUG "result": [ 06:25:29 DEBUG }, 06:25:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:25:29 DEBUG "metric": { 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG }, 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG { 06:25:29 DEBUG "value": [ 06:25:29 DEBUG }, 06:25:29 DEBUG "value": [ 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG "value": [ 06:25:29 DEBUG "pod": "ds-idrepo-1" 06:25:29 DEBUG }, 06:25:29 DEBUG "metric": { 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG "instance": "10.142.0.25:9100" 06:25:29 DEBUG }, 06:25:29 DEBUG 06:25:29 DEBUG "value": [ 06:25:29 DEBUG 1712726729, 06:25:29 DEBUG "0" 06:25:29 DEBUG ] 06:25:29 DEBUG } 06:25:29 DEBUG ] 06:25:29 DEBUG } 06:25:29 DEBUG } 06:25:29 DEBUG 06:25:29 DEBUG "value": [ 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG [...] 06:25:29 DEBUG [...] 06:25:29 DEBUG 06:25:29 DEBUG }, 06:25:29 DEBUG [...] 06:25:29 DEBUG [...] 06:25:29 DEBUG 06:25:29 DEBUG [...] 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG [...] 06:25:29 DEBUG "value": [ 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG [...] 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG "instance": "10.142.0.77:9100" 06:25:29 DEBUG }, 06:25:29 DEBUG "value": [ 06:25:29 DEBUG 1712726729, 06:25:29 DEBUG "392237450240" 06:25:29 DEBUG }, 06:25:29 DEBUG 06:25:29 DEBUG "pod": "ds-cts-2" 06:25:29 DEBUG 06:25:29 DEBUG ] 06:25:29 DEBUG } 06:25:29 DEBUG "instance": "10.142.0.77:9100" 06:25:29 DEBUG }, 06:25:29 DEBUG "value": [ 06:25:29 DEBUG [...] 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG ] 06:25:29 DEBUG } 06:25:29 DEBUG 1712726729, 06:25:29 DEBUG "0.06880000000000296" 06:25:29 DEBUG "value": [ 06:25:29 DEBUG 1712726729, 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG } 06:25:29 DEBUG "instance": "10.142.0.77:9100" 06:25:29 DEBUG }, 06:25:29 DEBUG "value": [ 06:25:29 DEBUG "1435.3666666666666" 06:25:29 DEBUG ] 06:25:29 DEBUG }, 06:25:29 DEBUG "value": [ 06:25:29 DEBUG 1712726729, 06:25:29 DEBUG 1712726729, 06:25:29 DEBUG "42588633.7" 06:25:29 DEBUG 06:25:29 DEBUG } 06:25:29 DEBUG ] 06:25:29 DEBUG } 06:25:29 DEBUG } 06:25:29 DEBUG 06:25:29 DEBUG 06:25:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:25:29 DEBUG "0" 06:25:29 DEBUG ] 06:25:29 DEBUG ] 06:25:29 DEBUG } 06:25:29 DEBUG ] 06:25:29 DEBUG }, 06:25:29 DEBUG } 06:25:29 DEBUG ] 06:25:29 DEBUG } 06:25:29 DEBUG } 06:25:29 DEBUG ] 06:25:29 DEBUG } 06:25:29 DEBUG ] 06:25:29 DEBUG } 06:25:29 DEBUG } 06:25:29 DEBUG "value": [ 06:25:29 DEBUG 1712726729, 06:25:29 DEBUG "1435.0666666666666" 06:25:29 DEBUG "pod": "ds-cts-2" 06:25:29 DEBUG } 06:25:29 DEBUG } 06:25:29 DEBUG ] 06:25:29 DEBUG }, 06:25:29 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:25:29 DEBUG } 06:25:29 DEBUG "value": [ 06:25:29 DEBUG }, 06:25:29 DEBUG ] 06:25:29 DEBUG 1712726729, 06:25:29 DEBUG "value": [ 06:25:29 DEBUG } 06:25:29 DEBUG } 06:25:29 DEBUG "29.03333333333333" 06:25:29 DEBUG 1712726729, 06:25:29 DEBUG "0" 06:25:29 DEBUG ] 06:25:29 DEBUG } 06:25:29 DEBUG ] 06:25:29 DEBUG } 06:25:29 DEBUG } 06:25:29 DEBUG ] 06:25:29 DEBUG } 06:25:29 DEBUG ] 06:25:29 DEBUG } 06:25:29 DEBUG } 06:25:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726730" 06:25:30 INFO [http_cmd]: http status code OK 06:25:30 DEBUG --- status code --- 06:25:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:30 DEBUG --- http response --- 06:25:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:25:30 DEBUG ----- output ----- 06:25:30 DEBUG { 06:25:30 DEBUG "status": "success", 06:25:30 DEBUG "data": { 06:25:30 DEBUG "resultType": "vector", 06:25:30 DEBUG "result": [ 06:25:30 DEBUG { 06:25:30 DEBUG "metric": { 06:25:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:25:30 DEBUG }, 06:25:30 DEBUG "value": [ 06:25:30 DEBUG 06:25:30 DEBUG 06:25:30 DEBUG 06:25:30 DEBUG [...] 06:25:30 DEBUG 06:25:30 DEBUG 06:25:30 DEBUG 06:25:30 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:25:30 DEBUG }, 06:25:30 DEBUG "value": [ 06:25:30 DEBUG 1712726730, 06:25:30 DEBUG "0" 06:25:30 DEBUG ] 06:25:30 DEBUG } 06:25:30 DEBUG ] 06:25:30 DEBUG } 06:25:30 DEBUG } 06:25:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712726730" 06:25:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712726730" 06:25:30 INFO [http_cmd]: http status code OK 06:25:30 DEBUG --- status code --- 06:25:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:30 DEBUG --- http response --- 06:25:30 INFO [http_cmd]: http status code OK 06:25:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:25:30 DEBUG --- status code --- 06:25:30 DEBUG ----- output ----- 06:25:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:30 DEBUG --- http response --- 06:25:30 DEBUG { 06:25:30 DEBUG "status": "success", 06:25:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:25:30 DEBUG "data": { 06:25:30 DEBUG "resultType": "vector", 06:25:30 DEBUG "result": [ 06:25:30 DEBUG ----- output ----- 06:25:30 DEBUG { 06:25:30 DEBUG { 06:25:30 DEBUG "metric": { 06:25:30 DEBUG "status": "success", 06:25:30 DEBUG "pod": "ds-idrepo-0" 06:25:30 DEBUG }, 06:25:30 DEBUG "value": [ 06:25:30 DEBUG 06:25:30 DEBUG 06:25:30 DEBUG 06:25:30 DEBUG [...] 06:25:30 DEBUG 06:25:30 DEBUG 06:25:30 DEBUG 06:25:30 DEBUG "pod": "ds-cts-2" 06:25:30 DEBUG }, 06:25:30 DEBUG "value": [ 06:25:30 DEBUG 1712726730, 06:25:30 DEBUG "0" 06:25:30 DEBUG ] 06:25:30 DEBUG } 06:25:30 DEBUG ] 06:25:30 DEBUG } 06:25:30 DEBUG } 06:25:30 DEBUG "data": { 06:25:30 DEBUG "resultType": "vector", 06:25:30 DEBUG "result": [ 06:25:30 DEBUG { 06:25:30 DEBUG "metric": { 06:25:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:25:30 DEBUG }, 06:25:30 DEBUG "value": [ 06:25:30 DEBUG 06:25:30 DEBUG 06:25:30 DEBUG 06:25:30 DEBUG [...] 06:25:30 DEBUG 06:25:30 DEBUG 06:25:30 DEBUG 06:25:30 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:25:30 DEBUG }, 06:25:30 DEBUG "value": [ 06:25:30 DEBUG 1712726730, 06:25:30 DEBUG "4.251330027160247" 06:25:30 DEBUG ] 06:25:30 DEBUG } 06:25:30 DEBUG ] 06:25:30 DEBUG } 06:25:30 DEBUG } 06:25:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712726730" 06:25:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726730" 06:25:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726730" 06:25:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726730" 06:25:30 INFO [http_cmd]: http status code OK 06:25:30 DEBUG --- status code --- 06:25:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:30 DEBUG --- http response --- 06:25:30 INFO [http_cmd]: http status code OK 06:25:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:25:30 DEBUG ----- output ----- 06:25:30 DEBUG --- status code --- 06:25:30 DEBUG { 06:25:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:30 DEBUG "status": "success", 06:25:30 DEBUG "data": { 06:25:30 DEBUG "resultType": "vector", 06:25:30 DEBUG "result": [ 06:25:30 DEBUG { 06:25:30 DEBUG "metric": { 06:25:30 DEBUG "pod": "ds-idrepo-2" 06:25:30 DEBUG }, 06:25:30 DEBUG "value": [ 06:25:30 DEBUG 06:25:30 DEBUG 06:25:30 DEBUG 06:25:30 DEBUG [...] 06:25:30 DEBUG 06:25:30 DEBUG 06:25:30 DEBUG 06:25:30 DEBUG "pod": "ds-cts-2" 06:25:30 DEBUG }, 06:25:30 DEBUG --- http response --- 06:25:30 DEBUG "value": [ 06:25:30 INFO [http_cmd]: http status code OK 06:25:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:25:30 DEBUG 1712726730, 06:25:30 DEBUG --- status code --- 06:25:30 INFO [http_cmd]: http status code OK 06:25:30 DEBUG ----- output ----- 06:25:30 DEBUG "0.00016442777777777781" 06:25:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:30 DEBUG --- status code --- 06:25:30 DEBUG { 06:25:30 DEBUG "status": "success", 06:25:30 DEBUG "data": { 06:25:30 DEBUG "resultType": "vector", 06:25:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:30 DEBUG --- http response --- 06:25:30 DEBUG --- http response --- 06:25:30 DEBUG "result": [ 06:25:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:25:30 DEBUG ] 06:25:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:25:30 DEBUG { 06:25:30 DEBUG "metric": { 06:25:30 DEBUG } 06:25:30 DEBUG ----- output ----- 06:25:30 DEBUG ----- output ----- 06:25:30 DEBUG { 06:25:30 DEBUG ] 06:25:30 DEBUG { 06:25:30 DEBUG "status": "success", 06:25:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:25:30 DEBUG } 06:25:30 DEBUG "status": "success", 06:25:30 DEBUG "data": { 06:25:30 DEBUG }, 06:25:30 DEBUG } 06:25:30 DEBUG "resultType": "vector", 06:25:30 DEBUG "data": { 06:25:30 DEBUG "value": [ 06:25:30 DEBUG "result": [ 06:25:30 DEBUG "resultType": "vector", 06:25:30 DEBUG 06:25:30 DEBUG { 06:25:30 DEBUG "result": [ 06:25:30 DEBUG 06:25:30 DEBUG "metric": { 06:25:30 DEBUG { 06:25:30 DEBUG 06:25:30 DEBUG [...] 06:25:30 DEBUG "metric": { 06:25:30 DEBUG 06:25:30 DEBUG 06:25:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:25:30 DEBUG 06:25:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:25:30 DEBUG }, 06:25:30 DEBUG }, 06:25:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:25:30 DEBUG "value": [ 06:25:30 DEBUG 06:25:30 DEBUG }, 06:25:30 DEBUG "value": [ 06:25:30 DEBUG 06:25:30 DEBUG "value": [ 06:25:30 DEBUG 06:25:30 DEBUG 06:25:30 DEBUG 06:25:30 DEBUG 06:25:30 DEBUG [...] 06:25:30 DEBUG [...] 06:25:30 DEBUG 06:25:30 DEBUG 06:25:30 DEBUG 06:25:30 DEBUG 06:25:30 DEBUG 06:25:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:25:30 DEBUG 1712726730, 06:25:30 DEBUG 06:25:30 DEBUG }, 06:25:30 DEBUG "value": [ 06:25:30 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:25:30 DEBUG 1712726730, 06:25:30 DEBUG "0.051" 06:25:30 DEBUG }, 06:25:30 DEBUG ] 06:25:30 DEBUG } 06:25:30 DEBUG "value": [ 06:25:30 DEBUG ] 06:25:30 DEBUG } 06:25:30 DEBUG 1712726730, 06:25:30 DEBUG } 06:25:30 DEBUG "1435.5666666666666" 06:25:30 DEBUG "0" 06:25:30 DEBUG ] 06:25:30 DEBUG ] 06:25:30 DEBUG } 06:25:30 DEBUG } 06:25:30 DEBUG ] 06:25:30 DEBUG ] 06:25:30 DEBUG } 06:25:30 DEBUG } 06:25:30 DEBUG } 06:25:30 DEBUG } 06:25:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726731" 06:25:31 INFO [http_cmd]: http status code OK 06:25:31 DEBUG --- status code --- 06:25:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:31 DEBUG --- http response --- 06:25:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:25:31 DEBUG ----- output ----- 06:25:31 DEBUG { 06:25:31 DEBUG "status": "success", 06:25:31 DEBUG "data": { 06:25:31 DEBUG "resultType": "vector", 06:25:31 DEBUG "result": [ 06:25:31 DEBUG { 06:25:31 DEBUG "metric": { 06:25:31 DEBUG "pod": "ds-idrepo-0" 06:25:31 DEBUG }, 06:25:31 DEBUG "value": [ 06:25:31 DEBUG 06:25:31 DEBUG 06:25:31 DEBUG 06:25:31 DEBUG [...] 06:25:31 DEBUG 06:25:31 DEBUG 06:25:31 DEBUG 06:25:31 DEBUG "pod": "ds-idrepo-1" 06:25:31 DEBUG }, 06:25:31 DEBUG "value": [ 06:25:31 DEBUG 1712726731, 06:25:31 DEBUG "0" 06:25:31 DEBUG ] 06:25:31 DEBUG } 06:25:31 DEBUG ] 06:25:31 DEBUG } 06:25:31 DEBUG } 06:25:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726731" 06:25:31 INFO [http_cmd]: http status code OK 06:25:31 DEBUG --- status code --- 06:25:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:31 DEBUG --- http response --- 06:25:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:25:31 DEBUG ----- output ----- 06:25:31 DEBUG { 06:25:31 DEBUG "status": "success", 06:25:31 DEBUG "data": { 06:25:31 DEBUG "resultType": "vector", 06:25:31 DEBUG "result": [ 06:25:31 DEBUG { 06:25:31 DEBUG "metric": { 06:25:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:25:31 DEBUG }, 06:25:31 DEBUG "value": [ 06:25:31 DEBUG 06:25:31 DEBUG 06:25:31 DEBUG 06:25:31 DEBUG [...] 06:25:31 DEBUG 06:25:31 DEBUG 06:25:31 DEBUG 06:25:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:25:31 DEBUG }, 06:25:31 DEBUG "value": [ 06:25:31 DEBUG 1712726731, 06:25:31 DEBUG "1439.9333333333334" 06:25:31 DEBUG ] 06:25:31 DEBUG } 06:25:31 DEBUG ] 06:25:31 DEBUG } 06:25:31 DEBUG } 06:25:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726731" 06:25:31 INFO [http_cmd]: http status code OK 06:25:31 DEBUG --- status code --- 06:25:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:31 DEBUG --- http response --- 06:25:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:25:31 DEBUG ----- output ----- 06:25:31 DEBUG { 06:25:31 DEBUG "status": "success", 06:25:31 DEBUG "data": { 06:25:31 DEBUG "resultType": "vector", 06:25:31 DEBUG "result": [ 06:25:31 DEBUG { 06:25:31 DEBUG "metric": { 06:25:31 DEBUG "pod": "ds-idrepo-0" 06:25:31 DEBUG }, 06:25:31 DEBUG "value": [ 06:25:31 DEBUG 06:25:31 DEBUG 06:25:31 DEBUG 06:25:31 DEBUG [...] 06:25:31 DEBUG 06:25:31 DEBUG 06:25:31 DEBUG 06:25:31 DEBUG "pod": "ds-idrepo-1" 06:25:31 DEBUG }, 06:25:31 DEBUG "value": [ 06:25:31 DEBUG 1712726731, 06:25:31 DEBUG "0" 06:25:31 DEBUG ] 06:25:31 DEBUG } 06:25:31 DEBUG ] 06:25:31 DEBUG } 06:25:31 DEBUG } 06:25:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726731" 06:25:31 INFO [http_cmd]: http status code OK 06:25:31 DEBUG --- status code --- 06:25:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:31 DEBUG --- http response --- 06:25:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:25:31 DEBUG ----- output ----- 06:25:31 DEBUG { 06:25:31 DEBUG "status": "success", 06:25:31 DEBUG "data": { 06:25:31 DEBUG "resultType": "vector", 06:25:31 DEBUG "result": [ 06:25:31 DEBUG { 06:25:31 DEBUG "metric": { 06:25:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:25:31 DEBUG }, 06:25:31 DEBUG "value": [ 06:25:31 DEBUG 06:25:31 DEBUG 06:25:31 DEBUG 06:25:31 DEBUG [...] 06:25:31 DEBUG 06:25:31 DEBUG 06:25:31 DEBUG 06:25:31 DEBUG "pod": "ds-cts-2" 06:25:31 DEBUG }, 06:25:31 DEBUG "value": [ 06:25:31 DEBUG 1712726731, 06:25:31 DEBUG "31644703.212721553" 06:25:31 DEBUG ] 06:25:31 DEBUG } 06:25:31 DEBUG ] 06:25:31 DEBUG } 06:25:31 DEBUG } 06:25:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712726731" 06:25:31 INFO [http_cmd]: http status code OK 06:25:31 DEBUG --- status code --- 06:25:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:31 DEBUG --- http response --- 06:25:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:25:31 DEBUG ----- output ----- 06:25:31 DEBUG { 06:25:31 DEBUG "status": "success", 06:25:31 DEBUG "data": { 06:25:31 DEBUG "resultType": "vector", 06:25:31 DEBUG "result": [ 06:25:31 DEBUG { 06:25:31 DEBUG "metric": { 06:25:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:25:31 DEBUG }, 06:25:31 DEBUG "value": [ 06:25:31 DEBUG 06:25:31 DEBUG 06:25:31 DEBUG 06:25:31 DEBUG [...] 06:25:31 DEBUG 06:25:31 DEBUG 06:25:31 DEBUG 06:25:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:25:31 DEBUG }, 06:25:31 DEBUG "value": [ 06:25:31 DEBUG 1712726731, 06:25:31 DEBUG "2488741888" 06:25:31 DEBUG ] 06:25:31 DEBUG } 06:25:31 DEBUG ] 06:25:31 DEBUG } 06:25:31 DEBUG } 06:25:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726731" 06:25:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726731" 06:25:31 INFO [http_cmd]: http status code OK 06:25:31 DEBUG --- status code --- 06:25:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:31 DEBUG --- http response --- 06:25:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:25:31 DEBUG ----- output ----- 06:25:31 DEBUG { 06:25:31 DEBUG "status": "success", 06:25:31 DEBUG "data": { 06:25:31 DEBUG "resultType": "vector", 06:25:31 DEBUG "result": [ 06:25:31 DEBUG { 06:25:31 DEBUG "metric": { 06:25:31 DEBUG "instance": "10.142.0.25:9100" 06:25:31 DEBUG }, 06:25:31 DEBUG "value": [ 06:25:31 DEBUG 06:25:31 DEBUG 06:25:31 DEBUG 06:25:31 DEBUG [...] 06:25:31 DEBUG 06:25:31 DEBUG 06:25:31 DEBUG 06:25:31 DEBUG "instance": "10.142.0.77:9100" 06:25:31 DEBUG }, 06:25:31 DEBUG "value": [ 06:25:31 DEBUG 1712726731, 06:25:31 DEBUG "42590649.03333333" 06:25:31 DEBUG ] 06:25:31 DEBUG } 06:25:31 DEBUG ] 06:25:31 DEBUG } 06:25:31 DEBUG } 06:25:31 INFO [http_cmd]: http status code OK 06:25:31 DEBUG --- status code --- 06:25:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:25:31 DEBUG --- http response --- 06:25:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:25:31 DEBUG ----- output ----- 06:25:31 DEBUG { 06:25:31 DEBUG "status": "success", 06:25:31 DEBUG "data": { 06:25:31 DEBUG "resultType": "vector", 06:25:31 DEBUG "result": [ 06:25:31 DEBUG { 06:25:31 DEBUG "metric": { 06:25:31 DEBUG "instance": "10.142.0.25:9100" 06:25:31 DEBUG }, 06:25:31 DEBUG "value": [ 06:25:31 DEBUG 06:25:31 DEBUG 06:25:31 DEBUG 06:25:31 DEBUG [...] 06:25:31 DEBUG 06:25:31 DEBUG 06:25:31 DEBUG 06:25:31 DEBUG "instance": "10.142.0.77:9100" 06:25:31 DEBUG }, 06:25:31 DEBUG "value": [ 06:25:31 DEBUG 1712726731, 06:25:31 DEBUG "0.021999999999999412" 06:25:31 DEBUG ] 06:25:31 DEBUG } 06:25:31 DEBUG ] 06:25:31 DEBUG } 06:25:31 DEBUG } 06:25:50 INFO 06:25:50 INFO [loop_until]: kubectl --namespace=xlou top pods 06:25:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:25:50 INFO [loop_until]: OK (rc = 0) 06:25:50 DEBUG --- stdout --- 06:25:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10833m 9000Mi am-654c4fcbcf-9scrj 9664m 8448Mi am-654c4fcbcf-v5qb5 8348m 8445Mi ds-cts-0 8396m 12641Mi ds-cts-1 8264m 13068Mi ds-cts-2 7589m 13394Mi ds-idrepo-0 10858m 5623Mi ds-idrepo-1 368m 5672Mi ds-idrepo-2 377m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 5m 2213Mi lodemon-b776b7d88-thrqf 5m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4271m 2304Mi 06:25:50 DEBUG --- stderr --- 06:25:50 DEBUG 06:25:58 INFO 06:25:58 INFO [loop_until]: kubectl --namespace=xlou top node 06:25:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:25:58 INFO [loop_until]: OK (rc = 0) 06:25:58 DEBUG --- stdout --- 06:25:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 93m 0% 2309Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9222m 58% 9317Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 80m 0% 3252Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13883m 87% 6704Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 11005m 69% 9333Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11918m 75% 9813Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 76m 0% 853Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10958m 68% 6300Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8602m 54% 13311Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 7783m 48% 14067Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 432m 2% 6391Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8730m 54% 14012Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 434m 2% 6352Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4460m 28% 6037Mi 10% 06:25:58 DEBUG --- stderr --- 06:25:58 DEBUG 06:26:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712726788" 06:26:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726788" 06:26:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726788" 06:26:28 ERROR [http_cmd]: ERROR 06:26:28 ERROR --- status code --- 06:26:28 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:26:28 ERROR --- http response --- 06:26:28 INFO [http_cmd]: http status code OK 06:26:28 DEBUG --- status code --- 06:26:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:28 DEBUG --- http response --- 06:26:28 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:26:28 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:26:28 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:26:28 INFO [http_cmd]: http status code OK 06:26:28 DEBUG --- status code --- 06:26:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:28 DEBUG --- http response --- 06:26:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:26:28 DEBUG ----- output ----- 06:26:28 DEBUG { 06:26:28 DEBUG "status": "success", 06:26:28 DEBUG "data": { 06:26:28 DEBUG "resultType": "vector", 06:26:28 DEBUG "result": [ 06:26:28 DEBUG { 06:26:28 DEBUG "metric": { 06:26:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:26:28 DEBUG }, 06:26:28 DEBUG "value": [ 06:26:28 DEBUG 06:26:28 DEBUG 06:26:28 DEBUG 06:26:28 DEBUG [...] 06:26:28 DEBUG 06:26:28 DEBUG 06:26:28 DEBUG 06:26:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:26:28 DEBUG }, 06:26:28 DEBUG "value": [ 06:26:28 DEBUG 1712726788, 06:26:28 DEBUG "0" 06:26:28 DEBUG ] 06:26:28 DEBUG } 06:26:28 DEBUG ] 06:26:28 DEBUG } 06:26:28 DEBUG } 06:26:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726789" 06:26:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726789" 06:26:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726789" 06:26:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726789" 06:26:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726789" 06:26:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726789" 06:26:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726789" 06:26:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726789" 06:26:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726789" 06:26:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726789" 06:26:29 INFO [http_cmd]: http status code OK 06:26:29 DEBUG --- status code --- 06:26:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:29 DEBUG --- http response --- 06:26:29 INFO [http_cmd]: http status code OK 06:26:29 INFO [http_cmd]: http status code OK 06:26:29 DEBUG --- status code --- 06:26:29 DEBUG --- status code --- 06:26:29 INFO [http_cmd]: http status code OK 06:26:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:29 INFO [http_cmd]: http status code OK 06:26:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:26:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:29 DEBUG --- http response --- 06:26:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:26:29 DEBUG ----- output ----- 06:26:29 DEBUG { 06:26:29 DEBUG --- status code --- 06:26:29 DEBUG ----- output ----- 06:26:29 DEBUG --- http response --- 06:26:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:26:29 DEBUG --- status code --- 06:26:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:29 DEBUG --- http response --- 06:26:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:29 DEBUG { 06:26:29 DEBUG "status": "success", 06:26:29 DEBUG "data": { 06:26:29 DEBUG "resultType": "vector", 06:26:29 DEBUG "result": [ 06:26:29 DEBUG --- http response --- 06:26:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:26:29 DEBUG ----- output ----- 06:26:29 INFO [http_cmd]: http status code OK 06:26:29 DEBUG "status": "success", 06:26:29 DEBUG "data": { 06:26:29 DEBUG "resultType": "vector", 06:26:29 DEBUG { 06:26:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:26:29 DEBUG ----- output ----- 06:26:29 DEBUG { 06:26:29 DEBUG "status": "success", 06:26:29 DEBUG "data": { 06:26:29 DEBUG --- status code --- 06:26:29 DEBUG ----- output ----- 06:26:29 DEBUG "result": [ 06:26:29 DEBUG { 06:26:29 DEBUG "metric": { 06:26:29 DEBUG "pod": "ds-idrepo-0" 06:26:29 DEBUG { 06:26:29 INFO [http_cmd]: http status code OK 06:26:29 DEBUG "resultType": "vector", 06:26:29 DEBUG "result": [ 06:26:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:29 DEBUG { 06:26:29 DEBUG "status": "success", 06:26:29 DEBUG }, 06:26:29 DEBUG "value": [ 06:26:29 DEBUG 06:26:29 DEBUG "status": "success", 06:26:29 DEBUG "data": { 06:26:29 DEBUG "resultType": "vector", 06:26:29 INFO [http_cmd]: http status code OK 06:26:29 DEBUG --- status code --- 06:26:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:29 DEBUG --- http response --- 06:26:29 DEBUG "data": { 06:26:29 INFO [http_cmd]: http status code OK 06:26:29 INFO [http_cmd]: http status code OK 06:26:29 DEBUG { 06:26:29 DEBUG "result": [ 06:26:29 DEBUG --- http response --- 06:26:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:26:29 DEBUG ----- output ----- 06:26:29 DEBUG { 06:26:29 DEBUG "status": "success", 06:26:29 DEBUG 06:26:29 DEBUG 06:26:29 DEBUG [...] 06:26:29 DEBUG 06:26:29 DEBUG 06:26:29 DEBUG 06:26:29 DEBUG "resultType": "vector", 06:26:29 DEBUG "result": [ 06:26:29 DEBUG { 06:26:29 DEBUG "metric": { 06:26:29 DEBUG "pod": "ds-idrepo-0" 06:26:29 DEBUG }, 06:26:29 DEBUG "value": [ 06:26:29 DEBUG 06:26:29 DEBUG 06:26:29 DEBUG 06:26:29 DEBUG { 06:26:29 DEBUG "metric": { 06:26:29 DEBUG "instance": "10.142.0.25:9100" 06:26:29 DEBUG }, 06:26:29 DEBUG "metric": { 06:26:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:26:29 DEBUG }, 06:26:29 DEBUG "value": [ 06:26:29 DEBUG --- status code --- 06:26:29 DEBUG --- status code --- 06:26:29 DEBUG "metric": { 06:26:29 DEBUG [...] 06:26:29 DEBUG 06:26:29 DEBUG 06:26:29 DEBUG "pod": "ds-idrepo-1" 06:26:29 DEBUG }, 06:26:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:26:29 DEBUG ----- output ----- 06:26:29 DEBUG { 06:26:29 DEBUG "status": "success", 06:26:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:26:29 DEBUG }, 06:26:29 DEBUG 06:26:29 DEBUG "pod": "ds-cts-2" 06:26:29 DEBUG }, 06:26:29 DEBUG "value": [ 06:26:29 DEBUG 1712726789, 06:26:29 DEBUG "0" 06:26:29 DEBUG ] 06:26:29 DEBUG } 06:26:29 DEBUG ] 06:26:29 DEBUG "value": [ 06:26:29 DEBUG "value": [ 06:26:29 DEBUG 1712726789, 06:26:29 DEBUG "0" 06:26:29 DEBUG ] 06:26:29 DEBUG } 06:26:29 DEBUG } 06:26:29 DEBUG "value": [ 06:26:29 DEBUG 06:26:29 DEBUG 06:26:29 DEBUG 06:26:29 DEBUG } 06:26:29 DEBUG "data": { 06:26:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:29 DEBUG --- http response --- 06:26:29 DEBUG "data": { 06:26:29 DEBUG 06:26:29 DEBUG 06:26:29 DEBUG [...] 06:26:29 DEBUG 06:26:29 DEBUG ] 06:26:29 DEBUG "resultType": "vector", 06:26:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:29 DEBUG --- http response --- 06:26:29 DEBUG "resultType": "vector", 06:26:29 DEBUG 06:26:29 DEBUG 06:26:29 DEBUG 06:26:29 DEBUG 06:26:29 DEBUG } 06:26:29 DEBUG } 06:26:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:26:29 DEBUG ----- output ----- 06:26:29 DEBUG { 06:26:29 DEBUG 06:26:29 DEBUG 06:26:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:26:29 DEBUG [...] 06:26:29 DEBUG "result": [ 06:26:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:26:29 DEBUG "result": [ 06:26:29 DEBUG "status": "success", 06:26:29 DEBUG [...] 06:26:29 DEBUG }, 06:26:29 DEBUG "value": [ 06:26:29 DEBUG 06:26:29 DEBUG { 06:26:29 DEBUG ----- output ----- 06:26:29 DEBUG { 06:26:29 DEBUG 06:26:29 DEBUG 06:26:29 DEBUG 1712726789, 06:26:29 DEBUG "258845.5" 06:26:29 DEBUG 06:26:29 DEBUG "metric": { 06:26:29 DEBUG { 06:26:29 DEBUG "metric": { 06:26:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:26:29 DEBUG "data": { 06:26:29 DEBUG "resultType": "vector", 06:26:29 DEBUG --- status code --- 06:26:29 DEBUG 06:26:29 DEBUG "pod": "ds-idrepo-0" 06:26:29 DEBUG "status": "success", 06:26:29 DEBUG 06:26:29 DEBUG }, 06:26:29 DEBUG "value": [ 06:26:29 DEBUG "result": [ 06:26:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:26:29 DEBUG }, 06:26:29 DEBUG "instance": "10.142.0.77:9100" 06:26:29 DEBUG }, 06:26:29 DEBUG 06:26:29 DEBUG ] 06:26:29 DEBUG } 06:26:29 DEBUG ] 06:26:29 DEBUG }, 06:26:29 DEBUG "value": [ 06:26:29 DEBUG 06:26:29 DEBUG 06:26:29 DEBUG 06:26:29 DEBUG [...] 06:26:29 DEBUG 06:26:29 DEBUG 06:26:29 DEBUG 06:26:29 DEBUG "pod": "ds-cts-2" 06:26:29 DEBUG }, 06:26:29 DEBUG "value": [ 06:26:29 DEBUG 1712726789, 06:26:29 DEBUG "29.897010298970105" 06:26:29 DEBUG ] 06:26:29 DEBUG } 06:26:29 DEBUG ] 06:26:29 DEBUG } 06:26:29 DEBUG } 06:26:29 DEBUG "value": [ 06:26:29 DEBUG "value": [ 06:26:29 DEBUG 1712726789, 06:26:29 DEBUG "394227262464" 06:26:29 DEBUG ] 06:26:29 DEBUG } 06:26:29 DEBUG ] 06:26:29 DEBUG 1712726789, 06:26:29 DEBUG } 06:26:29 DEBUG } 06:26:29 DEBUG } 06:26:29 DEBUG } 06:26:29 DEBUG "data": { 06:26:29 DEBUG "resultType": "vector", 06:26:29 DEBUG "1453.6546345365464" 06:26:29 DEBUG ] 06:26:29 DEBUG } 06:26:29 DEBUG { 06:26:29 DEBUG "metric": { 06:26:29 DEBUG "instance": "10.142.0.25:9100" 06:26:29 DEBUG }, 06:26:29 DEBUG --- http response --- 06:26:29 DEBUG 06:26:29 DEBUG "result": [ 06:26:29 DEBUG "value": [ 06:26:29 DEBUG 06:26:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:26:29 DEBUG 06:26:29 DEBUG { 06:26:29 DEBUG 06:26:29 DEBUG 06:26:29 DEBUG ----- output ----- 06:26:29 DEBUG [...] 06:26:29 DEBUG "metric": { 06:26:29 DEBUG "instance": "10.142.0.25:9100" 06:26:29 DEBUG }, 06:26:29 DEBUG { 06:26:29 DEBUG 06:26:29 DEBUG 06:26:29 DEBUG 06:26:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:26:29 DEBUG "status": "success", 06:26:29 DEBUG }, 06:26:29 DEBUG "value": [ 06:26:29 DEBUG 1712726789, 06:26:29 DEBUG "1452.9880345298805" 06:26:29 DEBUG ] 06:26:29 DEBUG } 06:26:29 DEBUG ] 06:26:29 DEBUG ] 06:26:29 DEBUG } 06:26:29 DEBUG } 06:26:29 DEBUG "value": [ 06:26:29 DEBUG 06:26:29 DEBUG 06:26:29 DEBUG 06:26:29 DEBUG [...] 06:26:29 DEBUG 06:26:29 DEBUG 06:26:29 DEBUG 06:26:29 DEBUG "instance": "10.142.0.77:9100" 06:26:29 DEBUG }, 06:26:29 DEBUG "value": [ 06:26:29 DEBUG 1712726789, 06:26:29 DEBUG "0.12430000000000091" 06:26:29 DEBUG ] 06:26:29 DEBUG } 06:26:29 DEBUG ] 06:26:29 DEBUG } 06:26:29 DEBUG } 06:26:29 DEBUG [...] 06:26:29 DEBUG 06:26:29 DEBUG 06:26:29 DEBUG 06:26:29 DEBUG "instance": "10.142.0.77:9100" 06:26:29 DEBUG }, 06:26:29 DEBUG "value": [ 06:26:29 DEBUG 1712726789, 06:26:29 DEBUG "42814752.06666666" 06:26:29 DEBUG ] 06:26:29 DEBUG } 06:26:29 DEBUG ] 06:26:29 DEBUG } 06:26:29 DEBUG } 06:26:29 DEBUG "data": { 06:26:29 DEBUG } 06:26:29 DEBUG "resultType": "vector", 06:26:29 DEBUG } 06:26:29 DEBUG "result": [ 06:26:29 DEBUG { 06:26:29 DEBUG "metric": { 06:26:29 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:26:29 DEBUG }, 06:26:29 DEBUG "value": [ 06:26:29 DEBUG 06:26:29 DEBUG 06:26:29 DEBUG 06:26:29 DEBUG [...] 06:26:29 DEBUG 06:26:29 DEBUG 06:26:29 DEBUG 06:26:29 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:26:29 DEBUG }, 06:26:29 DEBUG "value": [ 06:26:29 DEBUG 1712726789, 06:26:29 DEBUG "0" 06:26:29 DEBUG ] 06:26:29 DEBUG } 06:26:29 DEBUG ] 06:26:29 DEBUG } 06:26:29 DEBUG } 06:26:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726790" 06:26:30 INFO [http_cmd]: http status code OK 06:26:30 DEBUG --- status code --- 06:26:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:30 DEBUG --- http response --- 06:26:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:26:30 DEBUG ----- output ----- 06:26:30 DEBUG { 06:26:30 DEBUG "status": "success", 06:26:30 DEBUG "data": { 06:26:30 DEBUG "resultType": "vector", 06:26:30 DEBUG "result": [ 06:26:30 DEBUG { 06:26:30 DEBUG "metric": { 06:26:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:26:30 DEBUG }, 06:26:30 DEBUG "value": [ 06:26:30 DEBUG 06:26:30 DEBUG 06:26:30 DEBUG 06:26:30 DEBUG [...] 06:26:30 DEBUG 06:26:30 DEBUG 06:26:30 DEBUG 06:26:30 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:26:30 DEBUG }, 06:26:30 DEBUG "value": [ 06:26:30 DEBUG 1712726790, 06:26:30 DEBUG "79392403.65538177" 06:26:30 DEBUG ] 06:26:30 DEBUG } 06:26:30 DEBUG ] 06:26:30 DEBUG } 06:26:30 DEBUG } 06:26:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712726790" 06:26:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712726790" 06:26:30 INFO [http_cmd]: http status code OK 06:26:30 DEBUG --- status code --- 06:26:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:30 DEBUG --- http response --- 06:26:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:26:30 DEBUG ----- output ----- 06:26:30 DEBUG { 06:26:30 DEBUG "status": "success", 06:26:30 DEBUG "data": { 06:26:30 DEBUG "resultType": "vector", 06:26:30 DEBUG "result": [ 06:26:30 INFO [http_cmd]: http status code OK 06:26:30 DEBUG --- status code --- 06:26:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:30 DEBUG --- http response --- 06:26:30 DEBUG { 06:26:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:26:30 DEBUG "metric": { 06:26:30 DEBUG ----- output ----- 06:26:30 DEBUG "pod": "ds-idrepo-0" 06:26:30 DEBUG { 06:26:30 DEBUG }, 06:26:30 DEBUG "status": "success", 06:26:30 DEBUG "value": [ 06:26:30 DEBUG "data": { 06:26:30 DEBUG 06:26:30 DEBUG "resultType": "vector", 06:26:30 DEBUG 06:26:30 DEBUG "result": [ 06:26:30 DEBUG 06:26:30 DEBUG { 06:26:30 DEBUG [...] 06:26:30 DEBUG "metric": { 06:26:30 DEBUG 06:26:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:26:30 DEBUG 06:26:30 DEBUG }, 06:26:30 DEBUG 06:26:30 DEBUG "value": [ 06:26:30 DEBUG "pod": "ds-cts-2" 06:26:30 DEBUG 06:26:30 DEBUG }, 06:26:30 DEBUG 06:26:30 DEBUG "value": [ 06:26:30 DEBUG 06:26:30 DEBUG 1712726790, 06:26:30 DEBUG [...] 06:26:30 DEBUG "0" 06:26:30 DEBUG 06:26:30 DEBUG ] 06:26:30 DEBUG 06:26:30 DEBUG } 06:26:30 DEBUG 06:26:30 DEBUG ] 06:26:30 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:26:30 DEBUG } 06:26:30 DEBUG }, 06:26:30 DEBUG } 06:26:30 DEBUG "value": [ 06:26:30 DEBUG 1712726790, 06:26:30 DEBUG "4.347922416447548" 06:26:30 DEBUG ] 06:26:30 DEBUG } 06:26:30 DEBUG ] 06:26:30 DEBUG } 06:26:30 DEBUG } 06:26:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726791" 06:26:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712726791" 06:26:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726791" 06:26:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726791" 06:26:31 INFO [http_cmd]: http status code OK 06:26:31 INFO [http_cmd]: http status code OK 06:26:31 DEBUG --- status code --- 06:26:31 DEBUG --- status code --- 06:26:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:31 DEBUG --- http response --- 06:26:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:26:31 DEBUG ----- output ----- 06:26:31 DEBUG { 06:26:31 DEBUG "status": "success", 06:26:31 DEBUG "data": { 06:26:31 DEBUG "resultType": "vector", 06:26:31 DEBUG "result": [ 06:26:31 DEBUG { 06:26:31 DEBUG "metric": { 06:26:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:26:31 DEBUG }, 06:26:31 DEBUG "value": [ 06:26:31 DEBUG 06:26:31 DEBUG --- http response --- 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG [...] 06:26:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:26:31 INFO [http_cmd]: http status code OK 06:26:31 DEBUG --- status code --- 06:26:31 DEBUG ----- output ----- 06:26:31 INFO [http_cmd]: http status code OK 06:26:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:31 DEBUG 06:26:31 DEBUG { 06:26:31 DEBUG --- status code --- 06:26:31 DEBUG --- http response --- 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:26:31 DEBUG "status": "success", 06:26:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:26:31 DEBUG --- http response --- 06:26:31 DEBUG ----- output ----- 06:26:31 DEBUG { 06:26:31 DEBUG }, 06:26:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:26:31 DEBUG "status": "success", 06:26:31 DEBUG "data": { 06:26:31 DEBUG "value": [ 06:26:31 DEBUG ----- output ----- 06:26:31 DEBUG "data": { 06:26:31 DEBUG "resultType": "vector", 06:26:31 DEBUG 1712726791, 06:26:31 DEBUG { 06:26:31 DEBUG "resultType": "vector", 06:26:31 DEBUG "result": [ 06:26:31 DEBUG "0.051" 06:26:31 DEBUG "status": "success", 06:26:31 DEBUG "result": [ 06:26:31 DEBUG { 06:26:31 DEBUG ] 06:26:31 DEBUG "data": { 06:26:31 DEBUG { 06:26:31 DEBUG "metric": { 06:26:31 DEBUG } 06:26:31 DEBUG "resultType": "vector", 06:26:31 DEBUG "metric": { 06:26:31 DEBUG "pod": "ds-idrepo-2" 06:26:31 DEBUG ] 06:26:31 DEBUG "result": [ 06:26:31 DEBUG { 06:26:31 DEBUG "metric": { 06:26:31 DEBUG } 06:26:31 DEBUG }, 06:26:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:26:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:26:31 DEBUG } 06:26:31 DEBUG "value": [ 06:26:31 DEBUG }, 06:26:31 DEBUG "value": [ 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG }, 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG [...] 06:26:31 DEBUG 06:26:31 DEBUG "value": [ 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG [...] 06:26:31 DEBUG 06:26:31 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG }, 06:26:31 DEBUG "pod": "ds-cts-2" 06:26:31 DEBUG }, 06:26:31 DEBUG "value": [ 06:26:31 DEBUG "value": [ 06:26:31 DEBUG 1712726791, 06:26:31 DEBUG "0" 06:26:31 DEBUG 1712726791, 06:26:31 DEBUG ] 06:26:31 DEBUG [...] 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG "0.006092241627865093" 06:26:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:26:31 DEBUG }, 06:26:31 DEBUG "value": [ 06:26:31 DEBUG ] 06:26:31 DEBUG 1712726791, 06:26:31 DEBUG "1453.8546145385462" 06:26:31 DEBUG } 06:26:31 DEBUG ] 06:26:31 DEBUG } 06:26:31 DEBUG ] 06:26:31 DEBUG } 06:26:31 DEBUG } 06:26:31 DEBUG ] 06:26:31 DEBUG } 06:26:31 DEBUG ] 06:26:31 DEBUG } 06:26:31 DEBUG } 06:26:31 DEBUG } 06:26:31 DEBUG } 06:26:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726791" 06:26:31 INFO [http_cmd]: http status code OK 06:26:31 DEBUG --- status code --- 06:26:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:31 DEBUG --- http response --- 06:26:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:26:31 DEBUG ----- output ----- 06:26:31 DEBUG { 06:26:31 DEBUG "status": "success", 06:26:31 DEBUG "data": { 06:26:31 DEBUG "resultType": "vector", 06:26:31 DEBUG "result": [ 06:26:31 DEBUG { 06:26:31 DEBUG "metric": { 06:26:31 DEBUG "pod": "ds-idrepo-0" 06:26:31 DEBUG }, 06:26:31 DEBUG "value": [ 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG [...] 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG "pod": "ds-idrepo-1" 06:26:31 DEBUG }, 06:26:31 DEBUG "value": [ 06:26:31 DEBUG 1712726791, 06:26:31 DEBUG "0" 06:26:31 DEBUG ] 06:26:31 DEBUG } 06:26:31 DEBUG ] 06:26:31 DEBUG } 06:26:31 DEBUG } 06:26:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726791" 06:26:31 INFO [http_cmd]: http status code OK 06:26:31 DEBUG --- status code --- 06:26:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:31 DEBUG --- http response --- 06:26:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:26:31 DEBUG ----- output ----- 06:26:31 DEBUG { 06:26:31 DEBUG "status": "success", 06:26:31 DEBUG "data": { 06:26:31 DEBUG "resultType": "vector", 06:26:31 DEBUG "result": [ 06:26:31 DEBUG { 06:26:31 DEBUG "metric": { 06:26:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:26:31 DEBUG }, 06:26:31 DEBUG "value": [ 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG [...] 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:26:31 DEBUG }, 06:26:31 DEBUG "value": [ 06:26:31 DEBUG 1712726791, 06:26:31 DEBUG "1467.386594673866" 06:26:31 DEBUG ] 06:26:31 DEBUG } 06:26:31 DEBUG ] 06:26:31 DEBUG } 06:26:31 DEBUG } 06:26:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726791" 06:26:31 INFO [http_cmd]: http status code OK 06:26:31 DEBUG --- status code --- 06:26:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:31 DEBUG --- http response --- 06:26:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:26:31 DEBUG ----- output ----- 06:26:31 DEBUG { 06:26:31 DEBUG "status": "success", 06:26:31 DEBUG "data": { 06:26:31 DEBUG "resultType": "vector", 06:26:31 DEBUG "result": [ 06:26:31 DEBUG { 06:26:31 DEBUG "metric": { 06:26:31 DEBUG "pod": "ds-idrepo-0" 06:26:31 DEBUG }, 06:26:31 DEBUG "value": [ 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG [...] 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG "pod": "ds-idrepo-1" 06:26:31 DEBUG }, 06:26:31 DEBUG "value": [ 06:26:31 DEBUG 1712726791, 06:26:31 DEBUG "0" 06:26:31 DEBUG ] 06:26:31 DEBUG } 06:26:31 DEBUG ] 06:26:31 DEBUG } 06:26:31 DEBUG } 06:26:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726791" 06:26:31 INFO [http_cmd]: http status code OK 06:26:31 DEBUG --- status code --- 06:26:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:31 DEBUG --- http response --- 06:26:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:26:31 DEBUG ----- output ----- 06:26:31 DEBUG { 06:26:31 DEBUG "status": "success", 06:26:31 DEBUG "data": { 06:26:31 DEBUG "resultType": "vector", 06:26:31 DEBUG "result": [ 06:26:31 DEBUG { 06:26:31 DEBUG "metric": { 06:26:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:26:31 DEBUG }, 06:26:31 DEBUG "value": [ 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG [...] 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:26:31 DEBUG }, 06:26:31 DEBUG "value": [ 06:26:31 DEBUG 1712726791, 06:26:31 DEBUG "0" 06:26:31 DEBUG ] 06:26:31 DEBUG } 06:26:31 DEBUG ] 06:26:31 DEBUG } 06:26:31 DEBUG } 06:26:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712726791" 06:26:31 INFO [http_cmd]: http status code OK 06:26:31 DEBUG --- status code --- 06:26:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:31 DEBUG --- http response --- 06:26:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:26:31 DEBUG ----- output ----- 06:26:31 DEBUG { 06:26:31 DEBUG "status": "success", 06:26:31 DEBUG "data": { 06:26:31 DEBUG "resultType": "vector", 06:26:31 DEBUG "result": [ 06:26:31 DEBUG { 06:26:31 DEBUG "metric": { 06:26:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:26:31 DEBUG }, 06:26:31 DEBUG "value": [ 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG [...] 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG 06:26:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:26:31 DEBUG }, 06:26:31 DEBUG "value": [ 06:26:31 DEBUG 1712726791, 06:26:31 DEBUG "2484502528" 06:26:31 DEBUG ] 06:26:31 DEBUG } 06:26:31 DEBUG ] 06:26:31 DEBUG } 06:26:31 DEBUG } 06:26:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726792" 06:26:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726792" 06:26:32 INFO [http_cmd]: http status code OK 06:26:32 DEBUG --- status code --- 06:26:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:32 DEBUG --- http response --- 06:26:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:26:32 DEBUG ----- output ----- 06:26:32 DEBUG { 06:26:32 DEBUG "status": "success", 06:26:32 DEBUG "data": { 06:26:32 DEBUG "resultType": "vector", 06:26:32 DEBUG "result": [ 06:26:32 DEBUG { 06:26:32 DEBUG "metric": { 06:26:32 DEBUG "instance": "10.142.0.25:9100" 06:26:32 DEBUG }, 06:26:32 DEBUG "value": [ 06:26:32 DEBUG 06:26:32 DEBUG 06:26:32 DEBUG 06:26:32 DEBUG [...] 06:26:32 DEBUG 06:26:32 DEBUG 06:26:32 DEBUG 06:26:32 DEBUG "instance": "10.142.0.77:9100" 06:26:32 DEBUG }, 06:26:32 DEBUG "value": [ 06:26:32 DEBUG 1712726792, 06:26:32 DEBUG "42816682.03333333" 06:26:32 DEBUG ] 06:26:32 DEBUG } 06:26:32 DEBUG ] 06:26:32 DEBUG } 06:26:32 DEBUG } 06:26:32 INFO [http_cmd]: http status code OK 06:26:32 DEBUG --- status code --- 06:26:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:26:32 DEBUG --- http response --- 06:26:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:26:32 DEBUG ----- output ----- 06:26:32 DEBUG { 06:26:32 DEBUG "status": "success", 06:26:32 DEBUG "data": { 06:26:32 DEBUG "resultType": "vector", 06:26:32 DEBUG "result": [ 06:26:32 DEBUG { 06:26:32 DEBUG "metric": { 06:26:32 DEBUG "instance": "10.142.0.25:9100" 06:26:32 DEBUG }, 06:26:32 DEBUG "value": [ 06:26:32 DEBUG 06:26:32 DEBUG 06:26:32 DEBUG 06:26:32 DEBUG [...] 06:26:32 DEBUG 06:26:32 DEBUG 06:26:32 DEBUG 06:26:32 DEBUG "instance": "10.142.0.77:9100" 06:26:32 DEBUG }, 06:26:32 DEBUG "value": [ 06:26:32 DEBUG 1712726792, 06:26:32 DEBUG "0.0463333333333324" 06:26:32 DEBUG ] 06:26:32 DEBUG } 06:26:32 DEBUG ] 06:26:32 DEBUG } 06:26:32 DEBUG } 06:26:50 INFO 06:26:50 INFO [loop_until]: kubectl --namespace=xlou top pods 06:26:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:26:50 INFO [loop_until]: OK (rc = 0) 06:26:50 DEBUG --- stdout --- 06:26:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10889m 9002Mi am-654c4fcbcf-9scrj 9914m 8448Mi am-654c4fcbcf-v5qb5 8532m 8444Mi ds-cts-0 8477m 13223Mi ds-cts-1 8447m 12724Mi ds-cts-2 7334m 12782Mi ds-idrepo-0 10775m 5623Mi ds-idrepo-1 399m 5672Mi ds-idrepo-2 373m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1190Mi idm-6bfb7c96cc-vsb58 5m 2213Mi lodemon-b776b7d88-thrqf 2m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4279m 2449Mi 06:26:50 DEBUG --- stderr --- 06:26:50 DEBUG 06:26:58 INFO 06:26:58 INFO [loop_until]: kubectl --namespace=xlou top node 06:26:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:26:59 INFO [loop_until]: OK (rc = 0) 06:26:59 DEBUG --- stdout --- 06:26:59 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 96m 0% 2306Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9380m 59% 9320Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 81m 0% 3252Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13490m 84% 6724Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10915m 68% 9335Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11984m 75% 9806Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 78m 0% 852Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10498m 66% 6300Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8267m 52% 13920Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 7419m 46% 13499Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 432m 2% 6390Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8636m 54% 13436Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 460m 2% 6356Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4432m 27% 6044Mi 10% 06:26:59 DEBUG --- stderr --- 06:26:59 DEBUG 06:27:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726848" 06:27:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726848" 06:27:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712726848" 06:27:28 ERROR [http_cmd]: ERROR 06:27:28 ERROR --- status code --- 06:27:28 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:27:28 ERROR --- http response --- 06:27:28 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:27:28 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:27:28 INFO [http_cmd]: http status code OK 06:27:28 DEBUG --- status code --- 06:27:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:28 DEBUG --- http response --- 06:27:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:27:28 DEBUG ----- output ----- 06:27:28 DEBUG { 06:27:28 DEBUG "status": "success", 06:27:28 DEBUG "data": { 06:27:28 DEBUG "resultType": "vector", 06:27:28 DEBUG "result": [ 06:27:28 DEBUG { 06:27:28 DEBUG "metric": { 06:27:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:27:28 DEBUG }, 06:27:28 DEBUG "value": [ 06:27:28 DEBUG 06:27:28 DEBUG 06:27:28 DEBUG 06:27:28 DEBUG [...] 06:27:28 DEBUG 06:27:28 DEBUG 06:27:28 DEBUG 06:27:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:27:28 DEBUG }, 06:27:28 DEBUG "value": [ 06:27:28 DEBUG 1712726848, 06:27:28 DEBUG "0" 06:27:28 DEBUG ] 06:27:28 DEBUG } 06:27:28 DEBUG ] 06:27:28 DEBUG } 06:27:28 DEBUG } 06:27:28 INFO [http_cmd]: http status code OK 06:27:28 DEBUG --- status code --- 06:27:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:28 DEBUG --- http response --- 06:27:28 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:27:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726849" 06:27:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726849" 06:27:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726849" 06:27:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726849" 06:27:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726849" 06:27:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726849" 06:27:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726849" 06:27:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726849" 06:27:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726849" 06:27:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726849" 06:27:29 INFO [http_cmd]: http status code OK 06:27:30 INFO [http_cmd]: http status code OK 06:27:30 INFO [http_cmd]: http status code OK 06:27:30 DEBUG --- status code --- 06:27:30 INFO [http_cmd]: http status code OK 06:27:30 DEBUG --- status code --- 06:27:30 DEBUG --- status code --- 06:27:30 DEBUG --- status code --- 06:27:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:30 DEBUG --- http response --- 06:27:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:30 INFO [http_cmd]: http status code OK 06:27:30 DEBUG --- status code --- 06:27:30 DEBUG --- http response --- 06:27:30 INFO [http_cmd]: http status code OK 06:27:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:27:30 DEBUG ----- output ----- 06:27:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:30 DEBUG --- http response --- 06:27:30 INFO [http_cmd]: http status code OK 06:27:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:27:30 DEBUG --- status code --- 06:27:30 DEBUG --- http response --- 06:27:30 DEBUG { 06:27:30 INFO [http_cmd]: http status code OK 06:27:30 DEBUG --- status code --- 06:27:30 DEBUG --- http response --- 06:27:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:27:30 INFO [http_cmd]: http status code OK 06:27:30 DEBUG ----- output ----- 06:27:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:30 DEBUG "status": "success", 06:27:30 DEBUG "data": { 06:27:30 DEBUG "resultType": "vector", 06:27:30 DEBUG "result": [ 06:27:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:27:30 DEBUG ----- output ----- 06:27:30 DEBUG --- status code --- 06:27:30 DEBUG { 06:27:30 DEBUG --- http response --- 06:27:30 DEBUG { 06:27:30 DEBUG "metric": { 06:27:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:27:30 DEBUG }, 06:27:30 DEBUG ----- output ----- 06:27:30 DEBUG { 06:27:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:30 DEBUG "status": "success", 06:27:30 DEBUG "data": { 06:27:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:27:30 INFO [http_cmd]: http status code OK 06:27:30 DEBUG "value": [ 06:27:30 DEBUG 06:27:30 DEBUG 06:27:30 DEBUG { 06:27:30 DEBUG "status": "success", 06:27:30 DEBUG --- http response --- 06:27:30 DEBUG "resultType": "vector", 06:27:30 DEBUG ----- output ----- 06:27:30 DEBUG { 06:27:30 DEBUG --- status code --- 06:27:30 DEBUG 06:27:30 DEBUG --- status code --- 06:27:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:27:30 DEBUG "status": "success", 06:27:30 DEBUG "data": { 06:27:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:27:30 DEBUG ----- output ----- 06:27:30 DEBUG "result": [ 06:27:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:30 DEBUG --- http response --- 06:27:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:30 DEBUG ----- output ----- 06:27:30 DEBUG "data": { 06:27:30 DEBUG "resultType": "vector", 06:27:30 DEBUG { 06:27:30 DEBUG "status": "success", 06:27:30 DEBUG { 06:27:30 DEBUG "metric": { 06:27:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:27:30 DEBUG ----- output ----- 06:27:30 DEBUG --- http response --- 06:27:30 DEBUG { 06:27:30 DEBUG "resultType": "vector", 06:27:30 DEBUG "result": [ 06:27:30 DEBUG { 06:27:30 DEBUG "metric": { 06:27:30 DEBUG "pod": "ds-idrepo-0" 06:27:30 DEBUG [...] 06:27:30 DEBUG { 06:27:30 DEBUG "status": "success", 06:27:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:27:30 DEBUG "status": "success", 06:27:30 DEBUG "data": { 06:27:30 DEBUG "resultType": "vector", 06:27:30 DEBUG "result": [ 06:27:30 DEBUG "status": "success", 06:27:30 DEBUG "data": { 06:27:30 DEBUG 06:27:30 DEBUG "data": { 06:27:30 DEBUG "resultType": "vector", 06:27:30 DEBUG "result": [ 06:27:30 DEBUG ----- output ----- 06:27:30 DEBUG { 06:27:30 DEBUG "status": "success", 06:27:30 DEBUG "data": { 06:27:30 DEBUG "resultType": "vector", 06:27:30 DEBUG "result": [ 06:27:30 DEBUG { 06:27:30 DEBUG "metric": { 06:27:30 DEBUG "instance": "10.142.0.25:9100" 06:27:30 DEBUG { 06:27:30 DEBUG { 06:27:30 DEBUG "metric": { 06:27:30 DEBUG "pod": "ds-idrepo-0" 06:27:30 DEBUG --- http response --- 06:27:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:27:30 DEBUG ----- output ----- 06:27:30 DEBUG { 06:27:30 DEBUG "status": "success", 06:27:30 DEBUG }, 06:27:30 DEBUG "metric": { 06:27:30 DEBUG }, 06:27:30 DEBUG "value": [ 06:27:30 DEBUG 06:27:30 DEBUG "data": { 06:27:30 DEBUG "resultType": "vector", 06:27:30 DEBUG "result": [ 06:27:30 DEBUG }, 06:27:30 DEBUG "data": { 06:27:30 DEBUG "value": [ 06:27:30 DEBUG "instance": "10.142.0.25:9100" 06:27:30 DEBUG 06:27:30 DEBUG 06:27:30 DEBUG [...] 06:27:30 DEBUG { 06:27:30 DEBUG "metric": { 06:27:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:27:30 DEBUG "value": [ 06:27:30 DEBUG "resultType": "vector", 06:27:30 DEBUG 06:27:30 DEBUG }, 06:27:30 DEBUG 06:27:30 DEBUG 06:27:30 DEBUG 06:27:30 DEBUG }, 06:27:30 DEBUG "value": [ 06:27:30 DEBUG 06:27:30 DEBUG 06:27:30 DEBUG "result": [ 06:27:30 DEBUG 06:27:30 DEBUG 06:27:30 DEBUG [...] 06:27:30 DEBUG 06:27:30 DEBUG 06:27:30 DEBUG 06:27:30 DEBUG 06:27:30 DEBUG [...] 06:27:30 DEBUG 06:27:30 DEBUG { 06:27:30 DEBUG "value": [ 06:27:30 DEBUG 06:27:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:27:30 DEBUG 06:27:30 DEBUG "pod": "ds-idrepo-1" 06:27:30 DEBUG 06:27:30 DEBUG 06:27:30 DEBUG "resultType": "vector", 06:27:30 DEBUG 06:27:30 DEBUG [...] 06:27:30 DEBUG 06:27:30 DEBUG "instance": "10.142.0.77:9100" 06:27:30 DEBUG }, 06:27:30 DEBUG 06:27:30 DEBUG }, 06:27:30 DEBUG "result": [ 06:27:30 DEBUG 06:27:30 DEBUG "result": [ 06:27:30 DEBUG 06:27:30 DEBUG "metric": { 06:27:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:27:30 DEBUG }, 06:27:30 DEBUG "value": [ 06:27:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:27:30 DEBUG "value": [ 06:27:30 DEBUG { 06:27:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:27:30 DEBUG { 06:27:30 DEBUG 06:27:30 DEBUG }, 06:27:30 DEBUG "value": [ 06:27:30 DEBUG 1712726849, 06:27:30 DEBUG "396182019072" 06:27:30 DEBUG }, 06:27:30 DEBUG 1712726849, 06:27:30 DEBUG "metric": { 06:27:30 DEBUG }, 06:27:30 DEBUG "metric": { 06:27:30 DEBUG "pod": "ds-cts-2" 06:27:30 DEBUG 06:27:30 DEBUG 06:27:30 DEBUG ] 06:27:30 DEBUG "value": [ 06:27:30 DEBUG "value": [ 06:27:30 DEBUG "0" 06:27:30 DEBUG "pod": "ds-idrepo-0" 06:27:30 DEBUG "value": [ 06:27:30 DEBUG "instance": "10.142.0.25:9100" 06:27:30 DEBUG }, 06:27:30 DEBUG 06:27:30 DEBUG [...] 06:27:30 DEBUG } 06:27:30 DEBUG 06:27:30 DEBUG 1712726849, 06:27:30 DEBUG ] 06:27:30 DEBUG } 06:27:30 DEBUG 1712726849, 06:27:30 DEBUG }, 06:27:30 DEBUG "value": [ 06:27:30 DEBUG 1712726849, 06:27:30 DEBUG 06:27:30 DEBUG ] 06:27:30 DEBUG 06:27:30 DEBUG "258749.5" 06:27:30 DEBUG ] 06:27:30 DEBUG }, 06:27:30 DEBUG "1445.7369508699421" 06:27:30 DEBUG "value": [ 06:27:30 DEBUG "0" 06:27:30 DEBUG ] 06:27:30 DEBUG 06:27:30 DEBUG } 06:27:30 DEBUG 06:27:30 DEBUG [...] 06:27:30 DEBUG } 06:27:30 DEBUG "value": [ 06:27:30 DEBUG ] 06:27:30 DEBUG 06:27:30 DEBUG } 06:27:30 DEBUG 06:27:30 DEBUG } 06:27:30 DEBUG 06:27:30 DEBUG 06:27:30 DEBUG 06:27:30 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:27:30 DEBUG } 06:27:30 DEBUG 06:27:30 DEBUG 06:27:30 DEBUG [...] 06:27:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:27:30 DEBUG ] 06:27:30 DEBUG 06:27:30 DEBUG 06:27:30 DEBUG }, 06:27:30 DEBUG ] 06:27:30 DEBUG 06:27:30 DEBUG ] 06:27:30 DEBUG }, 06:27:30 DEBUG } 06:27:30 DEBUG 06:27:30 DEBUG } 06:27:30 DEBUG 06:27:30 DEBUG "value": [ 06:27:30 DEBUG } 06:27:30 DEBUG 06:27:30 DEBUG } 06:27:30 DEBUG "value": [ 06:27:30 DEBUG ] 06:27:30 DEBUG [...] 06:27:30 DEBUG 06:27:30 DEBUG 1712726849, 06:27:30 DEBUG } 06:27:30 DEBUG 06:27:30 DEBUG } 06:27:30 DEBUG 1712726849, 06:27:30 DEBUG } 06:27:30 DEBUG 06:27:30 DEBUG 06:27:30 DEBUG "0" 06:27:30 DEBUG "instance": "10.142.0.77:9100" 06:27:30 DEBUG "1445.0036664222384" 06:27:30 DEBUG } 06:27:30 DEBUG "pod": "ds-cts-2" 06:27:30 DEBUG 06:27:30 DEBUG ] 06:27:30 DEBUG }, 06:27:30 DEBUG "value": [ 06:27:30 DEBUG 1712726849, 06:27:30 DEBUG 06:27:30 DEBUG } 06:27:30 DEBUG ] 06:27:30 DEBUG } 06:27:30 DEBUG }, 06:27:30 DEBUG [...] 06:27:30 DEBUG 06:27:30 DEBUG "0.16026666666667236" 06:27:30 DEBUG ] 06:27:30 DEBUG } 06:27:30 DEBUG 06:27:30 DEBUG } 06:27:30 DEBUG "value": [ 06:27:30 DEBUG ] 06:27:30 DEBUG } 06:27:30 DEBUG 06:27:30 DEBUG 1712726849, 06:27:30 DEBUG "29.96267164378083" 06:27:30 DEBUG ] 06:27:30 DEBUG "instance": "10.142.0.77:9100" 06:27:30 DEBUG } 06:27:30 DEBUG } 06:27:30 DEBUG ] 06:27:30 DEBUG } 06:27:30 DEBUG ] 06:27:30 DEBUG } 06:27:30 DEBUG } 06:27:30 DEBUG ] 06:27:30 DEBUG } 06:27:30 DEBUG } 06:27:30 DEBUG }, 06:27:30 DEBUG "value": [ 06:27:30 DEBUG 1712726849, 06:27:30 DEBUG "42502343.96666667" 06:27:30 DEBUG ] 06:27:30 DEBUG } 06:27:30 DEBUG ] 06:27:30 DEBUG } 06:27:30 DEBUG } 06:27:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726850" 06:27:30 INFO [http_cmd]: http status code OK 06:27:30 DEBUG --- status code --- 06:27:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:30 DEBUG --- http response --- 06:27:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:27:30 DEBUG ----- output ----- 06:27:30 DEBUG { 06:27:30 DEBUG "status": "success", 06:27:30 DEBUG "data": { 06:27:30 DEBUG "resultType": "vector", 06:27:30 DEBUG "result": [ 06:27:30 DEBUG { 06:27:30 DEBUG "metric": { 06:27:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:27:30 DEBUG }, 06:27:30 DEBUG "value": [ 06:27:30 DEBUG 06:27:30 DEBUG 06:27:30 DEBUG 06:27:30 DEBUG [...] 06:27:30 DEBUG 06:27:30 DEBUG 06:27:30 DEBUG 06:27:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:27:30 DEBUG }, 06:27:30 DEBUG "value": [ 06:27:30 DEBUG 1712726850, 06:27:30 DEBUG "0" 06:27:30 DEBUG ] 06:27:30 DEBUG } 06:27:30 DEBUG ] 06:27:30 DEBUG } 06:27:30 DEBUG } 06:27:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712726850" 06:27:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712726850" 06:27:30 INFO [http_cmd]: http status code OK 06:27:30 DEBUG --- status code --- 06:27:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:30 DEBUG --- http response --- 06:27:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:27:30 DEBUG ----- output ----- 06:27:30 DEBUG { 06:27:30 DEBUG "status": "success", 06:27:30 DEBUG "data": { 06:27:30 DEBUG "resultType": "vector", 06:27:30 DEBUG "result": [ 06:27:30 DEBUG { 06:27:30 DEBUG "metric": { 06:27:30 DEBUG "pod": "ds-idrepo-0" 06:27:30 DEBUG }, 06:27:30 DEBUG "value": [ 06:27:30 DEBUG 06:27:30 DEBUG 06:27:30 DEBUG 06:27:30 DEBUG [...] 06:27:30 DEBUG 06:27:30 DEBUG 06:27:30 DEBUG 06:27:30 DEBUG "pod": "ds-cts-2" 06:27:30 INFO [http_cmd]: http status code OK 06:27:30 DEBUG }, 06:27:30 DEBUG --- status code --- 06:27:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:30 DEBUG --- http response --- 06:27:30 DEBUG "value": [ 06:27:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:27:30 DEBUG ----- output ----- 06:27:30 DEBUG { 06:27:30 DEBUG 1712726850, 06:27:30 DEBUG "status": "success", 06:27:30 DEBUG "data": { 06:27:30 DEBUG "resultType": "vector", 06:27:30 DEBUG "0" 06:27:30 DEBUG "result": [ 06:27:30 DEBUG ] 06:27:30 DEBUG { 06:27:30 DEBUG } 06:27:30 DEBUG "metric": { 06:27:30 DEBUG ] 06:27:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:27:30 DEBUG } 06:27:30 DEBUG }, 06:27:30 DEBUG } 06:27:30 DEBUG "value": [ 06:27:30 DEBUG 06:27:30 DEBUG 06:27:30 DEBUG 06:27:30 DEBUG [...] 06:27:30 DEBUG 06:27:30 DEBUG 06:27:30 DEBUG 06:27:30 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:27:30 DEBUG }, 06:27:30 DEBUG "value": [ 06:27:30 DEBUG 1712726850, 06:27:30 DEBUG "4.3743277210591875" 06:27:30 DEBUG ] 06:27:30 DEBUG } 06:27:30 DEBUG ] 06:27:30 DEBUG } 06:27:30 DEBUG } 06:27:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712726851" 06:27:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726851" 06:27:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726851" 06:27:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726851" 06:27:31 INFO [http_cmd]: http status code OK 06:27:31 DEBUG --- status code --- 06:27:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:31 DEBUG --- http response --- 06:27:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:27:31 DEBUG ----- output ----- 06:27:31 DEBUG { 06:27:31 DEBUG "status": "success", 06:27:31 DEBUG "data": { 06:27:31 DEBUG "resultType": "vector", 06:27:31 DEBUG "result": [ 06:27:31 DEBUG { 06:27:31 DEBUG "metric": { 06:27:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:27:31 DEBUG }, 06:27:31 DEBUG "value": [ 06:27:31 DEBUG 06:27:31 DEBUG 06:27:31 DEBUG 06:27:31 DEBUG [...] 06:27:31 DEBUG 06:27:31 DEBUG 06:27:31 DEBUG 06:27:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:27:31 DEBUG }, 06:27:31 DEBUG "value": [ 06:27:31 DEBUG 1712726851, 06:27:31 DEBUG "0.051" 06:27:31 DEBUG ] 06:27:31 DEBUG } 06:27:31 DEBUG ] 06:27:31 DEBUG } 06:27:31 DEBUG } 06:27:31 INFO [http_cmd]: http status code OK 06:27:31 DEBUG --- status code --- 06:27:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:31 DEBUG --- http response --- 06:27:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:27:31 DEBUG ----- output ----- 06:27:31 DEBUG { 06:27:31 DEBUG "status": "success", 06:27:31 DEBUG "data": { 06:27:31 DEBUG "resultType": "vector", 06:27:31 DEBUG "result": [ 06:27:31 DEBUG { 06:27:31 DEBUG "metric": { 06:27:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:27:31 DEBUG }, 06:27:31 DEBUG "value": [ 06:27:31 DEBUG 06:27:31 DEBUG 06:27:31 DEBUG 06:27:31 DEBUG [...] 06:27:31 DEBUG 06:27:31 DEBUG 06:27:31 DEBUG 06:27:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:27:31 DEBUG }, 06:27:31 DEBUG "value": [ 06:27:31 DEBUG 1712726851, 06:27:31 DEBUG "1445.470301979868" 06:27:31 DEBUG ] 06:27:31 DEBUG } 06:27:31 DEBUG ] 06:27:31 DEBUG } 06:27:31 DEBUG } 06:27:31 INFO [http_cmd]: http status code OK 06:27:31 DEBUG --- status code --- 06:27:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:31 DEBUG --- http response --- 06:27:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:27:31 INFO [http_cmd]: http status code OK 06:27:31 DEBUG --- status code --- 06:27:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:31 DEBUG --- http response --- 06:27:31 DEBUG ----- output ----- 06:27:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:27:31 DEBUG ----- output ----- 06:27:31 DEBUG { 06:27:31 DEBUG "status": "success", 06:27:31 DEBUG "data": { 06:27:31 DEBUG "resultType": "vector", 06:27:31 DEBUG "result": [ 06:27:31 DEBUG { 06:27:31 DEBUG "metric": { 06:27:31 DEBUG "pod": "ds-idrepo-2" 06:27:31 DEBUG }, 06:27:31 DEBUG "value": [ 06:27:31 DEBUG 06:27:31 DEBUG 06:27:31 DEBUG 06:27:31 DEBUG [...] 06:27:31 DEBUG 06:27:31 DEBUG 06:27:31 DEBUG 06:27:31 DEBUG "pod": "ds-cts-2" 06:27:31 DEBUG }, 06:27:31 DEBUG "value": [ 06:27:31 DEBUG 1712726851, 06:27:31 DEBUG "0.00013385659689819134" 06:27:31 DEBUG ] 06:27:31 DEBUG } 06:27:31 DEBUG ] 06:27:31 DEBUG } 06:27:31 DEBUG } 06:27:31 DEBUG { 06:27:31 DEBUG "status": "success", 06:27:31 DEBUG "data": { 06:27:31 DEBUG "resultType": "vector", 06:27:31 DEBUG "result": [ 06:27:31 DEBUG { 06:27:31 DEBUG "metric": { 06:27:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:27:31 DEBUG }, 06:27:31 DEBUG "value": [ 06:27:31 DEBUG 06:27:31 DEBUG 06:27:31 DEBUG 06:27:31 DEBUG [...] 06:27:31 DEBUG 06:27:31 DEBUG 06:27:31 DEBUG 06:27:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:27:31 DEBUG }, 06:27:31 DEBUG "value": [ 06:27:31 DEBUG 1712726851, 06:27:31 DEBUG "0" 06:27:31 DEBUG ] 06:27:31 DEBUG } 06:27:31 DEBUG ] 06:27:31 DEBUG } 06:27:31 DEBUG } 06:27:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726851" 06:27:31 INFO [http_cmd]: http status code OK 06:27:31 DEBUG --- status code --- 06:27:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:31 DEBUG --- http response --- 06:27:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:27:31 DEBUG ----- output ----- 06:27:31 DEBUG { 06:27:31 DEBUG "status": "success", 06:27:31 DEBUG "data": { 06:27:31 DEBUG "resultType": "vector", 06:27:31 DEBUG "result": [ 06:27:31 DEBUG { 06:27:31 DEBUG "metric": { 06:27:31 DEBUG "pod": "ds-idrepo-0" 06:27:31 DEBUG }, 06:27:31 DEBUG "value": [ 06:27:31 DEBUG 06:27:31 DEBUG 06:27:31 DEBUG 06:27:31 DEBUG [...] 06:27:31 DEBUG 06:27:31 DEBUG 06:27:31 DEBUG 06:27:31 DEBUG "pod": "ds-idrepo-1" 06:27:31 DEBUG }, 06:27:31 DEBUG "value": [ 06:27:31 DEBUG 1712726851, 06:27:31 DEBUG "0" 06:27:31 DEBUG ] 06:27:31 DEBUG } 06:27:31 DEBUG ] 06:27:31 DEBUG } 06:27:31 DEBUG } 06:27:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726851" 06:27:31 INFO [http_cmd]: http status code OK 06:27:31 DEBUG --- status code --- 06:27:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:31 DEBUG --- http response --- 06:27:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:27:31 DEBUG ----- output ----- 06:27:31 DEBUG { 06:27:31 DEBUG "status": "success", 06:27:31 DEBUG "data": { 06:27:31 DEBUG "resultType": "vector", 06:27:31 DEBUG "result": [ 06:27:31 DEBUG { 06:27:31 DEBUG "metric": { 06:27:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:27:31 DEBUG }, 06:27:31 DEBUG "value": [ 06:27:31 DEBUG 06:27:31 DEBUG 06:27:31 DEBUG 06:27:31 DEBUG [...] 06:27:31 DEBUG 06:27:31 DEBUG 06:27:31 DEBUG 06:27:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:27:31 DEBUG }, 06:27:31 DEBUG "value": [ 06:27:31 DEBUG 1712726851, 06:27:31 DEBUG "1435.9042730484634" 06:27:31 DEBUG ] 06:27:31 DEBUG } 06:27:31 DEBUG ] 06:27:31 DEBUG } 06:27:31 DEBUG } 06:27:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726851" 06:27:31 INFO [http_cmd]: http status code OK 06:27:31 DEBUG --- status code --- 06:27:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:31 DEBUG --- http response --- 06:27:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:27:31 DEBUG ----- output ----- 06:27:31 DEBUG { 06:27:31 DEBUG "status": "success", 06:27:31 DEBUG "data": { 06:27:31 DEBUG "resultType": "vector", 06:27:31 DEBUG "result": [ 06:27:31 DEBUG { 06:27:31 DEBUG "metric": { 06:27:31 DEBUG "pod": "ds-idrepo-0" 06:27:31 DEBUG }, 06:27:31 DEBUG "value": [ 06:27:31 DEBUG 06:27:31 DEBUG 06:27:31 DEBUG 06:27:31 DEBUG [...] 06:27:31 DEBUG 06:27:31 DEBUG 06:27:31 DEBUG 06:27:31 DEBUG "pod": "ds-idrepo-1" 06:27:31 DEBUG }, 06:27:31 DEBUG "value": [ 06:27:31 DEBUG 1712726851, 06:27:31 DEBUG "0" 06:27:31 DEBUG ] 06:27:31 DEBUG } 06:27:31 DEBUG ] 06:27:31 DEBUG } 06:27:31 DEBUG } 06:27:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726851" 06:27:32 INFO [http_cmd]: http status code OK 06:27:32 DEBUG --- status code --- 06:27:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:32 DEBUG --- http response --- 06:27:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:27:32 DEBUG ----- output ----- 06:27:32 DEBUG { 06:27:32 DEBUG "status": "success", 06:27:32 DEBUG "data": { 06:27:32 DEBUG "resultType": "vector", 06:27:32 DEBUG "result": [ 06:27:32 DEBUG { 06:27:32 DEBUG "metric": { 06:27:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:27:32 DEBUG }, 06:27:32 DEBUG "value": [ 06:27:32 DEBUG 06:27:32 DEBUG 06:27:32 DEBUG 06:27:32 DEBUG [...] 06:27:32 DEBUG 06:27:32 DEBUG 06:27:32 DEBUG 06:27:32 DEBUG "pod": "ds-cts-0" 06:27:32 DEBUG }, 06:27:32 DEBUG "value": [ 06:27:32 DEBUG 1712726851, 06:27:32 DEBUG "23632304.84498147" 06:27:32 DEBUG ] 06:27:32 DEBUG } 06:27:32 DEBUG ] 06:27:32 DEBUG } 06:27:32 DEBUG } 06:27:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712726852" 06:27:32 INFO [http_cmd]: http status code OK 06:27:32 DEBUG --- status code --- 06:27:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:32 DEBUG --- http response --- 06:27:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:27:32 DEBUG ----- output ----- 06:27:32 DEBUG { 06:27:32 DEBUG "status": "success", 06:27:32 DEBUG "data": { 06:27:32 DEBUG "resultType": "vector", 06:27:32 DEBUG "result": [ 06:27:32 DEBUG { 06:27:32 DEBUG "metric": { 06:27:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:27:32 DEBUG }, 06:27:32 DEBUG "value": [ 06:27:32 DEBUG 06:27:32 DEBUG 06:27:32 DEBUG 06:27:32 DEBUG [...] 06:27:32 DEBUG 06:27:32 DEBUG 06:27:32 DEBUG 06:27:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:27:32 DEBUG }, 06:27:32 DEBUG "value": [ 06:27:32 DEBUG 1712726852, 06:27:32 DEBUG "2483654656" 06:27:32 DEBUG ] 06:27:32 DEBUG } 06:27:32 DEBUG ] 06:27:32 DEBUG } 06:27:32 DEBUG } 06:27:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726852" 06:27:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726852" 06:27:32 INFO [http_cmd]: http status code OK 06:27:32 DEBUG --- status code --- 06:27:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:32 DEBUG --- http response --- 06:27:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:27:32 DEBUG ----- output ----- 06:27:32 DEBUG { 06:27:32 DEBUG "status": "success", 06:27:32 DEBUG "data": { 06:27:32 DEBUG "resultType": "vector", 06:27:32 DEBUG "result": [ 06:27:32 DEBUG { 06:27:32 DEBUG "metric": { 06:27:32 DEBUG "instance": "10.142.0.25:9100" 06:27:32 DEBUG }, 06:27:32 DEBUG "value": [ 06:27:32 DEBUG 06:27:32 DEBUG 06:27:32 DEBUG 06:27:32 DEBUG [...] 06:27:32 DEBUG 06:27:32 DEBUG 06:27:32 DEBUG 06:27:32 DEBUG "instance": "10.142.0.77:9100" 06:27:32 DEBUG }, 06:27:32 DEBUG "value": [ 06:27:32 DEBUG 1712726852, 06:27:32 DEBUG "42506589.599999994" 06:27:32 DEBUG ] 06:27:32 DEBUG } 06:27:32 DEBUG ] 06:27:32 DEBUG } 06:27:32 DEBUG } 06:27:32 INFO [http_cmd]: http status code OK 06:27:32 DEBUG --- status code --- 06:27:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:27:32 DEBUG --- http response --- 06:27:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:27:32 DEBUG ----- output ----- 06:27:32 DEBUG { 06:27:32 DEBUG "status": "success", 06:27:32 DEBUG "data": { 06:27:32 DEBUG "resultType": "vector", 06:27:32 DEBUG "result": [ 06:27:32 DEBUG { 06:27:32 DEBUG "metric": { 06:27:32 DEBUG "instance": "10.142.0.25:9100" 06:27:32 DEBUG }, 06:27:32 DEBUG "value": [ 06:27:32 DEBUG 06:27:32 DEBUG 06:27:32 DEBUG 06:27:32 DEBUG [...] 06:27:32 DEBUG 06:27:32 DEBUG 06:27:32 DEBUG 06:27:32 DEBUG "instance": "10.142.0.77:9100" 06:27:32 DEBUG }, 06:27:32 DEBUG "value": [ 06:27:32 DEBUG 1712726852, 06:27:32 DEBUG "0.048333333333333554" 06:27:32 DEBUG ] 06:27:32 DEBUG } 06:27:32 DEBUG ] 06:27:32 DEBUG } 06:27:32 DEBUG } 06:27:50 INFO 06:27:50 INFO [loop_until]: kubectl --namespace=xlou top pods 06:27:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:27:51 INFO [loop_until]: OK (rc = 0) 06:27:51 DEBUG --- stdout --- 06:27:51 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 11106m 9011Mi am-654c4fcbcf-9scrj 9890m 8448Mi am-654c4fcbcf-v5qb5 8433m 8446Mi ds-cts-0 8433m 13393Mi ds-cts-1 8861m 13393Mi ds-cts-2 7621m 13394Mi ds-idrepo-0 10479m 5623Mi ds-idrepo-1 363m 5672Mi ds-idrepo-2 375m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1190Mi idm-6bfb7c96cc-vsb58 5m 2213Mi lodemon-b776b7d88-thrqf 6m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4359m 2343Mi 06:27:51 DEBUG --- stderr --- 06:27:51 DEBUG 06:27:59 INFO 06:27:59 INFO [loop_until]: kubectl --namespace=xlou top node 06:27:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:27:59 INFO [loop_until]: OK (rc = 0) 06:27:59 DEBUG --- stdout --- 06:27:59 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 94m 0% 2308Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9288m 58% 9322Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 77m 0% 3254Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 14001m 88% 6738Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10932m 68% 9336Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11558m 72% 9812Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 75m 0% 852Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10909m 68% 6302Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8627m 54% 14101Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 7565m 47% 14067Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 432m 2% 6389Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8766m 55% 14096Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 413m 2% 6356Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4475m 28% 5936Mi 10% 06:27:59 DEBUG --- stderr --- 06:27:59 DEBUG 06:28:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712726908" 06:28:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726908" 06:28:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726908" 06:28:28 INFO [http_cmd]: http status code OK 06:28:28 ERROR [http_cmd]: ERROR 06:28:28 DEBUG --- status code --- 06:28:28 ERROR --- status code --- 06:28:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:28 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:28:28 DEBUG --- http response --- 06:28:28 ERROR --- http response --- 06:28:28 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:28:28 INFO [http_cmd]: http status code OK 06:28:28 DEBUG --- status code --- 06:28:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:28 DEBUG --- http response --- 06:28:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:28:28 DEBUG ----- output ----- 06:28:28 DEBUG { 06:28:28 DEBUG "status": "success", 06:28:28 DEBUG "data": { 06:28:28 DEBUG "resultType": "vector", 06:28:28 DEBUG "result": [ 06:28:28 DEBUG { 06:28:28 DEBUG "metric": { 06:28:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:28:28 DEBUG }, 06:28:28 DEBUG "value": [ 06:28:28 DEBUG 06:28:28 DEBUG 06:28:28 DEBUG 06:28:28 DEBUG [...] 06:28:28 DEBUG 06:28:28 DEBUG 06:28:28 DEBUG 06:28:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:28:28 DEBUG }, 06:28:28 DEBUG "value": [ 06:28:28 DEBUG 1712726908, 06:28:28 DEBUG "0" 06:28:28 DEBUG ] 06:28:28 DEBUG } 06:28:28 DEBUG ] 06:28:28 DEBUG } 06:28:28 DEBUG } 06:28:28 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:28:28 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:28:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726910" 06:28:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726910" 06:28:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726910" 06:28:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726910" 06:28:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726910" 06:28:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726910" 06:28:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726910" 06:28:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726910" 06:28:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726910" 06:28:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726910" 06:28:30 INFO [http_cmd]: http status code OK 06:28:30 DEBUG --- status code --- 06:28:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:30 DEBUG --- http response --- 06:28:30 INFO [http_cmd]: http status code OK 06:28:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:28:30 DEBUG ----- output ----- 06:28:30 INFO [http_cmd]: http status code OK 06:28:30 INFO [http_cmd]: http status code OK 06:28:30 DEBUG --- status code --- 06:28:30 INFO [http_cmd]: http status code OK 06:28:30 DEBUG --- status code --- 06:28:30 DEBUG { 06:28:30 DEBUG "status": "success", 06:28:30 DEBUG --- status code --- 06:28:30 INFO [http_cmd]: http status code OK 06:28:30 DEBUG --- status code --- 06:28:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:30 DEBUG --- http response --- 06:28:30 INFO [http_cmd]: http status code OK 06:28:30 INFO [http_cmd]: http status code OK 06:28:30 DEBUG "data": { 06:28:30 INFO [http_cmd]: http status code OK 06:28:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:30 DEBUG --- http response --- 06:28:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:28:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:30 DEBUG --- http response --- 06:28:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:28:30 DEBUG ----- output ----- 06:28:30 DEBUG { 06:28:30 DEBUG "resultType": "vector", 06:28:30 DEBUG "result": [ 06:28:30 DEBUG { 06:28:30 DEBUG "metric": { 06:28:30 DEBUG --- status code --- 06:28:30 INFO [http_cmd]: http status code OK 06:28:30 DEBUG --- status code --- 06:28:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:30 DEBUG --- http response --- 06:28:30 DEBUG --- status code --- 06:28:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:30 DEBUG "instance": "10.142.0.25:9100" 06:28:30 DEBUG }, 06:28:30 DEBUG "value": [ 06:28:30 DEBUG 06:28:30 DEBUG --- status code --- 06:28:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:28:30 DEBUG ----- output ----- 06:28:30 DEBUG { 06:28:30 DEBUG "status": "success", 06:28:30 DEBUG "data": { 06:28:30 DEBUG "resultType": "vector", 06:28:30 DEBUG "result": [ 06:28:30 DEBUG { 06:28:30 DEBUG "metric": { 06:28:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:28:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:30 DEBUG --- http response --- 06:28:30 DEBUG --- status code --- 06:28:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:30 DEBUG --- http response --- 06:28:30 DEBUG }, 06:28:30 DEBUG "value": [ 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:28:30 DEBUG ----- output ----- 06:28:30 DEBUG { 06:28:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:30 DEBUG --- http response --- 06:28:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:28:30 DEBUG --- http response --- 06:28:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:28:30 DEBUG 06:28:30 DEBUG [...] 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG --- http response --- 06:28:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:28:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:28:30 DEBUG ----- output ----- 06:28:30 DEBUG { 06:28:30 DEBUG ----- output ----- 06:28:30 DEBUG { 06:28:30 DEBUG "status": "success", 06:28:30 DEBUG "data": { 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG [...] 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG "instance": "10.142.0.77:9100" 06:28:30 DEBUG }, 06:28:30 DEBUG ----- output ----- 06:28:30 DEBUG { 06:28:30 DEBUG "status": "success", 06:28:30 DEBUG "value": [ 06:28:30 DEBUG 1712726910, 06:28:30 DEBUG "398458350592" 06:28:30 DEBUG ] 06:28:30 DEBUG } 06:28:30 DEBUG ] 06:28:30 DEBUG "data": { 06:28:30 DEBUG "resultType": "vector", 06:28:30 DEBUG "result": [ 06:28:30 DEBUG { 06:28:30 DEBUG "status": "success", 06:28:30 DEBUG "data": { 06:28:30 DEBUG "resultType": "vector", 06:28:30 DEBUG "result": [ 06:28:30 DEBUG { 06:28:30 DEBUG "metric": { 06:28:30 DEBUG "pod": "ds-idrepo-0" 06:28:30 DEBUG }, 06:28:30 DEBUG "value": [ 06:28:30 DEBUG 06:28:30 DEBUG "metric": { 06:28:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:28:30 DEBUG }, 06:28:30 DEBUG "value": [ 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG "status": "success", 06:28:30 DEBUG "data": { 06:28:30 DEBUG "resultType": "vector", 06:28:30 DEBUG "result": [ 06:28:30 DEBUG 06:28:30 DEBUG [...] 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG { 06:28:30 DEBUG "metric": { 06:28:30 DEBUG "pod": "ds-idrepo-0" 06:28:30 DEBUG }, 06:28:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:28:30 DEBUG }, 06:28:30 DEBUG "value": [ 06:28:30 DEBUG 1712726910, 06:28:30 DEBUG "1434.8" 06:28:30 DEBUG "value": [ 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG ] 06:28:30 DEBUG } 06:28:30 DEBUG ] 06:28:30 DEBUG } 06:28:30 DEBUG } 06:28:30 DEBUG [...] 06:28:30 DEBUG ----- output ----- 06:28:30 DEBUG "resultType": "vector", 06:28:30 DEBUG "result": [ 06:28:30 DEBUG ----- output ----- 06:28:30 DEBUG { 06:28:30 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG { 06:28:30 DEBUG { 06:28:30 DEBUG "metric": { 06:28:30 DEBUG } 06:28:30 DEBUG "status": "success", 06:28:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:28:30 DEBUG ----- output ----- 06:28:30 DEBUG { 06:28:30 DEBUG 06:28:30 DEBUG "status": "success", 06:28:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:28:30 DEBUG }, 06:28:30 DEBUG "value": [ 06:28:30 DEBUG 06:28:30 DEBUG "status": "success", 06:28:30 DEBUG "data": { 06:28:30 DEBUG "resultType": "vector", 06:28:30 DEBUG [...] 06:28:30 DEBUG "data": { 06:28:30 DEBUG "status": "success", 06:28:30 DEBUG } 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG [...] 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG "resultType": "vector", 06:28:30 DEBUG "data": { 06:28:30 DEBUG "resultType": "vector", 06:28:30 DEBUG "result": [ 06:28:30 DEBUG "result": [ 06:28:30 DEBUG }, 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG "result": [ 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:28:30 DEBUG "data": { 06:28:30 DEBUG "value": [ 06:28:30 DEBUG 1712726910, 06:28:30 DEBUG "0" 06:28:30 DEBUG ] 06:28:30 DEBUG { 06:28:30 DEBUG "metric": { 06:28:30 DEBUG "instance": "10.142.0.25:9100" 06:28:30 DEBUG }, 06:28:30 DEBUG "value": [ 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG "resultType": "vector", 06:28:30 DEBUG { 06:28:30 DEBUG "metric": { 06:28:30 DEBUG "instance": "10.142.0.25:9100" 06:28:30 DEBUG [...] 06:28:30 DEBUG { 06:28:30 DEBUG "metric": { 06:28:30 DEBUG "pod": "ds-idrepo-0" 06:28:30 DEBUG 06:28:30 DEBUG }, 06:28:30 DEBUG "value": [ 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG }, 06:28:30 DEBUG "value": [ 06:28:30 DEBUG "pod": "ds-cts-2" 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG [...] 06:28:30 DEBUG 06:28:30 DEBUG "instance": "10.142.0.77:9100" 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG }, 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG }, 06:28:30 DEBUG "value": [ 06:28:30 DEBUG 06:28:30 DEBUG [...] 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG "pod": "ds-cts-2" 06:28:30 DEBUG 1712726910, 06:28:30 DEBUG "42590220.166666664" 06:28:30 DEBUG ] 06:28:30 DEBUG }, 06:28:30 DEBUG "value": [ 06:28:30 DEBUG 1712726910, 06:28:30 DEBUG "0" 06:28:30 DEBUG ] 06:28:30 DEBUG } 06:28:30 DEBUG ] 06:28:30 DEBUG } 06:28:30 DEBUG } 06:28:30 DEBUG ] 06:28:30 DEBUG } 06:28:30 DEBUG } 06:28:30 DEBUG "pod": "ds-idrepo-1" 06:28:30 DEBUG } 06:28:30 DEBUG "result": [ 06:28:30 DEBUG { 06:28:30 DEBUG }, 06:28:30 DEBUG "value": [ 06:28:30 DEBUG 1712726910, 06:28:30 DEBUG "0" 06:28:30 DEBUG "metric": { 06:28:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:28:30 DEBUG ] 06:28:30 DEBUG } 06:28:30 DEBUG ] 06:28:30 DEBUG } 06:28:30 DEBUG }, 06:28:30 DEBUG "value": [ 06:28:30 DEBUG } 06:28:30 DEBUG "instance": "10.142.0.77:9100" 06:28:30 DEBUG }, 06:28:30 DEBUG "value": [ 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG 1712726910, 06:28:30 DEBUG "0.12169999999999857" 06:28:30 DEBUG ] 06:28:30 DEBUG 06:28:30 DEBUG [...] 06:28:30 DEBUG } 06:28:30 DEBUG ] 06:28:30 DEBUG } 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG } 06:28:30 DEBUG } 06:28:30 DEBUG ] 06:28:30 DEBUG 06:28:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:28:30 DEBUG } 06:28:30 DEBUG } 06:28:30 DEBUG }, 06:28:30 DEBUG "value": [ 06:28:30 DEBUG }, 06:28:30 DEBUG "value": [ 06:28:30 DEBUG 1712726910, 06:28:30 DEBUG "value": [ 06:28:30 DEBUG 1712726910, 06:28:30 DEBUG 1712726910, 06:28:30 DEBUG "258420.5" 06:28:30 DEBUG "1435.2" 06:28:30 DEBUG ] 06:28:30 DEBUG } 06:28:30 DEBUG ] 06:28:30 DEBUG ] 06:28:30 DEBUG } 06:28:30 DEBUG } 06:28:30 DEBUG } 06:28:30 DEBUG "29.763690297636902" 06:28:30 DEBUG ] 06:28:30 DEBUG } 06:28:30 DEBUG } 06:28:30 DEBUG ] 06:28:30 DEBUG } 06:28:30 DEBUG ] 06:28:30 DEBUG } 06:28:30 DEBUG } 06:28:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726910" 06:28:30 INFO [http_cmd]: http status code OK 06:28:30 DEBUG --- status code --- 06:28:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:30 DEBUG --- http response --- 06:28:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:28:30 DEBUG ----- output ----- 06:28:30 DEBUG { 06:28:30 DEBUG "status": "success", 06:28:30 DEBUG "data": { 06:28:30 DEBUG "resultType": "vector", 06:28:30 DEBUG "result": [ 06:28:30 DEBUG { 06:28:30 DEBUG "metric": { 06:28:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:28:30 DEBUG }, 06:28:30 DEBUG "value": [ 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG [...] 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:28:30 DEBUG }, 06:28:30 DEBUG "value": [ 06:28:30 DEBUG 1712726910, 06:28:30 DEBUG "38722249.93228066" 06:28:30 DEBUG ] 06:28:30 DEBUG } 06:28:30 DEBUG ] 06:28:30 DEBUG } 06:28:30 DEBUG } 06:28:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712726910" 06:28:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712726910" 06:28:30 INFO [http_cmd]: http status code OK 06:28:30 DEBUG --- status code --- 06:28:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:30 DEBUG --- http response --- 06:28:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:28:30 DEBUG ----- output ----- 06:28:30 DEBUG { 06:28:30 DEBUG "status": "success", 06:28:30 DEBUG "data": { 06:28:30 DEBUG "resultType": "vector", 06:28:30 DEBUG "result": [ 06:28:30 DEBUG { 06:28:30 DEBUG "metric": { 06:28:30 DEBUG "pod": "ds-idrepo-0" 06:28:30 DEBUG }, 06:28:30 DEBUG "value": [ 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG [...] 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG "pod": "ds-cts-2" 06:28:30 DEBUG }, 06:28:30 DEBUG "value": [ 06:28:30 DEBUG 1712726910, 06:28:30 DEBUG "0" 06:28:30 DEBUG ] 06:28:30 DEBUG } 06:28:30 DEBUG ] 06:28:30 DEBUG } 06:28:30 DEBUG } 06:28:30 INFO [http_cmd]: http status code OK 06:28:30 DEBUG --- status code --- 06:28:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:30 DEBUG --- http response --- 06:28:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:28:30 DEBUG ----- output ----- 06:28:30 DEBUG { 06:28:30 DEBUG "status": "success", 06:28:30 DEBUG "data": { 06:28:30 DEBUG "resultType": "vector", 06:28:30 DEBUG "result": [ 06:28:30 DEBUG { 06:28:30 DEBUG "metric": { 06:28:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:28:30 DEBUG }, 06:28:30 DEBUG "value": [ 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG [...] 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG 06:28:30 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:28:30 DEBUG }, 06:28:30 DEBUG "value": [ 06:28:30 DEBUG 1712726910, 06:28:30 DEBUG "4.372800952317637" 06:28:30 DEBUG ] 06:28:30 DEBUG } 06:28:30 DEBUG ] 06:28:30 DEBUG } 06:28:30 DEBUG } 06:28:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712726911" 06:28:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726911" 06:28:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726911" 06:28:31 INFO [http_cmd]: http status code OK 06:28:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726911" 06:28:31 DEBUG --- status code --- 06:28:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:31 DEBUG --- http response --- 06:28:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:28:31 DEBUG ----- output ----- 06:28:31 DEBUG { 06:28:31 DEBUG "status": "success", 06:28:31 DEBUG "data": { 06:28:31 DEBUG "resultType": "vector", 06:28:31 DEBUG "result": [ 06:28:31 DEBUG { 06:28:31 DEBUG "metric": { 06:28:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:28:31 DEBUG }, 06:28:31 DEBUG "value": [ 06:28:31 DEBUG 06:28:31 DEBUG 06:28:31 DEBUG 06:28:31 DEBUG [...] 06:28:31 DEBUG 06:28:31 DEBUG 06:28:31 DEBUG 06:28:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:28:31 DEBUG }, 06:28:31 DEBUG "value": [ 06:28:31 DEBUG 1712726911, 06:28:31 DEBUG "0.051" 06:28:31 DEBUG ] 06:28:31 DEBUG } 06:28:31 DEBUG ] 06:28:31 DEBUG } 06:28:31 DEBUG } 06:28:31 INFO [http_cmd]: http status code OK 06:28:31 DEBUG --- status code --- 06:28:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:31 DEBUG --- http response --- 06:28:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:28:31 DEBUG ----- output ----- 06:28:31 DEBUG { 06:28:31 DEBUG "status": "success", 06:28:31 DEBUG "data": { 06:28:31 DEBUG "resultType": "vector", 06:28:31 DEBUG "result": [ 06:28:31 DEBUG { 06:28:31 DEBUG "metric": { 06:28:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:28:31 DEBUG }, 06:28:31 DEBUG "value": [ 06:28:31 DEBUG 06:28:31 DEBUG 06:28:31 DEBUG 06:28:31 DEBUG [...] 06:28:31 DEBUG 06:28:31 DEBUG 06:28:31 DEBUG 06:28:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:28:31 DEBUG }, 06:28:31 INFO [http_cmd]: http status code OK 06:28:31 DEBUG "value": [ 06:28:31 DEBUG 1712726911, 06:28:31 DEBUG "1434.7333333333333" 06:28:31 DEBUG --- status code --- 06:28:31 DEBUG ] 06:28:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:31 DEBUG } 06:28:31 DEBUG --- http response --- 06:28:31 DEBUG ] 06:28:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:28:31 DEBUG ----- output ----- 06:28:31 DEBUG } 06:28:31 DEBUG } 06:28:31 DEBUG { 06:28:31 DEBUG "status": "success", 06:28:31 DEBUG "data": { 06:28:31 DEBUG "resultType": "vector", 06:28:31 DEBUG "result": [ 06:28:31 DEBUG { 06:28:31 DEBUG "metric": { 06:28:31 DEBUG "pod": "ds-idrepo-2" 06:28:31 DEBUG }, 06:28:31 DEBUG "value": [ 06:28:31 DEBUG 06:28:31 DEBUG 06:28:31 DEBUG 06:28:31 DEBUG [...] 06:28:31 DEBUG 06:28:31 DEBUG 06:28:31 DEBUG 06:28:31 DEBUG "pod": "ds-cts-2" 06:28:31 DEBUG }, 06:28:31 DEBUG "value": [ 06:28:31 DEBUG 1712726911, 06:28:31 DEBUG "0.000033330000333300005" 06:28:31 DEBUG ] 06:28:31 DEBUG } 06:28:31 DEBUG ] 06:28:31 DEBUG } 06:28:31 DEBUG } 06:28:31 INFO [http_cmd]: http status code OK 06:28:31 DEBUG --- status code --- 06:28:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:31 DEBUG --- http response --- 06:28:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:28:31 DEBUG ----- output ----- 06:28:31 DEBUG { 06:28:31 DEBUG "status": "success", 06:28:31 DEBUG "data": { 06:28:31 DEBUG "resultType": "vector", 06:28:31 DEBUG "result": [ 06:28:31 DEBUG { 06:28:31 DEBUG "metric": { 06:28:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:28:31 DEBUG }, 06:28:31 DEBUG "value": [ 06:28:31 DEBUG 06:28:31 DEBUG 06:28:31 DEBUG 06:28:31 DEBUG [...] 06:28:31 DEBUG 06:28:31 DEBUG 06:28:31 DEBUG 06:28:31 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:28:31 DEBUG }, 06:28:31 DEBUG "value": [ 06:28:31 DEBUG 1712726911, 06:28:31 DEBUG "0.02019661410037147" 06:28:31 DEBUG ] 06:28:31 DEBUG } 06:28:31 DEBUG ] 06:28:31 DEBUG } 06:28:31 DEBUG } 06:28:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726911" 06:28:31 INFO [http_cmd]: http status code OK 06:28:31 DEBUG --- status code --- 06:28:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:31 DEBUG --- http response --- 06:28:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:28:31 DEBUG ----- output ----- 06:28:31 DEBUG { 06:28:31 DEBUG "status": "success", 06:28:31 DEBUG "data": { 06:28:31 DEBUG "resultType": "vector", 06:28:31 DEBUG "result": [ 06:28:31 DEBUG { 06:28:31 DEBUG "metric": { 06:28:31 DEBUG "pod": "ds-idrepo-0" 06:28:31 DEBUG }, 06:28:31 DEBUG "value": [ 06:28:31 DEBUG 06:28:31 DEBUG 06:28:31 DEBUG 06:28:31 DEBUG [...] 06:28:31 DEBUG 06:28:31 DEBUG 06:28:31 DEBUG 06:28:31 DEBUG "pod": "ds-idrepo-1" 06:28:31 DEBUG }, 06:28:31 DEBUG "value": [ 06:28:31 DEBUG 1712726911, 06:28:31 DEBUG "0" 06:28:31 DEBUG ] 06:28:31 DEBUG } 06:28:31 DEBUG ] 06:28:31 DEBUG } 06:28:31 DEBUG } 06:28:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726911" 06:28:31 INFO [http_cmd]: http status code OK 06:28:31 DEBUG --- status code --- 06:28:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:31 DEBUG --- http response --- 06:28:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:28:31 DEBUG ----- output ----- 06:28:31 DEBUG { 06:28:31 DEBUG "status": "success", 06:28:31 DEBUG "data": { 06:28:31 DEBUG "resultType": "vector", 06:28:31 DEBUG "result": [ 06:28:31 DEBUG { 06:28:31 DEBUG "metric": { 06:28:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:28:31 DEBUG }, 06:28:31 DEBUG "value": [ 06:28:31 DEBUG 06:28:31 DEBUG 06:28:31 DEBUG 06:28:31 DEBUG [...] 06:28:31 DEBUG 06:28:31 DEBUG 06:28:31 DEBUG 06:28:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:28:31 DEBUG }, 06:28:31 DEBUG "value": [ 06:28:31 DEBUG 1712726911, 06:28:31 DEBUG "1433.5666666666666" 06:28:31 DEBUG ] 06:28:31 DEBUG } 06:28:31 DEBUG ] 06:28:31 DEBUG } 06:28:31 DEBUG } 06:28:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726911" 06:28:31 INFO [http_cmd]: http status code OK 06:28:31 DEBUG --- status code --- 06:28:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:31 DEBUG --- http response --- 06:28:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:28:31 DEBUG ----- output ----- 06:28:31 DEBUG { 06:28:31 DEBUG "status": "success", 06:28:31 DEBUG "data": { 06:28:31 DEBUG "resultType": "vector", 06:28:31 DEBUG "result": [ 06:28:31 DEBUG { 06:28:31 DEBUG "metric": { 06:28:31 DEBUG "pod": "ds-idrepo-0" 06:28:31 DEBUG }, 06:28:31 DEBUG "value": [ 06:28:31 DEBUG 06:28:31 DEBUG 06:28:31 DEBUG 06:28:31 DEBUG [...] 06:28:31 DEBUG 06:28:31 DEBUG 06:28:31 DEBUG 06:28:31 DEBUG "pod": "ds-idrepo-1" 06:28:31 DEBUG }, 06:28:31 DEBUG "value": [ 06:28:31 DEBUG 1712726911, 06:28:31 DEBUG "0" 06:28:31 DEBUG ] 06:28:31 DEBUG } 06:28:31 DEBUG ] 06:28:31 DEBUG } 06:28:31 DEBUG } 06:28:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726912" 06:28:32 INFO [http_cmd]: http status code OK 06:28:32 DEBUG --- status code --- 06:28:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:32 DEBUG --- http response --- 06:28:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:28:32 DEBUG ----- output ----- 06:28:32 DEBUG { 06:28:32 DEBUG "status": "success", 06:28:32 DEBUG "data": { 06:28:32 DEBUG "resultType": "vector", 06:28:32 DEBUG "result": [ 06:28:32 DEBUG { 06:28:32 DEBUG "metric": { 06:28:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:28:32 DEBUG }, 06:28:32 DEBUG "value": [ 06:28:32 DEBUG 06:28:32 DEBUG 06:28:32 DEBUG 06:28:32 DEBUG [...] 06:28:32 DEBUG 06:28:32 DEBUG 06:28:32 DEBUG 06:28:32 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:28:32 DEBUG }, 06:28:32 DEBUG "value": [ 06:28:32 DEBUG 1712726912, 06:28:32 DEBUG "0" 06:28:32 DEBUG ] 06:28:32 DEBUG } 06:28:32 DEBUG ] 06:28:32 DEBUG } 06:28:32 DEBUG } 06:28:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712726912" 06:28:32 INFO [http_cmd]: http status code OK 06:28:32 DEBUG --- status code --- 06:28:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:32 DEBUG --- http response --- 06:28:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:28:32 DEBUG ----- output ----- 06:28:32 DEBUG { 06:28:32 DEBUG "status": "success", 06:28:32 DEBUG "data": { 06:28:32 DEBUG "resultType": "vector", 06:28:32 DEBUG "result": [ 06:28:32 DEBUG { 06:28:32 DEBUG "metric": { 06:28:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:28:32 DEBUG }, 06:28:32 DEBUG "value": [ 06:28:32 DEBUG 06:28:32 DEBUG 06:28:32 DEBUG 06:28:32 DEBUG [...] 06:28:32 DEBUG 06:28:32 DEBUG 06:28:32 DEBUG 06:28:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:28:32 DEBUG }, 06:28:32 DEBUG "value": [ 06:28:32 DEBUG 1712726912, 06:28:32 DEBUG "2485129216" 06:28:32 DEBUG ] 06:28:32 DEBUG } 06:28:32 DEBUG ] 06:28:32 DEBUG } 06:28:32 DEBUG } 06:28:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726912" 06:28:32 INFO [http_cmd]: http status code OK 06:28:32 DEBUG --- status code --- 06:28:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:32 DEBUG --- http response --- 06:28:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726912" 06:28:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:28:32 DEBUG ----- output ----- 06:28:32 DEBUG { 06:28:32 DEBUG "status": "success", 06:28:32 DEBUG "data": { 06:28:32 DEBUG "resultType": "vector", 06:28:32 DEBUG "result": [ 06:28:32 DEBUG { 06:28:32 DEBUG "metric": { 06:28:32 DEBUG "instance": "10.142.0.25:9100" 06:28:32 DEBUG }, 06:28:32 DEBUG "value": [ 06:28:32 DEBUG 06:28:32 DEBUG 06:28:32 DEBUG 06:28:32 DEBUG [...] 06:28:32 DEBUG 06:28:32 DEBUG 06:28:32 DEBUG 06:28:32 DEBUG "instance": "10.142.0.77:9100" 06:28:32 DEBUG }, 06:28:32 DEBUG "value": [ 06:28:32 DEBUG 1712726912, 06:28:32 DEBUG "42592074.66666667" 06:28:32 DEBUG ] 06:28:32 DEBUG } 06:28:32 DEBUG ] 06:28:32 DEBUG } 06:28:32 DEBUG } 06:28:32 INFO [http_cmd]: http status code OK 06:28:32 DEBUG --- status code --- 06:28:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:28:32 DEBUG --- http response --- 06:28:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:28:32 DEBUG ----- output ----- 06:28:32 DEBUG { 06:28:32 DEBUG "status": "success", 06:28:32 DEBUG "data": { 06:28:32 DEBUG "resultType": "vector", 06:28:32 DEBUG "result": [ 06:28:32 DEBUG { 06:28:32 DEBUG "metric": { 06:28:32 DEBUG "instance": "10.142.0.25:9100" 06:28:32 DEBUG }, 06:28:32 DEBUG "value": [ 06:28:32 DEBUG 06:28:32 DEBUG 06:28:32 DEBUG 06:28:32 DEBUG [...] 06:28:32 DEBUG 06:28:32 DEBUG 06:28:32 DEBUG 06:28:32 DEBUG "instance": "10.142.0.77:9100" 06:28:32 DEBUG }, 06:28:32 DEBUG "value": [ 06:28:32 DEBUG 1712726912, 06:28:32 DEBUG "0.034666666666666866" 06:28:32 DEBUG ] 06:28:32 DEBUG } 06:28:32 DEBUG ] 06:28:32 DEBUG } 06:28:32 DEBUG } 06:28:51 INFO 06:28:51 INFO [loop_until]: kubectl --namespace=xlou top pods 06:28:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:28:51 INFO [loop_until]: OK (rc = 0) 06:28:51 DEBUG --- stdout --- 06:28:51 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10918m 8986Mi am-654c4fcbcf-9scrj 9958m 8450Mi am-654c4fcbcf-v5qb5 8403m 8457Mi ds-cts-0 8313m 12679Mi ds-cts-1 8482m 12909Mi ds-cts-2 7380m 13041Mi ds-idrepo-0 10697m 5623Mi ds-idrepo-1 352m 5672Mi ds-idrepo-2 369m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 12m 1190Mi idm-6bfb7c96cc-vsb58 5m 2213Mi lodemon-b776b7d88-thrqf 2m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4329m 2365Mi 06:28:51 DEBUG --- stderr --- 06:28:51 DEBUG 06:28:59 INFO 06:28:59 INFO [loop_until]: kubectl --namespace=xlou top node 06:28:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:28:59 INFO [loop_until]: OK (rc = 0) 06:28:59 DEBUG --- stdout --- 06:28:59 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 109m 0% 2320Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9333m 58% 9325Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 81m 0% 3252Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13977m 87% 6762Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10567m 66% 9336Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11968m 75% 9801Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 79m 0% 853Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10857m 68% 6306Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8595m 54% 13407Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 7547m 47% 13679Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 447m 2% 6392Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8564m 53% 13610Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 413m 2% 6353Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4514m 28% 6068Mi 10% 06:28:59 DEBUG --- stderr --- 06:28:59 DEBUG 06:29:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726968" 06:29:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726968" 06:29:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712726968" 06:29:28 INFO [http_cmd]: http status code OK 06:29:28 INFO [http_cmd]: http status code OK 06:29:28 DEBUG --- status code --- 06:29:28 DEBUG --- status code --- 06:29:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:28 DEBUG --- http response --- 06:29:28 DEBUG --- http response --- 06:29:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:29:28 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:29:28 DEBUG ----- output ----- 06:29:28 DEBUG { 06:29:28 DEBUG "status": "success", 06:29:28 DEBUG "data": { 06:29:28 DEBUG "resultType": "vector", 06:29:28 DEBUG "result": [ 06:29:28 DEBUG { 06:29:28 DEBUG "metric": { 06:29:28 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:29:28 DEBUG }, 06:29:28 DEBUG "value": [ 06:29:28 DEBUG 06:29:28 DEBUG 06:29:28 DEBUG 06:29:28 DEBUG [...] 06:29:28 DEBUG 06:29:28 DEBUG 06:29:28 DEBUG 06:29:28 DEBUG "pod": "am-654c4fcbcf-487cv" 06:29:28 DEBUG }, 06:29:28 DEBUG "value": [ 06:29:28 DEBUG 1712726968, 06:29:28 DEBUG "0" 06:29:28 DEBUG ] 06:29:28 DEBUG } 06:29:28 DEBUG ] 06:29:28 DEBUG } 06:29:28 DEBUG } 06:29:28 ERROR [http_cmd]: ERROR 06:29:28 ERROR --- status code --- 06:29:28 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:29:28 ERROR --- http response --- 06:29:28 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:29:28 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:29:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726970" 06:29:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726970" 06:29:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726970" 06:29:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726970" 06:29:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726970" 06:29:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726970" 06:29:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726970" 06:29:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726970" 06:29:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726970" 06:29:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726970" 06:29:30 INFO [http_cmd]: http status code OK 06:29:30 INFO [http_cmd]: http status code OK 06:29:30 INFO [http_cmd]: http status code OK 06:29:30 INFO [http_cmd]: http status code OK 06:29:30 INFO [http_cmd]: http status code OK 06:29:30 DEBUG --- status code --- 06:29:30 INFO [http_cmd]: http status code OK 06:29:30 DEBUG --- status code --- 06:29:30 DEBUG --- status code --- 06:29:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:30 DEBUG --- status code --- 06:29:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:30 DEBUG --- http response --- 06:29:30 DEBUG --- status code --- 06:29:30 INFO [http_cmd]: http status code OK 06:29:30 DEBUG --- http response --- 06:29:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:30 DEBUG --- http response --- 06:29:30 DEBUG --- status code --- 06:29:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:30 INFO [http_cmd]: http status code OK 06:29:30 DEBUG --- status code --- 06:29:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:29:30 DEBUG --- http response --- 06:29:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:29:30 DEBUG --- http response --- 06:29:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:29:30 DEBUG --- http response --- 06:29:30 DEBUG --- status code --- 06:29:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:30 DEBUG ----- output ----- 06:29:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:29:30 DEBUG ----- output ----- 06:29:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:29:30 DEBUG ----- output ----- 06:29:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:29:30 DEBUG ----- output ----- 06:29:30 DEBUG --- http response --- 06:29:30 DEBUG { 06:29:30 DEBUG ----- output ----- 06:29:30 DEBUG { 06:29:30 DEBUG { 06:29:30 DEBUG "status": "success", 06:29:30 DEBUG { 06:29:30 DEBUG "status": "success", 06:29:30 DEBUG "status": "success", 06:29:30 DEBUG "status": "success", 06:29:30 DEBUG "data": { 06:29:30 DEBUG "data": { 06:29:30 DEBUG "resultType": "vector", 06:29:30 DEBUG "data": { 06:29:30 DEBUG "resultType": "vector", 06:29:30 DEBUG "data": { 06:29:30 DEBUG "resultType": "vector", 06:29:30 DEBUG { 06:29:30 DEBUG "result": [ 06:29:30 DEBUG { 06:29:30 DEBUG "metric": { 06:29:30 DEBUG "result": [ 06:29:30 DEBUG { 06:29:30 DEBUG "result": [ 06:29:30 INFO [http_cmd]: http status code OK 06:29:30 INFO [http_cmd]: http status code OK 06:29:30 DEBUG "status": "success", 06:29:30 DEBUG "data": { 06:29:30 DEBUG "resultType": "vector", 06:29:30 DEBUG "result": [ 06:29:30 DEBUG { 06:29:30 DEBUG "resultType": "vector", 06:29:30 DEBUG { 06:29:30 DEBUG "metric": { 06:29:30 DEBUG "pod": "ds-idrepo-0" 06:29:30 DEBUG "metric": { 06:29:30 DEBUG "instance": "10.142.0.25:9100" 06:29:30 DEBUG }, 06:29:30 DEBUG "value": [ 06:29:30 DEBUG 06:29:30 DEBUG "result": [ 06:29:30 DEBUG ----- output ----- 06:29:30 DEBUG --- status code --- 06:29:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG [...] 06:29:30 DEBUG 06:29:30 DEBUG { 06:29:30 DEBUG "metric": { 06:29:30 DEBUG { 06:29:30 DEBUG --- http response --- 06:29:30 DEBUG --- status code --- 06:29:30 DEBUG 06:29:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:29:30 DEBUG "pod": "ds-idrepo-0" 06:29:30 DEBUG "metric": { 06:29:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:29:30 DEBUG "instance": "10.142.0.25:9100" 06:29:30 DEBUG "status": "success", 06:29:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:29:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:30 DEBUG 06:29:30 DEBUG ----- output ----- 06:29:30 DEBUG }, 06:29:30 DEBUG }, 06:29:30 DEBUG }, 06:29:30 DEBUG }, 06:29:30 DEBUG "data": { 06:29:30 DEBUG ----- output ----- 06:29:30 DEBUG --- http response --- 06:29:30 DEBUG "instance": "10.142.0.77:9100" 06:29:30 DEBUG { 06:29:30 DEBUG "status": "success", 06:29:30 DEBUG "data": { 06:29:30 DEBUG "resultType": "vector", 06:29:30 DEBUG "result": [ 06:29:30 DEBUG { 06:29:30 DEBUG "metric": { 06:29:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:29:30 DEBUG }, 06:29:30 DEBUG "value": [ 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG [...] 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG { 06:29:30 DEBUG "status": "success", 06:29:30 DEBUG "data": { 06:29:30 DEBUG "resultType": "vector", 06:29:30 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:29:30 DEBUG }, 06:29:30 DEBUG "value": [ 06:29:30 DEBUG 1712726970, 06:29:30 DEBUG "0" 06:29:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:30 DEBUG "value": [ 06:29:30 DEBUG 06:29:30 DEBUG "value": [ 06:29:30 DEBUG ] 06:29:30 DEBUG "result": [ 06:29:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:29:30 DEBUG ----- output ----- 06:29:30 DEBUG { 06:29:30 DEBUG "status": "success", 06:29:30 DEBUG --- http response --- 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG [...] 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG "pod": "ds-idrepo-1" 06:29:30 DEBUG }, 06:29:30 DEBUG "value": [ 06:29:30 DEBUG 1712726970, 06:29:30 DEBUG "0" 06:29:30 DEBUG ] 06:29:30 DEBUG } 06:29:30 DEBUG ] 06:29:30 DEBUG } 06:29:30 DEBUG } 06:29:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:29:30 DEBUG ----- output ----- 06:29:30 DEBUG { 06:29:30 DEBUG "status": "success", 06:29:30 DEBUG "data": { 06:29:30 DEBUG "resultType": "vector", 06:29:30 DEBUG "result": [ 06:29:30 DEBUG { 06:29:30 DEBUG "metric": { 06:29:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:29:30 DEBUG }, 06:29:30 DEBUG "value": [ 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG [...] 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:29:30 DEBUG }, 06:29:30 DEBUG "value": [ 06:29:30 DEBUG 1712726970, 06:29:30 DEBUG "258265.5" 06:29:30 DEBUG ] 06:29:30 DEBUG } 06:29:30 DEBUG ] 06:29:30 DEBUG } 06:29:30 DEBUG } 06:29:30 DEBUG { 06:29:30 DEBUG "metric": { 06:29:30 DEBUG "pod": "ds-idrepo-0" 06:29:30 DEBUG }, 06:29:30 DEBUG "value": [ 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG [...] 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG "pod": "ds-cts-2" 06:29:30 DEBUG }, 06:29:30 DEBUG "value": [ 06:29:30 DEBUG 1712726970, 06:29:30 DEBUG "29.8" 06:29:30 DEBUG ] 06:29:30 DEBUG } 06:29:30 DEBUG ] 06:29:30 DEBUG } 06:29:30 DEBUG } 06:29:30 DEBUG "data": { 06:29:30 DEBUG "resultType": "vector", 06:29:30 DEBUG "result": [ 06:29:30 DEBUG { 06:29:30 DEBUG "metric": { 06:29:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:29:30 DEBUG }, 06:29:30 DEBUG "value": [ 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG [...] 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:29:30 DEBUG }, 06:29:30 DEBUG "value": [ 06:29:30 DEBUG 1712726970, 06:29:30 DEBUG "1444" 06:29:30 DEBUG ] 06:29:30 DEBUG } 06:29:30 DEBUG ] 06:29:30 DEBUG } 06:29:30 DEBUG } 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG [...] 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG "pod": "ds-cts-2" 06:29:30 DEBUG }, 06:29:30 DEBUG "value": [ 06:29:30 DEBUG 1712726970, 06:29:30 DEBUG "0" 06:29:30 DEBUG ] 06:29:30 DEBUG } 06:29:30 DEBUG ] 06:29:30 DEBUG } 06:29:30 DEBUG } 06:29:30 DEBUG "resultType": "vector", 06:29:30 DEBUG "result": [ 06:29:30 DEBUG }, 06:29:30 DEBUG "value": [ 06:29:30 DEBUG 1712726970, 06:29:30 DEBUG "42836347.45508482" 06:29:30 DEBUG ] 06:29:30 DEBUG } 06:29:30 DEBUG ] 06:29:30 DEBUG } 06:29:30 DEBUG } 06:29:30 DEBUG "value": [ 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG [...] 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:29:30 DEBUG }, 06:29:30 DEBUG "value": [ 06:29:30 DEBUG 1712726970, 06:29:30 DEBUG "1444.0666666666666" 06:29:30 DEBUG ] 06:29:30 DEBUG } 06:29:30 DEBUG ] 06:29:30 DEBUG } 06:29:30 DEBUG } 06:29:30 DEBUG } 06:29:30 DEBUG ] 06:29:30 DEBUG } 06:29:30 DEBUG } 06:29:30 DEBUG "value": [ 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG [...] 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG "instance": "10.142.0.77:9100" 06:29:30 DEBUG }, 06:29:30 DEBUG "value": [ 06:29:30 DEBUG 1712726970, 06:29:30 DEBUG "400371827712" 06:29:30 DEBUG ] 06:29:30 DEBUG } 06:29:30 DEBUG ] 06:29:30 DEBUG } 06:29:30 DEBUG } 06:29:30 DEBUG { 06:29:30 DEBUG "metric": { 06:29:30 DEBUG "instance": "10.142.0.25:9100" 06:29:30 DEBUG }, 06:29:30 DEBUG "value": [ 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG [...] 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG "instance": "10.142.0.77:9100" 06:29:30 DEBUG }, 06:29:30 DEBUG "value": [ 06:29:30 DEBUG 1712726970, 06:29:30 DEBUG "0.15096163461217615" 06:29:30 DEBUG ] 06:29:30 DEBUG } 06:29:30 DEBUG ] 06:29:30 DEBUG } 06:29:30 DEBUG } 06:29:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726970" 06:29:30 INFO [http_cmd]: http status code OK 06:29:30 DEBUG --- status code --- 06:29:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:30 DEBUG --- http response --- 06:29:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:29:30 DEBUG ----- output ----- 06:29:30 DEBUG { 06:29:30 DEBUG "status": "success", 06:29:30 DEBUG "data": { 06:29:30 DEBUG "resultType": "vector", 06:29:30 DEBUG "result": [ 06:29:30 DEBUG { 06:29:30 DEBUG "metric": { 06:29:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:29:30 DEBUG }, 06:29:30 DEBUG "value": [ 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG [...] 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:29:30 DEBUG }, 06:29:30 DEBUG "value": [ 06:29:30 DEBUG 1712726970, 06:29:30 DEBUG "0" 06:29:30 DEBUG ] 06:29:30 DEBUG } 06:29:30 DEBUG ] 06:29:30 DEBUG } 06:29:30 DEBUG } 06:29:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712726970" 06:29:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712726970" 06:29:30 INFO [http_cmd]: http status code OK 06:29:30 DEBUG --- status code --- 06:29:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:30 DEBUG --- http response --- 06:29:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:29:30 DEBUG ----- output ----- 06:29:30 DEBUG { 06:29:30 DEBUG "status": "success", 06:29:30 DEBUG "data": { 06:29:30 DEBUG "resultType": "vector", 06:29:30 DEBUG "result": [ 06:29:30 DEBUG { 06:29:30 DEBUG "metric": { 06:29:30 DEBUG "pod": "ds-idrepo-0" 06:29:30 DEBUG }, 06:29:30 DEBUG "value": [ 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG [...] 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG "pod": "ds-cts-2" 06:29:30 DEBUG }, 06:29:30 DEBUG "value": [ 06:29:30 INFO [http_cmd]: http status code OK 06:29:30 DEBUG 1712726970, 06:29:30 DEBUG --- status code --- 06:29:30 DEBUG "0" 06:29:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:30 DEBUG ] 06:29:30 DEBUG --- http response --- 06:29:30 DEBUG } 06:29:30 DEBUG ] 06:29:30 DEBUG } 06:29:30 DEBUG } 06:29:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:29:30 DEBUG ----- output ----- 06:29:30 DEBUG { 06:29:30 DEBUG "status": "success", 06:29:30 DEBUG "data": { 06:29:30 DEBUG "resultType": "vector", 06:29:30 DEBUG "result": [ 06:29:30 DEBUG { 06:29:30 DEBUG "metric": { 06:29:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:29:30 DEBUG }, 06:29:30 DEBUG "value": [ 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG [...] 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG 06:29:30 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:29:30 DEBUG }, 06:29:30 DEBUG "value": [ 06:29:30 DEBUG 1712726970, 06:29:30 DEBUG "4.41077939300221" 06:29:30 DEBUG ] 06:29:30 DEBUG } 06:29:30 DEBUG ] 06:29:30 DEBUG } 06:29:30 DEBUG } 06:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712726971" 06:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726971" 06:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726971" 06:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726971" 06:29:31 INFO [http_cmd]: http status code OK 06:29:31 DEBUG --- status code --- 06:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:31 DEBUG --- http response --- 06:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:29:31 DEBUG ----- output ----- 06:29:31 DEBUG { 06:29:31 DEBUG "status": "success", 06:29:31 DEBUG "data": { 06:29:31 DEBUG "resultType": "vector", 06:29:31 DEBUG "result": [ 06:29:31 DEBUG { 06:29:31 DEBUG "metric": { 06:29:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:29:31 DEBUG }, 06:29:31 DEBUG "value": [ 06:29:31 DEBUG 06:29:31 DEBUG 06:29:31 DEBUG 06:29:31 DEBUG [...] 06:29:31 DEBUG 06:29:31 DEBUG 06:29:31 DEBUG 06:29:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:29:31 DEBUG }, 06:29:31 DEBUG "value": [ 06:29:31 DEBUG 1712726971, 06:29:31 DEBUG "0.051" 06:29:31 DEBUG ] 06:29:31 DEBUG } 06:29:31 DEBUG ] 06:29:31 DEBUG } 06:29:31 DEBUG } 06:29:31 INFO [http_cmd]: http status code OK 06:29:31 DEBUG --- status code --- 06:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:31 DEBUG --- http response --- 06:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:29:31 DEBUG ----- output ----- 06:29:31 DEBUG { 06:29:31 DEBUG "status": "success", 06:29:31 DEBUG "data": { 06:29:31 DEBUG "resultType": "vector", 06:29:31 DEBUG "result": [ 06:29:31 DEBUG { 06:29:31 DEBUG "metric": { 06:29:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:29:31 DEBUG }, 06:29:31 DEBUG "value": [ 06:29:31 DEBUG 06:29:31 DEBUG 06:29:31 DEBUG 06:29:31 INFO [http_cmd]: http status code OK 06:29:31 DEBUG [...] 06:29:31 DEBUG --- status code --- 06:29:31 DEBUG 06:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:31 DEBUG 06:29:31 DEBUG --- http response --- 06:29:31 DEBUG 06:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:29:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:29:31 DEBUG ----- output ----- 06:29:31 DEBUG }, 06:29:31 DEBUG { 06:29:31 DEBUG "value": [ 06:29:31 DEBUG "status": "success", 06:29:31 DEBUG 1712726971, 06:29:31 INFO [http_cmd]: http status code OK 06:29:31 DEBUG "data": { 06:29:31 DEBUG "1444.2" 06:29:31 DEBUG --- status code --- 06:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:31 DEBUG ] 06:29:31 DEBUG --- http response --- 06:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:29:31 DEBUG } 06:29:31 DEBUG ----- output ----- 06:29:31 DEBUG { 06:29:31 DEBUG ] 06:29:31 DEBUG "status": "success", 06:29:31 DEBUG "data": { 06:29:31 DEBUG } 06:29:31 DEBUG "resultType": "vector", 06:29:31 DEBUG "result": [ 06:29:31 DEBUG } 06:29:31 DEBUG { 06:29:31 DEBUG "metric": { 06:29:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:29:31 DEBUG }, 06:29:31 DEBUG "value": [ 06:29:31 DEBUG 06:29:31 DEBUG 06:29:31 DEBUG 06:29:31 DEBUG [...] 06:29:31 DEBUG 06:29:31 DEBUG 06:29:31 DEBUG 06:29:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:29:31 DEBUG }, 06:29:31 DEBUG "value": [ 06:29:31 DEBUG 1712726971, 06:29:31 DEBUG "8.432255115376776" 06:29:31 DEBUG ] 06:29:31 DEBUG } 06:29:31 DEBUG ] 06:29:31 DEBUG } 06:29:31 DEBUG "resultType": "vector", 06:29:31 DEBUG } 06:29:31 DEBUG "result": [ 06:29:31 DEBUG { 06:29:31 DEBUG "metric": { 06:29:31 DEBUG "pod": "ds-idrepo-2" 06:29:31 DEBUG }, 06:29:31 DEBUG "value": [ 06:29:31 DEBUG 06:29:31 DEBUG 06:29:31 DEBUG 06:29:31 DEBUG [...] 06:29:31 DEBUG 06:29:31 DEBUG 06:29:31 DEBUG 06:29:31 DEBUG "pod": "ds-cts-2" 06:29:31 DEBUG }, 06:29:31 DEBUG "value": [ 06:29:31 DEBUG 1712726971, 06:29:31 DEBUG "0" 06:29:31 DEBUG ] 06:29:31 DEBUG } 06:29:31 DEBUG ] 06:29:31 DEBUG } 06:29:31 DEBUG } 06:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726971" 06:29:31 INFO [http_cmd]: http status code OK 06:29:31 DEBUG --- status code --- 06:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:31 DEBUG --- http response --- 06:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:29:31 DEBUG ----- output ----- 06:29:31 DEBUG { 06:29:31 DEBUG "status": "success", 06:29:31 DEBUG "data": { 06:29:31 DEBUG "resultType": "vector", 06:29:31 DEBUG "result": [ 06:29:31 DEBUG { 06:29:31 DEBUG "metric": { 06:29:31 DEBUG "pod": "ds-idrepo-0" 06:29:31 DEBUG }, 06:29:31 DEBUG "value": [ 06:29:31 DEBUG 06:29:31 DEBUG 06:29:31 DEBUG 06:29:31 DEBUG [...] 06:29:31 DEBUG 06:29:31 DEBUG 06:29:31 DEBUG 06:29:31 DEBUG "pod": "ds-idrepo-1" 06:29:31 DEBUG }, 06:29:31 DEBUG "value": [ 06:29:31 DEBUG 1712726971, 06:29:31 DEBUG "0" 06:29:31 DEBUG ] 06:29:31 DEBUG } 06:29:31 DEBUG ] 06:29:31 DEBUG } 06:29:31 DEBUG } 06:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726971" 06:29:31 INFO [http_cmd]: http status code OK 06:29:31 DEBUG --- status code --- 06:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:31 DEBUG --- http response --- 06:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:29:31 DEBUG ----- output ----- 06:29:31 DEBUG { 06:29:31 DEBUG "status": "success", 06:29:31 DEBUG "data": { 06:29:31 DEBUG "resultType": "vector", 06:29:31 DEBUG "result": [ 06:29:31 DEBUG { 06:29:31 DEBUG "metric": { 06:29:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:29:31 DEBUG }, 06:29:31 DEBUG "value": [ 06:29:31 DEBUG 06:29:31 DEBUG 06:29:31 DEBUG 06:29:31 DEBUG [...] 06:29:31 DEBUG 06:29:31 DEBUG 06:29:31 DEBUG 06:29:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:29:31 DEBUG }, 06:29:31 DEBUG "value": [ 06:29:31 DEBUG 1712726971, 06:29:31 DEBUG "1439.7333333333333" 06:29:31 DEBUG ] 06:29:31 DEBUG } 06:29:31 DEBUG ] 06:29:31 DEBUG } 06:29:31 DEBUG } 06:29:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726971" 06:29:31 INFO [http_cmd]: http status code OK 06:29:31 DEBUG --- status code --- 06:29:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:31 DEBUG --- http response --- 06:29:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:29:31 DEBUG ----- output ----- 06:29:31 DEBUG { 06:29:31 DEBUG "status": "success", 06:29:31 DEBUG "data": { 06:29:31 DEBUG "resultType": "vector", 06:29:31 DEBUG "result": [ 06:29:31 DEBUG { 06:29:31 DEBUG "metric": { 06:29:31 DEBUG "pod": "ds-idrepo-0" 06:29:31 DEBUG }, 06:29:31 DEBUG "value": [ 06:29:31 DEBUG 06:29:31 DEBUG 06:29:31 DEBUG 06:29:31 DEBUG [...] 06:29:31 DEBUG 06:29:31 DEBUG 06:29:31 DEBUG 06:29:31 DEBUG "pod": "ds-idrepo-1" 06:29:31 DEBUG }, 06:29:31 DEBUG "value": [ 06:29:31 DEBUG 1712726971, 06:29:31 DEBUG "0" 06:29:31 DEBUG ] 06:29:31 DEBUG } 06:29:31 DEBUG ] 06:29:31 DEBUG } 06:29:31 DEBUG } 06:29:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712726972" 06:29:32 INFO [http_cmd]: http status code OK 06:29:32 DEBUG --- status code --- 06:29:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:32 DEBUG --- http response --- 06:29:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:29:32 DEBUG ----- output ----- 06:29:32 DEBUG { 06:29:32 DEBUG "status": "success", 06:29:32 DEBUG "data": { 06:29:32 DEBUG "resultType": "vector", 06:29:32 DEBUG "result": [ 06:29:32 DEBUG { 06:29:32 DEBUG "metric": { 06:29:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:29:32 DEBUG }, 06:29:32 DEBUG "value": [ 06:29:32 DEBUG 06:29:32 DEBUG 06:29:32 DEBUG 06:29:32 DEBUG [...] 06:29:32 DEBUG 06:29:32 DEBUG 06:29:32 DEBUG 06:29:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:29:32 DEBUG }, 06:29:32 DEBUG "value": [ 06:29:32 DEBUG 1712726972, 06:29:32 DEBUG "0" 06:29:32 DEBUG ] 06:29:32 DEBUG } 06:29:32 DEBUG ] 06:29:32 DEBUG } 06:29:32 DEBUG } 06:29:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712726972" 06:29:32 INFO [http_cmd]: http status code OK 06:29:32 DEBUG --- status code --- 06:29:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:32 DEBUG --- http response --- 06:29:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:29:32 DEBUG ----- output ----- 06:29:32 DEBUG { 06:29:32 DEBUG "status": "success", 06:29:32 DEBUG "data": { 06:29:32 DEBUG "resultType": "vector", 06:29:32 DEBUG "result": [ 06:29:32 DEBUG { 06:29:32 DEBUG "metric": { 06:29:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:29:32 DEBUG }, 06:29:32 DEBUG "value": [ 06:29:32 DEBUG 06:29:32 DEBUG 06:29:32 DEBUG 06:29:32 DEBUG [...] 06:29:32 DEBUG 06:29:32 DEBUG 06:29:32 DEBUG 06:29:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:29:32 DEBUG }, 06:29:32 DEBUG "value": [ 06:29:32 DEBUG 1712726972, 06:29:32 DEBUG "2487025664" 06:29:32 DEBUG ] 06:29:32 DEBUG } 06:29:32 DEBUG ] 06:29:32 DEBUG } 06:29:32 DEBUG } 06:29:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726972" 06:29:32 INFO [http_cmd]: http status code OK 06:29:32 DEBUG --- status code --- 06:29:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:32 DEBUG --- http response --- 06:29:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:29:32 DEBUG ----- output ----- 06:29:32 DEBUG { 06:29:32 DEBUG "status": "success", 06:29:32 DEBUG "data": { 06:29:32 DEBUG "resultType": "vector", 06:29:32 DEBUG "result": [ 06:29:32 DEBUG { 06:29:32 DEBUG "metric": { 06:29:32 DEBUG "instance": "10.142.0.25:9100" 06:29:32 DEBUG }, 06:29:32 DEBUG "value": [ 06:29:32 DEBUG 06:29:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712726972" 06:29:32 DEBUG 06:29:32 DEBUG 06:29:32 DEBUG [...] 06:29:32 DEBUG 06:29:32 DEBUG 06:29:32 DEBUG 06:29:32 DEBUG "instance": "10.142.0.77:9100" 06:29:32 DEBUG }, 06:29:32 DEBUG "value": [ 06:29:32 DEBUG 1712726972, 06:29:32 DEBUG "42837818.406053126" 06:29:32 DEBUG ] 06:29:32 DEBUG } 06:29:32 DEBUG ] 06:29:32 DEBUG } 06:29:32 DEBUG } 06:29:32 INFO [http_cmd]: http status code OK 06:29:32 DEBUG --- status code --- 06:29:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:29:32 DEBUG --- http response --- 06:29:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:29:32 DEBUG ----- output ----- 06:29:32 DEBUG { 06:29:32 DEBUG "status": "success", 06:29:32 DEBUG "data": { 06:29:32 DEBUG "resultType": "vector", 06:29:32 DEBUG "result": [ 06:29:32 DEBUG { 06:29:32 DEBUG "metric": { 06:29:32 DEBUG "instance": "10.142.0.25:9100" 06:29:32 DEBUG }, 06:29:32 DEBUG "value": [ 06:29:32 DEBUG 06:29:32 DEBUG 06:29:32 DEBUG 06:29:32 DEBUG [...] 06:29:32 DEBUG 06:29:32 DEBUG 06:29:32 DEBUG 06:29:32 DEBUG "instance": "10.142.0.77:9100" 06:29:32 DEBUG }, 06:29:32 DEBUG "value": [ 06:29:32 DEBUG 1712726972, 06:29:32 DEBUG "0.052664911169628216" 06:29:32 DEBUG ] 06:29:32 DEBUG } 06:29:32 DEBUG ] 06:29:32 DEBUG } 06:29:32 DEBUG } 06:29:51 INFO 06:29:51 INFO [loop_until]: kubectl --namespace=xlou top pods 06:29:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:29:51 INFO [loop_until]: OK (rc = 0) 06:29:51 DEBUG --- stdout --- 06:29:51 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10685m 8985Mi am-654c4fcbcf-9scrj 10032m 8459Mi am-654c4fcbcf-v5qb5 8619m 8472Mi ds-cts-0 8149m 13272Mi ds-cts-1 8286m 13393Mi ds-cts-2 7584m 12609Mi ds-idrepo-0 10838m 5623Mi ds-idrepo-1 370m 5672Mi ds-idrepo-2 429m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 6m 1190Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 5m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4305m 2464Mi 06:29:51 DEBUG --- stderr --- 06:29:51 DEBUG 06:29:59 INFO 06:29:59 INFO [loop_until]: kubectl --namespace=xlou top node 06:29:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:29:59 INFO [loop_until]: OK (rc = 0) 06:29:59 DEBUG --- stdout --- 06:29:59 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 98m 0% 2310Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9329m 58% 9323Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 82m 0% 3262Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 14100m 88% 6765Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 11001m 69% 9338Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11948m 75% 9805Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 77m 0% 857Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10873m 68% 6306Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8325m 52% 14025Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 7733m 48% 13247Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 481m 3% 6389Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8535m 53% 13203Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 414m 2% 6353Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4353m 27% 6047Mi 10% 06:29:59 DEBUG --- stderr --- 06:29:59 DEBUG 06:30:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727029" 06:30:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712727029" 06:30:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727029" 06:30:29 ERROR [http_cmd]: ERROR 06:30:29 ERROR --- status code --- 06:30:29 INFO [http_cmd]: http status code OK 06:30:29 INFO [http_cmd]: http status code OK 06:30:29 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:30:29 DEBUG --- status code --- 06:30:29 DEBUG --- status code --- 06:30:29 ERROR --- http response --- 06:30:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:29 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:30:29 DEBUG --- http response --- 06:30:29 DEBUG --- http response --- 06:30:29 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:30:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:30:29 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:30:29 DEBUG ----- output ----- 06:30:29 DEBUG { 06:30:29 DEBUG "status": "success", 06:30:29 DEBUG "data": { 06:30:29 DEBUG "resultType": "vector", 06:30:29 DEBUG "result": [ 06:30:29 DEBUG { 06:30:29 DEBUG "metric": { 06:30:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:30:29 DEBUG }, 06:30:29 DEBUG "value": [ 06:30:29 DEBUG 06:30:29 DEBUG 06:30:29 DEBUG 06:30:29 DEBUG [...] 06:30:29 DEBUG 06:30:29 DEBUG 06:30:29 DEBUG 06:30:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:30:29 DEBUG }, 06:30:29 DEBUG "value": [ 06:30:29 DEBUG 1712727029, 06:30:29 DEBUG "0" 06:30:29 DEBUG ] 06:30:29 DEBUG } 06:30:29 DEBUG ] 06:30:29 DEBUG } 06:30:29 DEBUG } 06:30:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727030" 06:30:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727030" 06:30:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727030" 06:30:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727030" 06:30:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727030" 06:30:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727030" 06:30:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727030" 06:30:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727030" 06:30:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727030" 06:30:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727030" 06:30:30 INFO [http_cmd]: http status code OK 06:30:30 DEBUG --- status code --- 06:30:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:30 DEBUG --- http response --- 06:30:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:30:30 DEBUG ----- output ----- 06:30:30 DEBUG { 06:30:30 DEBUG "status": "success", 06:30:30 DEBUG "data": { 06:30:30 DEBUG "resultType": "vector", 06:30:30 DEBUG "result": [ 06:30:30 DEBUG { 06:30:30 DEBUG "metric": { 06:30:30 DEBUG "pod": "ds-idrepo-0" 06:30:30 DEBUG }, 06:30:30 DEBUG "value": [ 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG [...] 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG "pod": "ds-idrepo-1" 06:30:30 DEBUG }, 06:30:30 DEBUG "value": [ 06:30:30 DEBUG 1712727030, 06:30:30 DEBUG "0" 06:30:30 INFO [http_cmd]: http status code OK 06:30:30 DEBUG ] 06:30:30 INFO [http_cmd]: http status code OK 06:30:30 DEBUG --- status code --- 06:30:30 INFO [http_cmd]: http status code OK 06:30:30 DEBUG } 06:30:30 INFO [http_cmd]: http status code OK 06:30:30 INFO [http_cmd]: http status code OK 06:30:30 DEBUG --- status code --- 06:30:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:30 DEBUG --- status code --- 06:30:30 INFO [http_cmd]: http status code OK 06:30:30 DEBUG ] 06:30:30 DEBUG } 06:30:30 DEBUG } 06:30:30 DEBUG --- status code --- 06:30:30 DEBUG --- status code --- 06:30:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:30 DEBUG --- http response --- 06:30:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:30:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:30 DEBUG --- status code --- 06:30:30 INFO [http_cmd]: http status code OK 06:30:30 INFO [http_cmd]: http status code OK 06:30:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:30 DEBUG --- http response --- 06:30:30 INFO [http_cmd]: http status code OK 06:30:30 DEBUG --- status code --- 06:30:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:30 DEBUG --- http response --- 06:30:30 DEBUG --- http response --- 06:30:30 DEBUG --- status code --- 06:30:30 DEBUG --- status code --- 06:30:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:30 DEBUG --- http response --- 06:30:30 DEBUG ----- output ----- 06:30:30 DEBUG { 06:30:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:30:30 DEBUG ----- output ----- 06:30:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:30:30 DEBUG ----- output ----- 06:30:30 DEBUG "status": "success", 06:30:30 DEBUG "data": { 06:30:30 DEBUG --- http response --- 06:30:30 DEBUG { 06:30:30 DEBUG "status": "success", 06:30:30 DEBUG --- http response --- 06:30:30 DEBUG --- http response --- 06:30:30 DEBUG { 06:30:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:30:30 DEBUG "resultType": "vector", 06:30:30 DEBUG "result": [ 06:30:30 DEBUG { 06:30:30 DEBUG "metric": { 06:30:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:30:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:30:30 DEBUG ----- output ----- 06:30:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:30:30 DEBUG ----- output ----- 06:30:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:30:30 DEBUG }, 06:30:30 DEBUG "value": [ 06:30:30 DEBUG 06:30:30 DEBUG ----- output ----- 06:30:30 DEBUG { 06:30:30 DEBUG "status": "success", 06:30:30 DEBUG "data": { 06:30:30 DEBUG "resultType": "vector", 06:30:30 DEBUG "result": [ 06:30:30 DEBUG { 06:30:30 DEBUG "metric": { 06:30:30 DEBUG "pod": "ds-idrepo-0" 06:30:30 DEBUG { 06:30:30 DEBUG "status": "success", 06:30:30 DEBUG }, 06:30:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:30:30 DEBUG --- http response --- 06:30:30 DEBUG ----- output ----- 06:30:30 DEBUG "data": { 06:30:30 DEBUG 06:30:30 DEBUG "status": "success", 06:30:30 DEBUG "data": { 06:30:30 DEBUG { 06:30:30 DEBUG "value": [ 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG { 06:30:30 DEBUG "resultType": "vector", 06:30:30 DEBUG 06:30:30 DEBUG "data": { 06:30:30 DEBUG "resultType": "vector", 06:30:30 DEBUG "status": "success", 06:30:30 DEBUG 06:30:30 DEBUG [...] 06:30:30 DEBUG 06:30:30 DEBUG "status": "success", 06:30:30 DEBUG "result": [ 06:30:30 DEBUG [...] 06:30:30 DEBUG "resultType": "vector", 06:30:30 DEBUG "result": [ 06:30:30 DEBUG "data": { 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG "pod": "ds-cts-2" 06:30:30 DEBUG "data": { 06:30:30 DEBUG { 06:30:30 DEBUG 06:30:30 DEBUG "result": [ 06:30:30 DEBUG { 06:30:30 DEBUG "resultType": "vector", 06:30:30 DEBUG }, 06:30:30 DEBUG ----- output ----- 06:30:30 DEBUG { 06:30:30 DEBUG "status": "success", 06:30:30 DEBUG "metric": { 06:30:30 DEBUG 06:30:30 DEBUG { 06:30:30 DEBUG "metric": { 06:30:30 DEBUG "result": [ 06:30:30 DEBUG "value": [ 06:30:30 DEBUG 1712727030, 06:30:30 DEBUG "resultType": "vector", 06:30:30 DEBUG "data": { 06:30:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:30:30 DEBUG 06:30:30 DEBUG "metric": { 06:30:30 DEBUG "instance": "10.142.0.25:9100" 06:30:30 DEBUG { 06:30:30 DEBUG "0" 06:30:30 DEBUG ] 06:30:30 DEBUG "result": [ 06:30:30 DEBUG "resultType": "vector", 06:30:30 DEBUG }, 06:30:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:30:30 DEBUG "pod": "ds-idrepo-0" 06:30:30 DEBUG }, 06:30:30 DEBUG "metric": { 06:30:30 DEBUG } 06:30:30 DEBUG ] 06:30:30 DEBUG { 06:30:30 DEBUG "metric": { 06:30:30 DEBUG "value": [ 06:30:30 DEBUG }, 06:30:30 DEBUG }, 06:30:30 DEBUG "value": [ 06:30:30 DEBUG "instance": "10.142.0.25:9100" 06:30:30 DEBUG } 06:30:30 DEBUG } 06:30:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:30:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:30:30 DEBUG 06:30:30 DEBUG "value": [ 06:30:30 DEBUG "value": [ 06:30:30 DEBUG 06:30:30 DEBUG }, 06:30:30 DEBUG }, 06:30:30 DEBUG "value": [ 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG "value": [ 06:30:30 DEBUG 1712727030, 06:30:30 DEBUG "1435.1" 06:30:30 DEBUG ] 06:30:30 DEBUG ----- output ----- 06:30:30 DEBUG { 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG } 06:30:30 DEBUG ] 06:30:30 DEBUG } 06:30:30 DEBUG "status": "success", 06:30:30 DEBUG "data": { 06:30:30 DEBUG [...] 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG } 06:30:30 DEBUG "result": [ 06:30:30 DEBUG { 06:30:30 DEBUG "resultType": "vector", 06:30:30 DEBUG "result": [ 06:30:30 DEBUG 06:30:30 DEBUG [...] 06:30:30 DEBUG 06:30:30 DEBUG "metric": { 06:30:30 DEBUG [...] 06:30:30 DEBUG { 06:30:30 DEBUG "metric": { 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG [...] 06:30:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:30:30 DEBUG 06:30:30 DEBUG "instance": "10.142.0.25:9100" 06:30:30 DEBUG }, 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG }, 06:30:30 DEBUG 06:30:30 DEBUG "value": [ 06:30:30 DEBUG 06:30:30 DEBUG "pod": "ds-cts-2" 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG "value": [ 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG }, 06:30:30 DEBUG "instance": "10.142.0.77:9100" 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:30:30 DEBUG }, 06:30:30 DEBUG 06:30:30 DEBUG "value": [ 06:30:30 DEBUG }, 06:30:30 DEBUG "instance": "10.142.0.77:9100" 06:30:30 DEBUG 06:30:30 DEBUG "value": [ 06:30:30 DEBUG 1712727030, 06:30:30 DEBUG 06:30:30 DEBUG 1712727030, 06:30:30 DEBUG "value": [ 06:30:30 DEBUG }, 06:30:30 DEBUG "value": [ 06:30:30 DEBUG "1434.8666666666666" 06:30:30 DEBUG [...] 06:30:30 DEBUG 06:30:30 DEBUG "29.89501749708382" 06:30:30 DEBUG 1712727030, 06:30:30 DEBUG 1712727030, 06:30:30 DEBUG "402477308928" 06:30:30 DEBUG ] 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG ] 06:30:30 DEBUG "0.10549999999999796" 06:30:30 DEBUG ] 06:30:30 DEBUG } 06:30:30 DEBUG } 06:30:30 DEBUG ] 06:30:30 DEBUG } 06:30:30 DEBUG } 06:30:30 DEBUG ] 06:30:30 DEBUG ] 06:30:30 DEBUG } 06:30:30 DEBUG } 06:30:30 DEBUG [...] 06:30:30 DEBUG 06:30:30 DEBUG ] 06:30:30 DEBUG } 06:30:30 DEBUG } 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG } 06:30:30 DEBUG ] 06:30:30 DEBUG [...] 06:30:30 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:30:30 DEBUG "instance": "10.142.0.77:9100" 06:30:30 DEBUG } 06:30:30 DEBUG } 06:30:30 DEBUG 06:30:30 DEBUG }, 06:30:30 DEBUG }, 06:30:30 DEBUG } 06:30:30 DEBUG 06:30:30 DEBUG "value": [ 06:30:30 DEBUG 1712727030, 06:30:30 DEBUG "0" 06:30:30 DEBUG ] 06:30:30 DEBUG } 06:30:30 DEBUG ] 06:30:30 DEBUG } 06:30:30 DEBUG } 06:30:30 DEBUG "value": [ 06:30:30 DEBUG 06:30:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:30:30 DEBUG }, 06:30:30 DEBUG "value": [ 06:30:30 DEBUG 1712727030, 06:30:30 DEBUG "258459.5" 06:30:30 DEBUG ] 06:30:30 DEBUG } 06:30:30 DEBUG ] 06:30:30 DEBUG } 06:30:30 DEBUG } 06:30:30 DEBUG 1712727030, 06:30:30 DEBUG "42480895.666666664" 06:30:30 DEBUG ] 06:30:30 DEBUG } 06:30:30 DEBUG ] 06:30:30 DEBUG } 06:30:30 DEBUG } 06:30:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727030" 06:30:30 INFO [http_cmd]: http status code OK 06:30:30 DEBUG --- status code --- 06:30:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:30 DEBUG --- http response --- 06:30:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:30:30 DEBUG ----- output ----- 06:30:30 DEBUG { 06:30:30 DEBUG "status": "success", 06:30:30 DEBUG "data": { 06:30:30 DEBUG "resultType": "vector", 06:30:30 DEBUG "result": [ 06:30:30 DEBUG { 06:30:30 DEBUG "metric": { 06:30:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:30:30 DEBUG }, 06:30:30 DEBUG "value": [ 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG [...] 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:30:30 DEBUG }, 06:30:30 DEBUG "value": [ 06:30:30 DEBUG 1712727030, 06:30:30 DEBUG "12412.37255240007" 06:30:30 DEBUG ] 06:30:30 DEBUG } 06:30:30 DEBUG ] 06:30:30 DEBUG } 06:30:30 DEBUG } 06:30:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712727030" 06:30:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712727030" 06:30:30 INFO [http_cmd]: http status code OK 06:30:30 DEBUG --- status code --- 06:30:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:30 DEBUG --- http response --- 06:30:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:30:30 DEBUG ----- output ----- 06:30:30 DEBUG { 06:30:30 DEBUG "status": "success", 06:30:30 DEBUG "data": { 06:30:30 DEBUG "resultType": "vector", 06:30:30 DEBUG "result": [ 06:30:30 DEBUG { 06:30:30 DEBUG "metric": { 06:30:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:30:30 DEBUG }, 06:30:30 DEBUG "value": [ 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG [...] 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:30:30 DEBUG }, 06:30:30 DEBUG "value": [ 06:30:30 DEBUG 1712727030, 06:30:30 DEBUG "4.395539116709531" 06:30:30 DEBUG ] 06:30:30 DEBUG } 06:30:30 DEBUG ] 06:30:30 DEBUG } 06:30:30 DEBUG } 06:30:30 INFO [http_cmd]: http status code OK 06:30:30 DEBUG --- status code --- 06:30:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:30 DEBUG --- http response --- 06:30:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:30:30 DEBUG ----- output ----- 06:30:30 DEBUG { 06:30:30 DEBUG "status": "success", 06:30:30 DEBUG "data": { 06:30:30 DEBUG "resultType": "vector", 06:30:30 DEBUG "result": [ 06:30:30 DEBUG { 06:30:30 DEBUG "metric": { 06:30:30 DEBUG "pod": "ds-idrepo-0" 06:30:30 DEBUG }, 06:30:30 DEBUG "value": [ 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG [...] 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG 06:30:30 DEBUG "pod": "ds-cts-2" 06:30:30 DEBUG }, 06:30:30 DEBUG "value": [ 06:30:30 DEBUG 1712727030, 06:30:30 DEBUG "0" 06:30:30 DEBUG ] 06:30:30 DEBUG } 06:30:30 DEBUG ] 06:30:30 DEBUG } 06:30:30 DEBUG } 06:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712727031" 06:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727031" 06:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727031" 06:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727031" 06:30:31 INFO [http_cmd]: http status code OK 06:30:31 DEBUG --- status code --- 06:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:31 DEBUG --- http response --- 06:30:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:30:31 DEBUG ----- output ----- 06:30:31 DEBUG { 06:30:31 DEBUG "status": "success", 06:30:31 DEBUG "data": { 06:30:31 DEBUG "resultType": "vector", 06:30:31 DEBUG "result": [ 06:30:31 DEBUG { 06:30:31 DEBUG "metric": { 06:30:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:30:31 DEBUG }, 06:30:31 DEBUG "value": [ 06:30:31 DEBUG 06:30:31 DEBUG 06:30:31 DEBUG 06:30:31 DEBUG [...] 06:30:31 DEBUG 06:30:31 DEBUG 06:30:31 DEBUG 06:30:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:30:31 DEBUG }, 06:30:31 DEBUG "value": [ 06:30:31 DEBUG 1712727031, 06:30:31 DEBUG "0.051" 06:30:31 DEBUG ] 06:30:31 DEBUG } 06:30:31 DEBUG ] 06:30:31 DEBUG } 06:30:31 DEBUG } 06:30:31 INFO [http_cmd]: http status code OK 06:30:31 DEBUG --- status code --- 06:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:31 INFO [http_cmd]: http status code OK 06:30:31 DEBUG --- http response --- 06:30:31 DEBUG --- status code --- 06:30:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:30:31 DEBUG ----- output ----- 06:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:31 DEBUG { 06:30:31 DEBUG "status": "success", 06:30:31 DEBUG "data": { 06:30:31 DEBUG "resultType": "vector", 06:30:31 DEBUG --- http response --- 06:30:31 DEBUG "result": [ 06:30:31 DEBUG { 06:30:31 DEBUG "metric": { 06:30:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:30:31 DEBUG }, 06:30:31 DEBUG "value": [ 06:30:31 DEBUG 06:30:31 DEBUG 06:30:31 DEBUG 06:30:31 DEBUG [...] 06:30:31 DEBUG 06:30:31 DEBUG 06:30:31 DEBUG 06:30:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:30:31 DEBUG ----- output ----- 06:30:31 INFO [http_cmd]: http status code OK 06:30:31 DEBUG --- status code --- 06:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:31 DEBUG --- http response --- 06:30:31 DEBUG { 06:30:31 DEBUG "status": "success", 06:30:31 DEBUG "data": { 06:30:31 DEBUG "resultType": "vector", 06:30:31 DEBUG "result": [ 06:30:31 DEBUG { 06:30:31 DEBUG "metric": { 06:30:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:30:31 DEBUG }, 06:30:31 DEBUG "value": [ 06:30:31 DEBUG 06:30:31 DEBUG 06:30:31 DEBUG 06:30:31 DEBUG [...] 06:30:31 DEBUG 06:30:31 DEBUG 06:30:31 DEBUG 06:30:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:30:31 DEBUG }, 06:30:31 DEBUG "value": [ 06:30:31 DEBUG 1712727031, 06:30:31 DEBUG "1435.1333333333332" 06:30:31 DEBUG ] 06:30:31 DEBUG } 06:30:31 DEBUG ] 06:30:31 DEBUG } 06:30:31 DEBUG } 06:30:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:30:31 DEBUG ----- output ----- 06:30:31 DEBUG { 06:30:31 DEBUG "status": "success", 06:30:31 DEBUG "data": { 06:30:31 DEBUG "resultType": "vector", 06:30:31 DEBUG "result": [ 06:30:31 DEBUG { 06:30:31 DEBUG "metric": { 06:30:31 DEBUG "pod": "ds-idrepo-2" 06:30:31 DEBUG }, 06:30:31 DEBUG "value": [ 06:30:31 DEBUG 06:30:31 DEBUG 06:30:31 DEBUG 06:30:31 DEBUG [...] 06:30:31 DEBUG 06:30:31 DEBUG 06:30:31 DEBUG 06:30:31 DEBUG "pod": "ds-cts-2" 06:30:31 DEBUG }, 06:30:31 DEBUG "value": [ 06:30:31 DEBUG 1712727031, 06:30:31 DEBUG "0.00003332777870354941" 06:30:31 DEBUG ] 06:30:31 DEBUG } 06:30:31 DEBUG ] 06:30:31 DEBUG } 06:30:31 DEBUG } 06:30:31 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:30:31 DEBUG }, 06:30:31 DEBUG "value": [ 06:30:31 DEBUG 1712727031, 06:30:31 DEBUG "0.009639647801243028" 06:30:31 DEBUG ] 06:30:31 DEBUG } 06:30:31 DEBUG ] 06:30:31 DEBUG } 06:30:31 DEBUG } 06:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727031" 06:30:31 INFO [http_cmd]: http status code OK 06:30:31 DEBUG --- status code --- 06:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:31 DEBUG --- http response --- 06:30:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:30:31 DEBUG ----- output ----- 06:30:31 DEBUG { 06:30:31 DEBUG "status": "success", 06:30:31 DEBUG "data": { 06:30:31 DEBUG "resultType": "vector", 06:30:31 DEBUG "result": [ 06:30:31 DEBUG { 06:30:31 DEBUG "metric": { 06:30:31 DEBUG "pod": "ds-idrepo-0" 06:30:31 DEBUG }, 06:30:31 DEBUG "value": [ 06:30:31 DEBUG 06:30:31 DEBUG 06:30:31 DEBUG 06:30:31 DEBUG [...] 06:30:31 DEBUG 06:30:31 DEBUG 06:30:31 DEBUG 06:30:31 DEBUG "pod": "ds-idrepo-1" 06:30:31 DEBUG }, 06:30:31 DEBUG "value": [ 06:30:31 DEBUG 1712727031, 06:30:31 DEBUG "0" 06:30:31 DEBUG ] 06:30:31 DEBUG } 06:30:31 DEBUG ] 06:30:31 DEBUG } 06:30:31 DEBUG } 06:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727031" 06:30:31 INFO [http_cmd]: http status code OK 06:30:31 DEBUG --- status code --- 06:30:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:31 DEBUG --- http response --- 06:30:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:30:31 DEBUG ----- output ----- 06:30:31 DEBUG { 06:30:31 DEBUG "status": "success", 06:30:31 DEBUG "data": { 06:30:31 DEBUG "resultType": "vector", 06:30:31 DEBUG "result": [ 06:30:31 DEBUG { 06:30:31 DEBUG "metric": { 06:30:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:30:31 DEBUG }, 06:30:31 DEBUG "value": [ 06:30:31 DEBUG 06:30:31 DEBUG 06:30:31 DEBUG 06:30:31 DEBUG [...] 06:30:31 DEBUG 06:30:31 DEBUG 06:30:31 DEBUG 06:30:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:30:31 DEBUG }, 06:30:31 DEBUG "value": [ 06:30:31 DEBUG 1712727031, 06:30:31 DEBUG "1436.2" 06:30:31 DEBUG ] 06:30:31 DEBUG } 06:30:31 DEBUG ] 06:30:31 DEBUG } 06:30:31 DEBUG } 06:30:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727031" 06:30:32 INFO [http_cmd]: http status code OK 06:30:32 DEBUG --- status code --- 06:30:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:32 DEBUG --- http response --- 06:30:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:30:32 DEBUG ----- output ----- 06:30:32 DEBUG { 06:30:32 DEBUG "status": "success", 06:30:32 DEBUG "data": { 06:30:32 DEBUG "resultType": "vector", 06:30:32 DEBUG "result": [ 06:30:32 DEBUG { 06:30:32 DEBUG "metric": { 06:30:32 DEBUG "pod": "ds-idrepo-0" 06:30:32 DEBUG }, 06:30:32 DEBUG "value": [ 06:30:32 DEBUG 06:30:32 DEBUG 06:30:32 DEBUG 06:30:32 DEBUG [...] 06:30:32 DEBUG 06:30:32 DEBUG 06:30:32 DEBUG 06:30:32 DEBUG "pod": "ds-idrepo-1" 06:30:32 DEBUG }, 06:30:32 DEBUG "value": [ 06:30:32 DEBUG 1712727031, 06:30:32 DEBUG "0" 06:30:32 DEBUG ] 06:30:32 DEBUG } 06:30:32 DEBUG ] 06:30:32 DEBUG } 06:30:32 DEBUG } 06:30:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727032" 06:30:32 INFO [http_cmd]: http status code OK 06:30:32 DEBUG --- status code --- 06:30:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:32 DEBUG --- http response --- 06:30:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:30:32 DEBUG ----- output ----- 06:30:32 DEBUG { 06:30:32 DEBUG "status": "success", 06:30:32 DEBUG "data": { 06:30:32 DEBUG "resultType": "vector", 06:30:32 DEBUG "result": [ 06:30:32 DEBUG { 06:30:32 DEBUG "metric": { 06:30:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:30:32 DEBUG }, 06:30:32 DEBUG "value": [ 06:30:32 DEBUG 06:30:32 DEBUG 06:30:32 DEBUG 06:30:32 DEBUG [...] 06:30:32 DEBUG 06:30:32 DEBUG 06:30:32 DEBUG 06:30:32 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:30:32 DEBUG }, 06:30:32 DEBUG "value": [ 06:30:32 DEBUG 1712727032, 06:30:32 DEBUG "0" 06:30:32 DEBUG ] 06:30:32 DEBUG } 06:30:32 DEBUG ] 06:30:32 DEBUG } 06:30:32 DEBUG } 06:30:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712727032" 06:30:32 INFO [http_cmd]: http status code OK 06:30:32 DEBUG --- status code --- 06:30:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:32 DEBUG --- http response --- 06:30:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:30:32 DEBUG ----- output ----- 06:30:32 DEBUG { 06:30:32 DEBUG "status": "success", 06:30:32 DEBUG "data": { 06:30:32 DEBUG "resultType": "vector", 06:30:32 DEBUG "result": [ 06:30:32 DEBUG { 06:30:32 DEBUG "metric": { 06:30:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:30:32 DEBUG }, 06:30:32 DEBUG "value": [ 06:30:32 DEBUG 06:30:32 DEBUG 06:30:32 DEBUG 06:30:32 DEBUG [...] 06:30:32 DEBUG 06:30:32 DEBUG 06:30:32 DEBUG 06:30:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:30:32 DEBUG }, 06:30:32 DEBUG "value": [ 06:30:32 DEBUG 1712727032, 06:30:32 DEBUG "2488918016" 06:30:32 DEBUG ] 06:30:32 DEBUG } 06:30:32 DEBUG ] 06:30:32 DEBUG } 06:30:32 DEBUG } 06:30:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727032" 06:30:32 INFO [http_cmd]: http status code OK 06:30:32 DEBUG --- status code --- 06:30:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:32 DEBUG --- http response --- 06:30:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:30:32 DEBUG ----- output ----- 06:30:32 DEBUG { 06:30:32 DEBUG "status": "success", 06:30:32 DEBUG "data": { 06:30:32 DEBUG "resultType": "vector", 06:30:32 DEBUG "result": [ 06:30:32 DEBUG { 06:30:32 DEBUG "metric": { 06:30:32 DEBUG "instance": "10.142.0.25:9100" 06:30:32 DEBUG }, 06:30:32 DEBUG "value": [ 06:30:32 DEBUG 06:30:32 DEBUG 06:30:32 DEBUG 06:30:32 DEBUG [...] 06:30:32 DEBUG 06:30:32 DEBUG 06:30:32 DEBUG 06:30:32 DEBUG "instance": "10.142.0.77:9100" 06:30:32 DEBUG }, 06:30:32 DEBUG "value": [ 06:30:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727032" 06:30:32 DEBUG 1712727032, 06:30:32 DEBUG "42482851.53333333" 06:30:32 DEBUG ] 06:30:32 DEBUG } 06:30:32 DEBUG ] 06:30:32 DEBUG } 06:30:32 DEBUG } 06:30:32 INFO [http_cmd]: http status code OK 06:30:32 DEBUG --- status code --- 06:30:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:30:32 DEBUG --- http response --- 06:30:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:30:32 DEBUG ----- output ----- 06:30:32 DEBUG { 06:30:32 DEBUG "status": "success", 06:30:32 DEBUG "data": { 06:30:32 DEBUG "resultType": "vector", 06:30:32 DEBUG "result": [ 06:30:32 DEBUG { 06:30:32 DEBUG "metric": { 06:30:32 DEBUG "instance": "10.142.0.25:9100" 06:30:32 DEBUG }, 06:30:32 DEBUG "value": [ 06:30:32 DEBUG 06:30:32 DEBUG 06:30:32 DEBUG 06:30:32 DEBUG [...] 06:30:32 DEBUG 06:30:32 DEBUG 06:30:32 DEBUG 06:30:32 DEBUG "instance": "10.142.0.77:9100" 06:30:32 DEBUG }, 06:30:32 DEBUG "value": [ 06:30:32 DEBUG 1712727032, 06:30:32 DEBUG "0.031666666666668064" 06:30:32 DEBUG ] 06:30:32 DEBUG } 06:30:32 DEBUG ] 06:30:32 DEBUG } 06:30:32 DEBUG } 06:30:51 INFO 06:30:51 INFO [loop_until]: kubectl --namespace=xlou top pods 06:30:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:30:51 INFO [loop_until]: OK (rc = 0) 06:30:51 DEBUG --- stdout --- 06:30:51 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10612m 8988Mi am-654c4fcbcf-9scrj 9755m 8465Mi am-654c4fcbcf-v5qb5 8247m 8511Mi ds-cts-0 8498m 13392Mi ds-cts-1 8566m 12966Mi ds-cts-2 7708m 13353Mi ds-idrepo-0 10810m 5623Mi ds-idrepo-1 367m 5673Mi ds-idrepo-2 386m 5702Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 9m 1191Mi idm-6bfb7c96cc-vsb58 17m 2213Mi lodemon-b776b7d88-thrqf 6m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4347m 2440Mi 06:30:51 DEBUG --- stderr --- 06:30:51 DEBUG 06:30:59 INFO 06:30:59 INFO [loop_until]: kubectl --namespace=xlou top node 06:30:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:30:59 INFO [loop_until]: OK (rc = 0) 06:30:59 DEBUG --- stdout --- 06:30:59 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 110m 0% 2311Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9033m 56% 9374Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 93m 0% 3260Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13699m 86% 6797Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 11049m 69% 9342Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11945m 75% 9805Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 76m 0% 852Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10894m 68% 6309Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8633m 54% 14098Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 7800m 49% 14032Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 449m 2% 6389Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8570m 53% 13934Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 432m 2% 6354Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4511m 28% 6163Mi 10% 06:30:59 DEBUG --- stderr --- 06:30:59 DEBUG 06:31:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712727089" 06:31:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727089" 06:31:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727089" 06:31:29 ERROR [http_cmd]: ERROR 06:31:29 INFO [http_cmd]: http status code OK 06:31:29 DEBUG --- status code --- 06:31:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:29 DEBUG --- http response --- 06:31:29 ERROR --- status code --- 06:31:29 INFO [http_cmd]: http status code OK 06:31:29 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:31:29 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:31:29 DEBUG --- status code --- 06:31:29 ERROR --- http response --- 06:31:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:29 DEBUG --- http response --- 06:31:29 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:31:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:31:29 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:31:29 DEBUG ----- output ----- 06:31:29 DEBUG { 06:31:29 DEBUG "status": "success", 06:31:29 DEBUG "data": { 06:31:29 DEBUG "resultType": "vector", 06:31:29 DEBUG "result": [ 06:31:29 DEBUG { 06:31:29 DEBUG "metric": { 06:31:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:31:29 DEBUG }, 06:31:29 DEBUG "value": [ 06:31:29 DEBUG 06:31:29 DEBUG 06:31:29 DEBUG 06:31:29 DEBUG [...] 06:31:29 DEBUG 06:31:29 DEBUG 06:31:29 DEBUG 06:31:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:31:29 DEBUG }, 06:31:29 DEBUG "value": [ 06:31:29 DEBUG 1712727089, 06:31:29 DEBUG "0" 06:31:29 DEBUG ] 06:31:29 DEBUG } 06:31:29 DEBUG ] 06:31:29 DEBUG } 06:31:29 DEBUG } 06:31:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727090" 06:31:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727090" 06:31:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727090" 06:31:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727090" 06:31:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727090" 06:31:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727090" 06:31:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727090" 06:31:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727090" 06:31:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727090" 06:31:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727090" 06:31:30 INFO [http_cmd]: http status code OK 06:31:30 DEBUG --- status code --- 06:31:30 INFO [http_cmd]: http status code OK 06:31:30 INFO [http_cmd]: http status code OK 06:31:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:30 DEBUG --- http response --- 06:31:30 INFO [http_cmd]: http status code OK 06:31:30 DEBUG --- status code --- 06:31:30 INFO [http_cmd]: http status code OK 06:31:30 DEBUG --- status code --- 06:31:30 INFO [http_cmd]: http status code OK 06:31:30 DEBUG --- status code --- 06:31:30 INFO [http_cmd]: http status code OK 06:31:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:31:30 DEBUG --- status code --- 06:31:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:30 DEBUG --- status code --- 06:31:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:30 DEBUG --- http response --- 06:31:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:30 INFO [http_cmd]: http status code OK 06:31:30 DEBUG --- status code --- 06:31:30 DEBUG ----- output ----- 06:31:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:30 DEBUG --- http response --- 06:31:30 INFO [http_cmd]: http status code OK 06:31:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:30 DEBUG --- http response --- 06:31:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:31:30 DEBUG --- http response --- 06:31:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:30 DEBUG --- http response --- 06:31:30 DEBUG { 06:31:30 DEBUG --- http response --- 06:31:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:31:30 DEBUG --- status code --- 06:31:30 INFO [http_cmd]: http status code OK 06:31:30 DEBUG --- status code --- 06:31:30 DEBUG ----- output ----- 06:31:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:31:30 DEBUG --- status code --- 06:31:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:30 DEBUG "status": "success", 06:31:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:31:30 DEBUG ----- output ----- 06:31:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:30 DEBUG --- http response --- 06:31:30 DEBUG { 06:31:30 DEBUG ----- output ----- 06:31:30 DEBUG --- http response --- 06:31:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:31:30 DEBUG "data": { 06:31:30 DEBUG ----- output ----- 06:31:30 DEBUG { 06:31:30 DEBUG --- http response --- 06:31:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:31:30 DEBUG ----- output ----- 06:31:30 DEBUG { 06:31:30 DEBUG { 06:31:30 DEBUG ----- output ----- 06:31:30 DEBUG { 06:31:30 DEBUG "resultType": "vector", 06:31:30 DEBUG { 06:31:30 DEBUG "status": "success", 06:31:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:31:30 DEBUG ----- output ----- 06:31:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:31:30 DEBUG ----- output ----- 06:31:30 DEBUG "status": "success", 06:31:30 DEBUG "status": "success", 06:31:30 DEBUG "data": { 06:31:30 DEBUG "result": [ 06:31:30 DEBUG "status": "success", 06:31:30 DEBUG "data": { 06:31:30 DEBUG "status": "success", 06:31:30 DEBUG "data": { 06:31:30 DEBUG "resultType": "vector", 06:31:30 DEBUG "status": "success", 06:31:30 DEBUG "data": { 06:31:30 DEBUG "resultType": "vector", 06:31:30 DEBUG "result": [ 06:31:30 DEBUG { 06:31:30 DEBUG "data": { 06:31:30 DEBUG "resultType": "vector", 06:31:30 DEBUG { 06:31:30 DEBUG { 06:31:30 DEBUG "result": [ 06:31:30 DEBUG "data": { 06:31:30 DEBUG "resultType": "vector", 06:31:30 DEBUG { 06:31:30 DEBUG "metric": { 06:31:30 DEBUG "metric": { 06:31:30 DEBUG "resultType": "vector", 06:31:30 DEBUG "result": [ 06:31:30 DEBUG "status": "success", 06:31:30 DEBUG "status": "success", 06:31:30 DEBUG { 06:31:30 DEBUG "resultType": "vector", 06:31:30 DEBUG "result": [ 06:31:30 DEBUG "instance": "10.142.0.25:9100" 06:31:30 DEBUG }, 06:31:30 DEBUG "pod": "ds-idrepo-0" 06:31:30 DEBUG "result": [ 06:31:30 DEBUG { 06:31:30 DEBUG "data": { 06:31:30 DEBUG "data": { 06:31:30 DEBUG "metric": { 06:31:30 DEBUG "result": [ 06:31:30 DEBUG { 06:31:30 DEBUG "value": [ 06:31:30 DEBUG 06:31:30 DEBUG }, 06:31:30 DEBUG { 06:31:30 DEBUG "metric": { 06:31:30 DEBUG "resultType": "vector", 06:31:30 DEBUG "resultType": "vector", 06:31:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:31:30 DEBUG }, 06:31:30 DEBUG "metric": { 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG "value": [ 06:31:30 DEBUG "metric": { 06:31:30 DEBUG "instance": "10.142.0.25:9100" 06:31:30 DEBUG "result": [ 06:31:30 DEBUG "result": [ 06:31:30 DEBUG { 06:31:30 DEBUG "value": [ 06:31:30 DEBUG "pod": "ds-idrepo-0" 06:31:30 DEBUG [...] 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG "pod": "ds-idrepo-0" 06:31:30 DEBUG }, 06:31:30 DEBUG { 06:31:30 DEBUG "metric": { 06:31:30 DEBUG { 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:31:30 DEBUG 06:31:30 DEBUG }, 06:31:30 DEBUG "value": [ 06:31:30 DEBUG "metric": { 06:31:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:31:30 DEBUG }, 06:31:30 DEBUG "value": [ 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG ----- output ----- 06:31:30 DEBUG 06:31:30 DEBUG "value": [ 06:31:30 DEBUG 06:31:30 DEBUG "instance": "10.142.0.25:9100" 06:31:30 DEBUG 06:31:30 DEBUG "metric": { 06:31:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:31:30 DEBUG [...] 06:31:30 DEBUG 06:31:30 DEBUG { 06:31:30 DEBUG "status": "success", 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG }, 06:31:30 DEBUG "value": [ 06:31:30 DEBUG }, 06:31:30 DEBUG "value": [ 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG "data": { 06:31:30 DEBUG "resultType": "vector", 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG }, 06:31:30 DEBUG "value": [ 06:31:30 DEBUG 06:31:30 DEBUG "result": [ 06:31:30 DEBUG { 06:31:30 DEBUG 06:31:30 DEBUG [...] 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG [...] 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG "metric": { 06:31:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:31:30 DEBUG [...] 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG [...] 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG "instance": "10.142.0.77:9100" 06:31:30 DEBUG }, 06:31:30 DEBUG "value": [ 06:31:30 DEBUG 06:31:30 DEBUG [...] 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:31:30 DEBUG }, 06:31:30 DEBUG "value": [ 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:31:30 DEBUG "instance": "10.142.0.77:9100" 06:31:30 DEBUG 06:31:30 DEBUG 1712727090, 06:31:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:31:30 DEBUG }, 06:31:30 DEBUG "value": [ 06:31:30 DEBUG 06:31:30 DEBUG }, 06:31:30 DEBUG "value": [ 06:31:30 DEBUG 1712727090, 06:31:30 DEBUG "1430.0666666666666" 06:31:30 DEBUG ] 06:31:30 DEBUG "42615710.233333334" 06:31:30 DEBUG 1712727090, 06:31:30 DEBUG "1429.8666666666666" 06:31:30 DEBUG ] 06:31:30 DEBUG } 06:31:30 DEBUG ] 06:31:30 DEBUG } 06:31:30 DEBUG } 06:31:30 DEBUG 06:31:30 DEBUG [...] 06:31:30 DEBUG ] 06:31:30 DEBUG "pod": "ds-cts-2" 06:31:30 DEBUG }, 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:31:30 DEBUG }, 06:31:30 DEBUG "value": [ 06:31:30 DEBUG } 06:31:30 DEBUG ] 06:31:30 DEBUG } 06:31:30 DEBUG "value": [ 06:31:30 DEBUG }, 06:31:30 DEBUG "value": [ 06:31:30 DEBUG 1712727090, 06:31:30 DEBUG "0" 06:31:30 DEBUG ] 06:31:30 DEBUG } 06:31:30 DEBUG }, 06:31:30 DEBUG 1712727090, 06:31:30 DEBUG "258337.5" 06:31:30 DEBUG ] 06:31:30 DEBUG } 06:31:30 DEBUG } 06:31:30 DEBUG ] 06:31:30 DEBUG } 06:31:30 DEBUG "value": [ 06:31:30 DEBUG 06:31:30 DEBUG ] 06:31:30 DEBUG } 06:31:30 DEBUG } 06:31:30 DEBUG } 06:31:30 DEBUG 1712727090, 06:31:30 DEBUG "instance": "10.142.0.77:9100" 06:31:30 DEBUG 1712727090, 06:31:30 DEBUG 06:31:30 DEBUG } 06:31:30 DEBUG [...] 06:31:30 DEBUG "0.15243333333333275" 06:31:30 DEBUG }, 06:31:30 DEBUG "29.8" 06:31:30 DEBUG 06:31:30 DEBUG [...] 06:31:30 DEBUG 06:31:30 DEBUG ] 06:31:30 DEBUG "value": [ 06:31:30 DEBUG ] 06:31:30 DEBUG 06:31:30 DEBUG ] 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG 1712727090, 06:31:30 DEBUG } 06:31:30 DEBUG 06:31:30 DEBUG } 06:31:30 DEBUG "pod": "ds-cts-2" 06:31:30 DEBUG }, 06:31:30 DEBUG "404634407936" 06:31:30 DEBUG ] 06:31:30 DEBUG 06:31:30 DEBUG } 06:31:30 DEBUG "value": [ 06:31:30 DEBUG 1712727090, 06:31:30 DEBUG "0" 06:31:30 DEBUG } 06:31:30 DEBUG "pod": "ds-idrepo-1" 06:31:30 DEBUG }, 06:31:30 DEBUG ] 06:31:30 DEBUG ] 06:31:30 DEBUG } 06:31:30 DEBUG "value": [ 06:31:30 DEBUG 1712727090, 06:31:30 DEBUG } 06:31:30 DEBUG } 06:31:30 DEBUG ] 06:31:30 DEBUG "0" 06:31:30 DEBUG ] 06:31:30 DEBUG } 06:31:30 DEBUG } 06:31:30 DEBUG } 06:31:30 DEBUG ] 06:31:30 DEBUG ] 06:31:30 DEBUG } 06:31:30 DEBUG } 06:31:30 DEBUG } 06:31:30 DEBUG } 06:31:30 DEBUG } 06:31:30 DEBUG ] 06:31:30 DEBUG } 06:31:30 DEBUG } 06:31:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727090" 06:31:30 INFO [http_cmd]: http status code OK 06:31:30 DEBUG --- status code --- 06:31:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:30 DEBUG --- http response --- 06:31:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:31:30 DEBUG ----- output ----- 06:31:30 DEBUG { 06:31:30 DEBUG "status": "success", 06:31:30 DEBUG "data": { 06:31:30 DEBUG "resultType": "vector", 06:31:30 DEBUG "result": [ 06:31:30 DEBUG { 06:31:30 DEBUG "metric": { 06:31:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:31:30 DEBUG }, 06:31:30 DEBUG "value": [ 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG [...] 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:31:30 DEBUG }, 06:31:30 DEBUG "value": [ 06:31:30 DEBUG 1712727090, 06:31:30 DEBUG "38093561.47745325" 06:31:30 DEBUG ] 06:31:30 DEBUG } 06:31:30 DEBUG ] 06:31:30 DEBUG } 06:31:30 DEBUG } 06:31:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712727090" 06:31:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712727090" 06:31:30 INFO [http_cmd]: http status code OK 06:31:30 DEBUG --- status code --- 06:31:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:30 DEBUG --- http response --- 06:31:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:31:30 DEBUG ----- output ----- 06:31:30 DEBUG { 06:31:30 DEBUG "status": "success", 06:31:30 DEBUG "data": { 06:31:30 DEBUG "resultType": "vector", 06:31:30 DEBUG "result": [ 06:31:30 DEBUG { 06:31:30 DEBUG "metric": { 06:31:30 DEBUG "pod": "ds-idrepo-0" 06:31:30 DEBUG }, 06:31:30 DEBUG "value": [ 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG [...] 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG "pod": "ds-cts-2" 06:31:30 DEBUG }, 06:31:30 DEBUG "value": [ 06:31:30 DEBUG 1712727090, 06:31:30 DEBUG "0" 06:31:30 DEBUG ] 06:31:30 DEBUG } 06:31:30 DEBUG ] 06:31:30 DEBUG } 06:31:30 DEBUG } 06:31:30 INFO [http_cmd]: http status code OK 06:31:30 DEBUG --- status code --- 06:31:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:30 DEBUG --- http response --- 06:31:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:31:30 DEBUG ----- output ----- 06:31:30 DEBUG { 06:31:30 DEBUG "status": "success", 06:31:30 DEBUG "data": { 06:31:30 DEBUG "resultType": "vector", 06:31:30 DEBUG "result": [ 06:31:30 DEBUG { 06:31:30 DEBUG "metric": { 06:31:30 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:31:30 DEBUG }, 06:31:30 DEBUG "value": [ 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG [...] 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG 06:31:30 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:31:30 DEBUG }, 06:31:30 DEBUG "value": [ 06:31:30 DEBUG 1712727090, 06:31:30 DEBUG "4.47182121592144" 06:31:30 DEBUG ] 06:31:30 DEBUG } 06:31:30 DEBUG ] 06:31:30 DEBUG } 06:31:30 DEBUG } 06:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712727091" 06:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727091" 06:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727091" 06:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727091" 06:31:31 INFO [http_cmd]: http status code OK 06:31:31 DEBUG --- status code --- 06:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:31 DEBUG --- http response --- 06:31:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:31:31 DEBUG ----- output ----- 06:31:31 DEBUG { 06:31:31 DEBUG "status": "success", 06:31:31 DEBUG "data": { 06:31:31 DEBUG "resultType": "vector", 06:31:31 DEBUG "result": [ 06:31:31 DEBUG { 06:31:31 DEBUG "metric": { 06:31:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:31:31 DEBUG }, 06:31:31 DEBUG "value": [ 06:31:31 DEBUG 06:31:31 DEBUG 06:31:31 DEBUG 06:31:31 DEBUG [...] 06:31:31 DEBUG 06:31:31 DEBUG 06:31:31 DEBUG 06:31:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:31:31 DEBUG }, 06:31:31 DEBUG "value": [ 06:31:31 DEBUG 1712727091, 06:31:31 DEBUG "0.051" 06:31:31 DEBUG ] 06:31:31 DEBUG } 06:31:31 INFO [http_cmd]: http status code OK 06:31:31 DEBUG ] 06:31:31 DEBUG --- status code --- 06:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:31 DEBUG --- http response --- 06:31:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:31:31 DEBUG ----- output ----- 06:31:31 DEBUG { 06:31:31 DEBUG "status": "success", 06:31:31 DEBUG "data": { 06:31:31 DEBUG "resultType": "vector", 06:31:31 DEBUG "result": [ 06:31:31 DEBUG { 06:31:31 DEBUG } 06:31:31 INFO [http_cmd]: http status code OK 06:31:31 DEBUG "metric": { 06:31:31 DEBUG } 06:31:31 DEBUG --- status code --- 06:31:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:31 DEBUG }, 06:31:31 DEBUG --- http response --- 06:31:31 DEBUG "value": [ 06:31:31 INFO [http_cmd]: http status code OK 06:31:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:31:31 DEBUG 06:31:31 DEBUG --- status code --- 06:31:31 DEBUG ----- output ----- 06:31:31 DEBUG 06:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:31 DEBUG { 06:31:31 DEBUG 06:31:31 DEBUG --- http response --- 06:31:31 DEBUG "status": "success", 06:31:31 DEBUG [...] 06:31:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:31:31 DEBUG "data": { 06:31:31 DEBUG 06:31:31 DEBUG ----- output ----- 06:31:31 DEBUG "resultType": "vector", 06:31:31 DEBUG 06:31:31 DEBUG { 06:31:31 DEBUG "result": [ 06:31:31 DEBUG 06:31:31 DEBUG "status": "success", 06:31:31 DEBUG { 06:31:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:31:31 DEBUG "data": { 06:31:31 DEBUG "metric": { 06:31:31 DEBUG }, 06:31:31 DEBUG "resultType": "vector", 06:31:31 DEBUG "pod": "ds-idrepo-2" 06:31:31 DEBUG "value": [ 06:31:31 DEBUG "result": [ 06:31:31 DEBUG }, 06:31:31 DEBUG 1712727091, 06:31:31 DEBUG { 06:31:31 DEBUG "value": [ 06:31:31 DEBUG "1430.1666666666667" 06:31:31 DEBUG "metric": { 06:31:31 DEBUG 06:31:31 DEBUG ] 06:31:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:31:31 DEBUG }, 06:31:31 DEBUG } 06:31:31 DEBUG "value": [ 06:31:31 DEBUG 06:31:31 DEBUG ] 06:31:31 DEBUG 06:31:31 DEBUG 06:31:31 DEBUG 06:31:31 DEBUG 06:31:31 DEBUG } 06:31:31 DEBUG } 06:31:31 DEBUG [...] 06:31:31 DEBUG [...] 06:31:31 DEBUG 06:31:31 DEBUG 06:31:31 DEBUG 06:31:31 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:31:31 DEBUG }, 06:31:31 DEBUG "value": [ 06:31:31 DEBUG 1712727091, 06:31:31 DEBUG "0.005755984977620586" 06:31:31 DEBUG ] 06:31:31 DEBUG } 06:31:31 DEBUG ] 06:31:31 DEBUG } 06:31:31 DEBUG } 06:31:31 DEBUG 06:31:31 DEBUG 06:31:31 DEBUG 06:31:31 DEBUG "pod": "ds-cts-2" 06:31:31 DEBUG }, 06:31:31 DEBUG "value": [ 06:31:31 DEBUG 1712727091, 06:31:31 DEBUG "0.00011720555555555557" 06:31:31 DEBUG ] 06:31:31 DEBUG } 06:31:31 DEBUG ] 06:31:31 DEBUG } 06:31:31 DEBUG } 06:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727091" 06:31:31 INFO [http_cmd]: http status code OK 06:31:31 DEBUG --- status code --- 06:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:31 DEBUG --- http response --- 06:31:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:31:31 DEBUG ----- output ----- 06:31:31 DEBUG { 06:31:31 DEBUG "status": "success", 06:31:31 DEBUG "data": { 06:31:31 DEBUG "resultType": "vector", 06:31:31 DEBUG "result": [ 06:31:31 DEBUG { 06:31:31 DEBUG "metric": { 06:31:31 DEBUG "pod": "ds-idrepo-0" 06:31:31 DEBUG }, 06:31:31 DEBUG "value": [ 06:31:31 DEBUG 06:31:31 DEBUG 06:31:31 DEBUG 06:31:31 DEBUG [...] 06:31:31 DEBUG 06:31:31 DEBUG 06:31:31 DEBUG 06:31:31 DEBUG "pod": "ds-idrepo-1" 06:31:31 DEBUG }, 06:31:31 DEBUG "value": [ 06:31:31 DEBUG 1712727091, 06:31:31 DEBUG "0" 06:31:31 DEBUG ] 06:31:31 DEBUG } 06:31:31 DEBUG ] 06:31:31 DEBUG } 06:31:31 DEBUG } 06:31:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727091" 06:31:31 INFO [http_cmd]: http status code OK 06:31:31 DEBUG --- status code --- 06:31:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:31 DEBUG --- http response --- 06:31:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:31:31 DEBUG ----- output ----- 06:31:31 DEBUG { 06:31:31 DEBUG "status": "success", 06:31:31 DEBUG "data": { 06:31:31 DEBUG "resultType": "vector", 06:31:31 DEBUG "result": [ 06:31:31 DEBUG { 06:31:31 DEBUG "metric": { 06:31:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:31:31 DEBUG }, 06:31:31 DEBUG "value": [ 06:31:31 DEBUG 06:31:31 DEBUG 06:31:31 DEBUG 06:31:31 DEBUG [...] 06:31:31 DEBUG 06:31:31 DEBUG 06:31:31 DEBUG 06:31:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:31:31 DEBUG }, 06:31:31 DEBUG "value": [ 06:31:31 DEBUG 1712727091, 06:31:31 DEBUG "1436.3" 06:31:31 DEBUG ] 06:31:31 DEBUG } 06:31:31 DEBUG ] 06:31:31 DEBUG } 06:31:31 DEBUG } 06:31:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727092" 06:31:32 INFO [http_cmd]: http status code OK 06:31:32 DEBUG --- status code --- 06:31:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:32 DEBUG --- http response --- 06:31:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:31:32 DEBUG ----- output ----- 06:31:32 DEBUG { 06:31:32 DEBUG "status": "success", 06:31:32 DEBUG "data": { 06:31:32 DEBUG "resultType": "vector", 06:31:32 DEBUG "result": [ 06:31:32 DEBUG { 06:31:32 DEBUG "metric": { 06:31:32 DEBUG "pod": "ds-idrepo-0" 06:31:32 DEBUG }, 06:31:32 DEBUG "value": [ 06:31:32 DEBUG 06:31:32 DEBUG 06:31:32 DEBUG 06:31:32 DEBUG [...] 06:31:32 DEBUG 06:31:32 DEBUG 06:31:32 DEBUG 06:31:32 DEBUG "pod": "ds-idrepo-1" 06:31:32 DEBUG }, 06:31:32 DEBUG "value": [ 06:31:32 DEBUG 1712727092, 06:31:32 DEBUG "0" 06:31:32 DEBUG ] 06:31:32 DEBUG } 06:31:32 DEBUG ] 06:31:32 DEBUG } 06:31:32 DEBUG } 06:31:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727092" 06:31:32 INFO [http_cmd]: http status code OK 06:31:32 DEBUG --- status code --- 06:31:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:32 DEBUG --- http response --- 06:31:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:31:32 DEBUG ----- output ----- 06:31:32 DEBUG { 06:31:32 DEBUG "status": "success", 06:31:32 DEBUG "data": { 06:31:32 DEBUG "resultType": "vector", 06:31:32 DEBUG "result": [ 06:31:32 DEBUG { 06:31:32 DEBUG "metric": { 06:31:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:31:32 DEBUG }, 06:31:32 DEBUG "value": [ 06:31:32 DEBUG 06:31:32 DEBUG 06:31:32 DEBUG 06:31:32 DEBUG [...] 06:31:32 DEBUG 06:31:32 DEBUG 06:31:32 DEBUG 06:31:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:31:32 DEBUG }, 06:31:32 DEBUG "value": [ 06:31:32 DEBUG 1712727092, 06:31:32 DEBUG "0" 06:31:32 DEBUG ] 06:31:32 DEBUG } 06:31:32 DEBUG ] 06:31:32 DEBUG } 06:31:32 DEBUG } 06:31:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712727092" 06:31:32 INFO [http_cmd]: http status code OK 06:31:32 DEBUG --- status code --- 06:31:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:32 DEBUG --- http response --- 06:31:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:31:32 DEBUG ----- output ----- 06:31:32 DEBUG { 06:31:32 DEBUG "status": "success", 06:31:32 DEBUG "data": { 06:31:32 DEBUG "resultType": "vector", 06:31:32 DEBUG "result": [ 06:31:32 DEBUG { 06:31:32 DEBUG "metric": { 06:31:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:31:32 DEBUG }, 06:31:32 DEBUG "value": [ 06:31:32 DEBUG 06:31:32 DEBUG 06:31:32 DEBUG 06:31:32 DEBUG [...] 06:31:32 DEBUG 06:31:32 DEBUG 06:31:32 DEBUG 06:31:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:31:32 DEBUG }, 06:31:32 DEBUG "value": [ 06:31:32 DEBUG 1712727092, 06:31:32 DEBUG "2485145600" 06:31:32 DEBUG ] 06:31:32 DEBUG } 06:31:32 DEBUG ] 06:31:32 DEBUG } 06:31:32 DEBUG } 06:31:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727092" 06:31:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727092" 06:31:32 INFO [http_cmd]: http status code OK 06:31:32 DEBUG --- status code --- 06:31:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:32 DEBUG --- http response --- 06:31:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:31:32 DEBUG ----- output ----- 06:31:32 DEBUG { 06:31:32 DEBUG "status": "success", 06:31:32 DEBUG "data": { 06:31:32 DEBUG "resultType": "vector", 06:31:32 DEBUG "result": [ 06:31:32 DEBUG { 06:31:32 DEBUG "metric": { 06:31:32 DEBUG "instance": "10.142.0.25:9100" 06:31:32 DEBUG }, 06:31:32 DEBUG "value": [ 06:31:32 DEBUG 06:31:32 DEBUG 06:31:32 DEBUG 06:31:32 DEBUG [...] 06:31:32 DEBUG 06:31:32 DEBUG 06:31:32 DEBUG 06:31:32 DEBUG "instance": "10.142.0.77:9100" 06:31:32 DEBUG }, 06:31:32 DEBUG "value": [ 06:31:32 DEBUG 1712727092, 06:31:32 DEBUG "42617514.96666667" 06:31:32 DEBUG ] 06:31:32 DEBUG } 06:31:32 DEBUG ] 06:31:32 DEBUG } 06:31:32 DEBUG } 06:31:32 INFO [http_cmd]: http status code OK 06:31:32 DEBUG --- status code --- 06:31:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:32 DEBUG --- http response --- 06:31:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:31:32 DEBUG ----- output ----- 06:31:32 DEBUG { 06:31:32 DEBUG "status": "success", 06:31:32 DEBUG "data": { 06:31:32 DEBUG "resultType": "vector", 06:31:32 DEBUG "result": [ 06:31:32 DEBUG { 06:31:32 DEBUG "metric": { 06:31:32 DEBUG "instance": "10.142.0.25:9100" 06:31:32 DEBUG }, 06:31:32 DEBUG "value": [ 06:31:32 DEBUG 06:31:32 DEBUG 06:31:32 DEBUG 06:31:32 DEBUG [...] 06:31:32 DEBUG 06:31:32 DEBUG 06:31:32 DEBUG 06:31:32 DEBUG "instance": "10.142.0.77:9100" 06:31:32 DEBUG }, 06:31:32 DEBUG "value": [ 06:31:32 DEBUG 1712727092, 06:31:32 DEBUG "0.054000000000000145" 06:31:32 DEBUG ] 06:31:32 DEBUG } 06:31:32 DEBUG ] 06:31:32 DEBUG } 06:31:32 DEBUG } 06:31:51 INFO 06:31:51 INFO [loop_until]: kubectl --namespace=xlou top pods 06:31:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:31:51 INFO [loop_until]: OK (rc = 0) 06:31:51 DEBUG --- stdout --- 06:31:51 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10968m 9002Mi am-654c4fcbcf-9scrj 10046m 8456Mi am-654c4fcbcf-v5qb5 8433m 8489Mi ds-cts-0 8391m 12845Mi ds-cts-1 8792m 12556Mi ds-cts-2 7524m 12753Mi ds-idrepo-0 10613m 5623Mi ds-idrepo-1 366m 5673Mi ds-idrepo-2 418m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1191Mi idm-6bfb7c96cc-vsb58 3m 2213Mi lodemon-b776b7d88-thrqf 2m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4360m 2501Mi 06:31:51 DEBUG --- stderr --- 06:31:51 DEBUG 06:31:59 INFO 06:31:59 INFO [loop_until]: kubectl --namespace=xlou top node 06:31:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:32:00 INFO [loop_until]: OK (rc = 0) 06:32:00 DEBUG --- stdout --- 06:32:00 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 89m 0% 2307Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9256m 58% 9361Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 85m 0% 3258Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13978m 87% 6780Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10877m 68% 9335Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11953m 75% 9809Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 78m 0% 854Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10900m 68% 6305Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8356m 52% 13522Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 7719m 48% 13419Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 416m 2% 6392Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8682m 54% 13359Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 428m 2% 6358Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4479m 28% 6088Mi 10% 06:32:00 DEBUG --- stderr --- 06:32:00 DEBUG 06:32:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727149" 06:32:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727149" 06:32:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712727149" 06:32:29 ERROR [http_cmd]: ERROR 06:32:29 INFO [http_cmd]: http status code OK 06:32:29 INFO [http_cmd]: http status code OK 06:32:29 DEBUG --- status code --- 06:32:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:29 DEBUG --- http response --- 06:32:29 DEBUG --- status code --- 06:32:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:32:29 DEBUG ----- output ----- 06:32:29 DEBUG { 06:32:29 DEBUG "status": "success", 06:32:29 DEBUG "data": { 06:32:29 DEBUG "resultType": "vector", 06:32:29 DEBUG "result": [ 06:32:29 DEBUG { 06:32:29 DEBUG "metric": { 06:32:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:32:29 DEBUG }, 06:32:29 DEBUG "value": [ 06:32:29 DEBUG 06:32:29 DEBUG 06:32:29 DEBUG 06:32:29 DEBUG [...] 06:32:29 DEBUG 06:32:29 DEBUG 06:32:29 DEBUG 06:32:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:32:29 DEBUG }, 06:32:29 DEBUG "value": [ 06:32:29 DEBUG 1712727149, 06:32:29 DEBUG "0" 06:32:29 DEBUG ] 06:32:29 DEBUG } 06:32:29 DEBUG ] 06:32:29 DEBUG } 06:32:29 DEBUG } 06:32:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:29 ERROR --- status code --- 06:32:29 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:32:29 ERROR --- http response --- 06:32:29 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:32:29 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:32:29 DEBUG --- http response --- 06:32:29 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:32:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727150" 06:32:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727150" 06:32:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727150" 06:32:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727150" 06:32:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727150" 06:32:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727150" 06:32:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727150" 06:32:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727150" 06:32:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727150" 06:32:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727150" 06:32:30 INFO [http_cmd]: http status code OK 06:32:30 DEBUG --- status code --- 06:32:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:30 DEBUG --- http response --- 06:32:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:32:30 DEBUG ----- output ----- 06:32:30 DEBUG { 06:32:30 DEBUG "status": "success", 06:32:30 DEBUG "data": { 06:32:30 DEBUG "resultType": "vector", 06:32:30 DEBUG "result": [ 06:32:30 DEBUG { 06:32:30 DEBUG "metric": { 06:32:30 DEBUG "instance": "10.142.0.25:9100" 06:32:30 DEBUG }, 06:32:30 DEBUG "value": [ 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG [...] 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG "instance": "10.142.0.77:9100" 06:32:30 DEBUG }, 06:32:30 DEBUG "value": [ 06:32:30 DEBUG 1712727150, 06:32:30 DEBUG "42553534.766666666" 06:32:30 DEBUG ] 06:32:30 DEBUG } 06:32:30 DEBUG ] 06:32:30 DEBUG } 06:32:30 DEBUG } 06:32:30 INFO [http_cmd]: http status code OK 06:32:30 DEBUG --- status code --- 06:32:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:30 DEBUG --- http response --- 06:32:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:32:30 DEBUG ----- output ----- 06:32:30 DEBUG { 06:32:30 DEBUG "status": "success", 06:32:30 DEBUG "data": { 06:32:30 DEBUG "resultType": "vector", 06:32:30 DEBUG "result": [ 06:32:30 DEBUG { 06:32:30 DEBUG "metric": { 06:32:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:32:30 DEBUG }, 06:32:30 DEBUG "value": [ 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG [...] 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:32:30 DEBUG }, 06:32:30 DEBUG "value": [ 06:32:30 DEBUG 1712727150, 06:32:30 DEBUG "1436.7666666666667" 06:32:30 DEBUG ] 06:32:30 DEBUG } 06:32:30 DEBUG ] 06:32:30 DEBUG } 06:32:30 DEBUG } 06:32:30 INFO [http_cmd]: http status code OK 06:32:30 DEBUG --- status code --- 06:32:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:30 DEBUG --- http response --- 06:32:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:32:30 DEBUG ----- output ----- 06:32:30 DEBUG { 06:32:30 DEBUG "status": "success", 06:32:30 DEBUG "data": { 06:32:30 DEBUG "resultType": "vector", 06:32:30 DEBUG "result": [ 06:32:30 DEBUG { 06:32:30 DEBUG "metric": { 06:32:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:32:30 DEBUG }, 06:32:30 DEBUG "value": [ 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG [...] 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:32:30 DEBUG }, 06:32:30 DEBUG "value": [ 06:32:30 DEBUG 1712727150, 06:32:30 DEBUG "1436.6" 06:32:30 DEBUG ] 06:32:30 DEBUG } 06:32:30 DEBUG ] 06:32:30 DEBUG } 06:32:30 DEBUG } 06:32:30 INFO [http_cmd]: http status code OK 06:32:30 DEBUG --- status code --- 06:32:30 INFO [http_cmd]: http status code OK 06:32:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:30 DEBUG --- status code --- 06:32:30 DEBUG --- http response --- 06:32:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:32:30 DEBUG --- http response --- 06:32:30 DEBUG ----- output ----- 06:32:30 DEBUG { 06:32:30 DEBUG "status": "success", 06:32:30 DEBUG "data": { 06:32:30 DEBUG "resultType": "vector", 06:32:30 DEBUG "result": [ 06:32:30 DEBUG { 06:32:30 DEBUG "metric": { 06:32:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:32:30 DEBUG }, 06:32:30 DEBUG "value": [ 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:32:30 DEBUG ----- output ----- 06:32:30 DEBUG { 06:32:30 DEBUG "status": "success", 06:32:30 DEBUG "data": { 06:32:30 DEBUG "resultType": "vector", 06:32:30 DEBUG "result": [ 06:32:30 DEBUG { 06:32:30 DEBUG "metric": { 06:32:30 DEBUG "pod": "ds-idrepo-0" 06:32:30 DEBUG }, 06:32:30 DEBUG "value": [ 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG [...] 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG "pod": "ds-idrepo-1" 06:32:30 DEBUG }, 06:32:30 DEBUG "value": [ 06:32:30 DEBUG 1712727150, 06:32:30 DEBUG "0" 06:32:30 DEBUG ] 06:32:30 DEBUG } 06:32:30 DEBUG ] 06:32:30 DEBUG } 06:32:30 DEBUG } 06:32:30 DEBUG 06:32:30 DEBUG [...] 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:32:30 DEBUG }, 06:32:30 DEBUG "value": [ 06:32:30 DEBUG 1712727150, 06:32:30 DEBUG "0" 06:32:30 DEBUG ] 06:32:30 DEBUG } 06:32:30 DEBUG ] 06:32:30 DEBUG } 06:32:30 DEBUG } 06:32:30 INFO [http_cmd]: http status code OK 06:32:30 INFO [http_cmd]: http status code OK 06:32:30 INFO [http_cmd]: http status code OK 06:32:30 DEBUG --- status code --- 06:32:30 INFO [http_cmd]: http status code OK 06:32:30 DEBUG --- status code --- 06:32:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:30 DEBUG --- status code --- 06:32:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:30 DEBUG --- http response --- 06:32:30 DEBUG --- http response --- 06:32:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:32:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:30 DEBUG --- http response --- 06:32:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:32:30 DEBUG --- status code --- 06:32:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:30 DEBUG --- http response --- 06:32:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:32:30 DEBUG ----- output ----- 06:32:30 INFO [http_cmd]: http status code OK 06:32:30 DEBUG --- status code --- 06:32:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:32:30 DEBUG ----- output ----- 06:32:30 DEBUG { 06:32:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:30 DEBUG ----- output ----- 06:32:30 DEBUG { 06:32:30 DEBUG "status": "success", 06:32:30 DEBUG { 06:32:30 DEBUG "status": "success", 06:32:30 DEBUG "data": { 06:32:30 DEBUG "status": "success", 06:32:30 DEBUG "data": { 06:32:30 DEBUG --- http response --- 06:32:30 DEBUG ----- output ----- 06:32:30 DEBUG "resultType": "vector", 06:32:30 DEBUG "result": [ 06:32:30 DEBUG "resultType": "vector", 06:32:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:32:30 DEBUG { 06:32:30 DEBUG "status": "success", 06:32:30 DEBUG { 06:32:30 DEBUG "result": [ 06:32:30 DEBUG ----- output ----- 06:32:30 DEBUG { 06:32:30 DEBUG "status": "success", 06:32:30 DEBUG "data": { 06:32:30 DEBUG { 06:32:30 DEBUG "data": { 06:32:30 DEBUG "data": { 06:32:30 DEBUG "metric": { 06:32:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:32:30 DEBUG "metric": { 06:32:30 DEBUG "instance": "10.142.0.25:9100" 06:32:30 DEBUG }, 06:32:30 DEBUG "value": [ 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG [...] 06:32:30 DEBUG "resultType": "vector", 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG "instance": "10.142.0.77:9100" 06:32:30 DEBUG "result": [ 06:32:30 DEBUG }, 06:32:30 DEBUG "value": [ 06:32:30 DEBUG 1712727150, 06:32:30 DEBUG "0.15366666666666523" 06:32:30 DEBUG { 06:32:30 DEBUG ] 06:32:30 DEBUG } 06:32:30 DEBUG ] 06:32:30 DEBUG } 06:32:30 DEBUG "metric": { 06:32:30 DEBUG } 06:32:30 DEBUG "resultType": "vector", 06:32:30 DEBUG "result": [ 06:32:30 DEBUG }, 06:32:30 DEBUG "value": [ 06:32:30 DEBUG 06:32:30 DEBUG "resultType": "vector", 06:32:30 DEBUG "instance": "10.142.0.25:9100" 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG [...] 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:32:30 DEBUG }, 06:32:30 DEBUG "value": [ 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG [...] 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG "instance": "10.142.0.77:9100" 06:32:30 DEBUG }, 06:32:30 DEBUG { 06:32:30 DEBUG "metric": { 06:32:30 DEBUG "value": [ 06:32:30 DEBUG 1712727150, 06:32:30 DEBUG "406497500160" 06:32:30 DEBUG ] 06:32:30 DEBUG } 06:32:30 DEBUG ] 06:32:30 DEBUG } 06:32:30 DEBUG } 06:32:30 DEBUG "result": [ 06:32:30 DEBUG }, 06:32:30 DEBUG "value": [ 06:32:30 DEBUG 1712727150, 06:32:30 DEBUG "258272" 06:32:30 DEBUG ] 06:32:30 DEBUG "pod": "ds-idrepo-0" 06:32:30 DEBUG }, 06:32:30 DEBUG } 06:32:30 DEBUG ] 06:32:30 DEBUG } 06:32:30 DEBUG } 06:32:30 DEBUG { 06:32:30 DEBUG "metric": { 06:32:30 DEBUG "pod": "ds-idrepo-0" 06:32:30 DEBUG }, 06:32:30 DEBUG "value": [ 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG [...] 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG "pod": "ds-cts-2" 06:32:30 DEBUG }, 06:32:30 DEBUG "value": [ 06:32:30 DEBUG 1712727150, 06:32:30 DEBUG "0" 06:32:30 DEBUG ] 06:32:30 DEBUG } 06:32:30 DEBUG ] 06:32:30 DEBUG } 06:32:30 DEBUG } 06:32:30 DEBUG "value": [ 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG [...] 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG "pod": "ds-cts-2" 06:32:30 DEBUG }, 06:32:30 DEBUG "value": [ 06:32:30 DEBUG 1712727150, 06:32:30 DEBUG "29.66963363002967" 06:32:30 DEBUG ] 06:32:30 DEBUG } 06:32:30 DEBUG ] 06:32:30 DEBUG } 06:32:30 DEBUG } 06:32:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727150" 06:32:30 INFO [http_cmd]: http status code OK 06:32:30 DEBUG --- status code --- 06:32:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:30 DEBUG --- http response --- 06:32:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:32:30 DEBUG ----- output ----- 06:32:30 DEBUG { 06:32:30 DEBUG "status": "success", 06:32:30 DEBUG "data": { 06:32:30 DEBUG "resultType": "vector", 06:32:30 DEBUG "result": [ 06:32:30 DEBUG { 06:32:30 DEBUG "metric": { 06:32:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:32:30 DEBUG }, 06:32:30 DEBUG "value": [ 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG [...] 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG 06:32:30 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:32:30 DEBUG }, 06:32:30 DEBUG "value": [ 06:32:30 DEBUG 1712727150, 06:32:30 DEBUG "87751133.89281611" 06:32:30 DEBUG ] 06:32:30 DEBUG } 06:32:30 DEBUG ] 06:32:30 DEBUG } 06:32:30 DEBUG } 06:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712727151" 06:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712727151" 06:32:31 INFO [http_cmd]: http status code OK 06:32:31 DEBUG --- status code --- 06:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:31 DEBUG --- http response --- 06:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:32:31 DEBUG ----- output ----- 06:32:31 DEBUG { 06:32:31 DEBUG "status": "success", 06:32:31 DEBUG "data": { 06:32:31 DEBUG "resultType": "vector", 06:32:31 DEBUG "result": [ 06:32:31 DEBUG { 06:32:31 DEBUG "metric": { 06:32:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:32:31 DEBUG }, 06:32:31 DEBUG "value": [ 06:32:31 DEBUG 06:32:31 DEBUG 06:32:31 DEBUG 06:32:31 DEBUG [...] 06:32:31 DEBUG 06:32:31 DEBUG 06:32:31 DEBUG 06:32:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:32:31 DEBUG }, 06:32:31 DEBUG "value": [ 06:32:31 DEBUG 1712727151, 06:32:31 DEBUG "4.398379639853715" 06:32:31 DEBUG ] 06:32:31 DEBUG } 06:32:31 DEBUG ] 06:32:31 DEBUG } 06:32:31 DEBUG } 06:32:31 INFO [http_cmd]: http status code OK 06:32:31 DEBUG --- status code --- 06:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:31 DEBUG --- http response --- 06:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:32:31 DEBUG ----- output ----- 06:32:31 DEBUG { 06:32:31 DEBUG "status": "success", 06:32:31 DEBUG "data": { 06:32:31 DEBUG "resultType": "vector", 06:32:31 DEBUG "result": [ 06:32:31 DEBUG { 06:32:31 DEBUG "metric": { 06:32:31 DEBUG "pod": "ds-idrepo-0" 06:32:31 DEBUG }, 06:32:31 DEBUG "value": [ 06:32:31 DEBUG 06:32:31 DEBUG 06:32:31 DEBUG 06:32:31 DEBUG [...] 06:32:31 DEBUG 06:32:31 DEBUG 06:32:31 DEBUG 06:32:31 DEBUG "pod": "ds-cts-2" 06:32:31 DEBUG }, 06:32:31 DEBUG "value": [ 06:32:31 DEBUG 1712727151, 06:32:31 DEBUG "0" 06:32:31 DEBUG ] 06:32:31 DEBUG } 06:32:31 DEBUG ] 06:32:31 DEBUG } 06:32:31 DEBUG } 06:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712727151" 06:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727151" 06:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727151" 06:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727151" 06:32:31 INFO [http_cmd]: http status code OK 06:32:31 DEBUG --- status code --- 06:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:31 DEBUG --- http response --- 06:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:32:31 DEBUG ----- output ----- 06:32:31 DEBUG { 06:32:31 DEBUG "status": "success", 06:32:31 DEBUG "data": { 06:32:31 DEBUG "resultType": "vector", 06:32:31 DEBUG "result": [ 06:32:31 DEBUG { 06:32:31 DEBUG "metric": { 06:32:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:32:31 DEBUG }, 06:32:31 DEBUG "value": [ 06:32:31 DEBUG 06:32:31 DEBUG 06:32:31 DEBUG 06:32:31 DEBUG [...] 06:32:31 DEBUG 06:32:31 INFO [http_cmd]: http status code OK 06:32:31 DEBUG --- status code --- 06:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:31 DEBUG --- http response --- 06:32:31 DEBUG 06:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:32:31 DEBUG 06:32:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:32:31 DEBUG }, 06:32:31 DEBUG "value": [ 06:32:31 DEBUG 1712727151, 06:32:31 DEBUG "0.051" 06:32:31 DEBUG ] 06:32:31 DEBUG } 06:32:31 DEBUG ] 06:32:31 DEBUG } 06:32:31 DEBUG } 06:32:31 DEBUG ----- output ----- 06:32:31 DEBUG { 06:32:31 DEBUG "status": "success", 06:32:31 DEBUG "data": { 06:32:31 DEBUG "resultType": "vector", 06:32:31 DEBUG "result": [ 06:32:31 DEBUG { 06:32:31 DEBUG "metric": { 06:32:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:32:31 DEBUG }, 06:32:31 DEBUG "value": [ 06:32:31 DEBUG 06:32:31 DEBUG 06:32:31 DEBUG 06:32:31 DEBUG [...] 06:32:31 DEBUG 06:32:31 DEBUG 06:32:31 DEBUG 06:32:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:32:31 DEBUG }, 06:32:31 DEBUG "value": [ 06:32:31 DEBUG 1712727151, 06:32:31 DEBUG "1436.3333333333333" 06:32:31 DEBUG ] 06:32:31 DEBUG } 06:32:31 INFO [http_cmd]: http status code OK 06:32:31 DEBUG --- status code --- 06:32:31 INFO [http_cmd]: http status code OK 06:32:31 DEBUG ] 06:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:31 DEBUG --- status code --- 06:32:31 DEBUG } 06:32:31 DEBUG --- http response --- 06:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:31 DEBUG } 06:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:32:31 DEBUG --- http response --- 06:32:31 DEBUG ----- output ----- 06:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:32:31 DEBUG { 06:32:31 DEBUG ----- output ----- 06:32:31 DEBUG "status": "success", 06:32:31 DEBUG { 06:32:31 DEBUG "data": { 06:32:31 DEBUG "status": "success", 06:32:31 DEBUG "resultType": "vector", 06:32:31 DEBUG "data": { 06:32:31 DEBUG "result": [ 06:32:31 DEBUG "resultType": "vector", 06:32:31 DEBUG { 06:32:31 DEBUG "result": [ 06:32:31 DEBUG { 06:32:31 DEBUG "metric": { 06:32:31 DEBUG "metric": { 06:32:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:32:31 DEBUG "pod": "ds-idrepo-2" 06:32:31 DEBUG }, 06:32:31 DEBUG }, 06:32:31 DEBUG "value": [ 06:32:31 DEBUG "value": [ 06:32:31 DEBUG 06:32:31 DEBUG 06:32:31 DEBUG 06:32:31 DEBUG 06:32:31 DEBUG 06:32:31 DEBUG [...] 06:32:31 DEBUG 06:32:31 DEBUG [...] 06:32:31 DEBUG 06:32:31 DEBUG 06:32:31 DEBUG 06:32:31 DEBUG "pod": "ds-cts-2" 06:32:31 DEBUG }, 06:32:31 DEBUG "value": [ 06:32:31 DEBUG 1712727151, 06:32:31 DEBUG "0.00048185040726294844" 06:32:31 DEBUG ] 06:32:31 DEBUG } 06:32:31 DEBUG ] 06:32:31 DEBUG } 06:32:31 DEBUG } 06:32:31 DEBUG 06:32:31 DEBUG 06:32:31 DEBUG 06:32:31 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:32:31 DEBUG }, 06:32:31 DEBUG "value": [ 06:32:31 DEBUG 1712727151, 06:32:31 DEBUG "4.344529687328107" 06:32:31 DEBUG ] 06:32:31 DEBUG } 06:32:31 DEBUG ] 06:32:31 DEBUG } 06:32:31 DEBUG } 06:32:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727151" 06:32:31 INFO [http_cmd]: http status code OK 06:32:31 DEBUG --- status code --- 06:32:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:31 DEBUG --- http response --- 06:32:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:32:31 DEBUG ----- output ----- 06:32:31 DEBUG { 06:32:31 DEBUG "status": "success", 06:32:31 DEBUG "data": { 06:32:31 DEBUG "resultType": "vector", 06:32:31 DEBUG "result": [ 06:32:31 DEBUG { 06:32:31 DEBUG "metric": { 06:32:31 DEBUG "pod": "ds-idrepo-0" 06:32:31 DEBUG }, 06:32:31 DEBUG "value": [ 06:32:31 DEBUG 06:32:31 DEBUG 06:32:31 DEBUG 06:32:31 DEBUG [...] 06:32:31 DEBUG 06:32:31 DEBUG 06:32:31 DEBUG 06:32:31 DEBUG "pod": "ds-idrepo-1" 06:32:31 DEBUG }, 06:32:31 DEBUG "value": [ 06:32:31 DEBUG 1712727151, 06:32:31 DEBUG "0" 06:32:31 DEBUG ] 06:32:31 DEBUG } 06:32:31 DEBUG ] 06:32:31 DEBUG } 06:32:31 DEBUG } 06:32:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727152" 06:32:32 INFO [http_cmd]: http status code OK 06:32:32 DEBUG --- status code --- 06:32:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:32 DEBUG --- http response --- 06:32:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:32:32 DEBUG ----- output ----- 06:32:32 DEBUG { 06:32:32 DEBUG "status": "success", 06:32:32 DEBUG "data": { 06:32:32 DEBUG "resultType": "vector", 06:32:32 DEBUG "result": [ 06:32:32 DEBUG { 06:32:32 DEBUG "metric": { 06:32:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:32:32 DEBUG }, 06:32:32 DEBUG "value": [ 06:32:32 DEBUG 06:32:32 DEBUG 06:32:32 DEBUG 06:32:32 DEBUG [...] 06:32:32 DEBUG 06:32:32 DEBUG 06:32:32 DEBUG 06:32:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:32:32 DEBUG }, 06:32:32 DEBUG "value": [ 06:32:32 DEBUG 1712727152, 06:32:32 DEBUG "1460.8666666666666" 06:32:32 DEBUG ] 06:32:32 DEBUG } 06:32:32 DEBUG ] 06:32:32 DEBUG } 06:32:32 DEBUG } 06:32:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727152" 06:32:32 INFO [http_cmd]: http status code OK 06:32:32 DEBUG --- status code --- 06:32:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:32 DEBUG --- http response --- 06:32:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:32:32 DEBUG ----- output ----- 06:32:32 DEBUG { 06:32:32 DEBUG "status": "success", 06:32:32 DEBUG "data": { 06:32:32 DEBUG "resultType": "vector", 06:32:32 DEBUG "result": [ 06:32:32 DEBUG { 06:32:32 DEBUG "metric": { 06:32:32 DEBUG "pod": "ds-idrepo-0" 06:32:32 DEBUG }, 06:32:32 DEBUG "value": [ 06:32:32 DEBUG 06:32:32 DEBUG 06:32:32 DEBUG 06:32:32 DEBUG [...] 06:32:32 DEBUG 06:32:32 DEBUG 06:32:32 DEBUG 06:32:32 DEBUG "pod": "ds-idrepo-1" 06:32:32 DEBUG }, 06:32:32 DEBUG "value": [ 06:32:32 DEBUG 1712727152, 06:32:32 DEBUG "0" 06:32:32 DEBUG ] 06:32:32 DEBUG } 06:32:32 DEBUG ] 06:32:32 DEBUG } 06:32:32 DEBUG } 06:32:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727152" 06:32:32 INFO [http_cmd]: http status code OK 06:32:32 DEBUG --- status code --- 06:32:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:32 DEBUG --- http response --- 06:32:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:32:32 DEBUG ----- output ----- 06:32:32 DEBUG { 06:32:32 DEBUG "status": "success", 06:32:32 DEBUG "data": { 06:32:32 DEBUG "resultType": "vector", 06:32:32 DEBUG "result": [ 06:32:32 DEBUG { 06:32:32 DEBUG "metric": { 06:32:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:32:32 DEBUG }, 06:32:32 DEBUG "value": [ 06:32:32 DEBUG 06:32:32 DEBUG 06:32:32 DEBUG 06:32:32 DEBUG [...] 06:32:32 DEBUG 06:32:32 DEBUG 06:32:32 DEBUG 06:32:32 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:32:32 DEBUG }, 06:32:32 DEBUG "value": [ 06:32:32 DEBUG 1712727152, 06:32:32 DEBUG "0" 06:32:32 DEBUG ] 06:32:32 DEBUG } 06:32:32 DEBUG ] 06:32:32 DEBUG } 06:32:32 DEBUG } 06:32:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712727152" 06:32:32 INFO [http_cmd]: http status code OK 06:32:32 DEBUG --- status code --- 06:32:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:32 DEBUG --- http response --- 06:32:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:32:32 DEBUG ----- output ----- 06:32:32 DEBUG { 06:32:32 DEBUG "status": "success", 06:32:32 DEBUG "data": { 06:32:32 DEBUG "resultType": "vector", 06:32:32 DEBUG "result": [ 06:32:32 DEBUG { 06:32:32 DEBUG "metric": { 06:32:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:32:32 DEBUG }, 06:32:32 DEBUG "value": [ 06:32:32 DEBUG 06:32:32 DEBUG 06:32:32 DEBUG 06:32:32 DEBUG [...] 06:32:32 DEBUG 06:32:32 DEBUG 06:32:32 DEBUG 06:32:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:32:32 DEBUG }, 06:32:32 DEBUG "value": [ 06:32:32 DEBUG 1712727152, 06:32:32 DEBUG "2485874688" 06:32:32 DEBUG ] 06:32:32 DEBUG } 06:32:32 DEBUG ] 06:32:32 DEBUG } 06:32:32 DEBUG } 06:32:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727152" 06:32:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727152" 06:32:32 INFO [http_cmd]: http status code OK 06:32:32 DEBUG --- status code --- 06:32:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:32 DEBUG --- http response --- 06:32:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:32:32 DEBUG ----- output ----- 06:32:32 DEBUG { 06:32:32 DEBUG "status": "success", 06:32:32 DEBUG "data": { 06:32:32 DEBUG "resultType": "vector", 06:32:32 DEBUG "result": [ 06:32:32 DEBUG { 06:32:32 DEBUG "metric": { 06:32:32 DEBUG "instance": "10.142.0.25:9100" 06:32:32 DEBUG }, 06:32:32 DEBUG "value": [ 06:32:32 DEBUG 06:32:32 DEBUG 06:32:32 DEBUG 06:32:32 DEBUG [...] 06:32:32 DEBUG 06:32:32 DEBUG 06:32:32 DEBUG 06:32:32 DEBUG "instance": "10.142.0.77:9100" 06:32:32 DEBUG }, 06:32:32 DEBUG "value": [ 06:32:32 DEBUG 1712727152, 06:32:32 DEBUG "42557343.36666667" 06:32:32 DEBUG ] 06:32:32 DEBUG } 06:32:32 DEBUG ] 06:32:32 DEBUG } 06:32:32 DEBUG } 06:32:32 INFO [http_cmd]: http status code OK 06:32:32 DEBUG --- status code --- 06:32:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:32 DEBUG --- http response --- 06:32:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:32:32 DEBUG ----- output ----- 06:32:32 DEBUG { 06:32:32 DEBUG "status": "success", 06:32:32 DEBUG "data": { 06:32:32 DEBUG "resultType": "vector", 06:32:32 DEBUG "result": [ 06:32:32 DEBUG { 06:32:32 DEBUG "metric": { 06:32:32 DEBUG "instance": "10.142.0.25:9100" 06:32:32 DEBUG }, 06:32:32 DEBUG "value": [ 06:32:32 DEBUG 06:32:32 DEBUG 06:32:32 DEBUG 06:32:32 DEBUG [...] 06:32:32 DEBUG 06:32:32 DEBUG 06:32:32 DEBUG 06:32:32 DEBUG "instance": "10.142.0.77:9100" 06:32:32 DEBUG }, 06:32:32 DEBUG "value": [ 06:32:32 DEBUG 1712727152, 06:32:32 DEBUG "0.05100000000000003" 06:32:32 DEBUG ] 06:32:32 DEBUG } 06:32:32 DEBUG ] 06:32:32 DEBUG } 06:32:32 DEBUG } 06:32:51 INFO 06:32:51 INFO [loop_until]: kubectl --namespace=xlou top pods 06:32:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:32:51 INFO [loop_until]: OK (rc = 0) 06:32:51 DEBUG --- stdout --- 06:32:51 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10808m 9041Mi am-654c4fcbcf-9scrj 9766m 8464Mi am-654c4fcbcf-v5qb5 8482m 8492Mi ds-cts-0 8493m 13393Mi ds-cts-1 8641m 13396Mi ds-cts-2 7584m 13394Mi ds-idrepo-0 10827m 5623Mi ds-idrepo-1 393m 5673Mi ds-idrepo-2 373m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1191Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 2m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4421m 2402Mi 06:32:51 DEBUG --- stderr --- 06:32:51 DEBUG 06:33:00 INFO 06:33:00 INFO [loop_until]: kubectl --namespace=xlou top node 06:33:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:33:00 INFO [loop_until]: OK (rc = 0) 06:33:00 DEBUG --- stdout --- 06:33:00 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 94m 0% 2304Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9212m 57% 9364Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 80m 0% 3256Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 14177m 89% 6813Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10592m 66% 9344Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11944m 75% 9827Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 79m 0% 852Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10886m 68% 6305Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8610m 54% 14090Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 7756m 48% 14076Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 428m 2% 6392Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8450m 53% 14099Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 456m 2% 6357Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4480m 28% 5987Mi 10% 06:33:00 DEBUG --- stderr --- 06:33:00 DEBUG 06:33:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727209" 06:33:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712727209" 06:33:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727209" 06:33:29 ERROR [http_cmd]: ERROR 06:33:29 ERROR --- status code --- 06:33:29 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:33:29 ERROR --- http response --- 06:33:29 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:33:29 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:33:29 INFO [http_cmd]: http status code OK 06:33:29 INFO [http_cmd]: http status code OK 06:33:29 DEBUG --- status code --- 06:33:29 DEBUG --- status code --- 06:33:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:29 DEBUG --- http response --- 06:33:29 DEBUG --- http response --- 06:33:29 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:33:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:33:29 DEBUG ----- output ----- 06:33:29 DEBUG { 06:33:29 DEBUG "status": "success", 06:33:29 DEBUG "data": { 06:33:29 DEBUG "resultType": "vector", 06:33:29 DEBUG "result": [ 06:33:29 DEBUG { 06:33:29 DEBUG "metric": { 06:33:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:33:29 DEBUG }, 06:33:29 DEBUG "value": [ 06:33:29 DEBUG 06:33:29 DEBUG 06:33:29 DEBUG 06:33:29 DEBUG [...] 06:33:29 DEBUG 06:33:29 DEBUG 06:33:29 DEBUG 06:33:29 DEBUG "pod": "am-654c4fcbcf-487cv" 06:33:29 DEBUG }, 06:33:29 DEBUG "value": [ 06:33:29 DEBUG 1712727209, 06:33:29 DEBUG "0" 06:33:29 DEBUG ] 06:33:29 DEBUG } 06:33:29 DEBUG ] 06:33:29 DEBUG } 06:33:29 DEBUG } 06:33:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727210" 06:33:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727210" 06:33:30 INFO [http_cmd]: http status code OK 06:33:30 DEBUG --- status code --- 06:33:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:30 DEBUG --- http response --- 06:33:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:33:30 DEBUG ----- output ----- 06:33:30 DEBUG { 06:33:30 DEBUG "status": "success", 06:33:30 DEBUG "data": { 06:33:30 DEBUG "resultType": "vector", 06:33:30 DEBUG "result": [ 06:33:30 DEBUG { 06:33:30 DEBUG "metric": { 06:33:30 DEBUG "instance": "10.142.0.25:9100" 06:33:30 DEBUG }, 06:33:30 DEBUG "value": [ 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG [...] 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG "instance": "10.142.0.77:9100" 06:33:30 DEBUG }, 06:33:30 DEBUG "value": [ 06:33:30 DEBUG 1712727210, 06:33:30 DEBUG "42508058.2" 06:33:30 DEBUG ] 06:33:30 DEBUG } 06:33:30 DEBUG ] 06:33:30 DEBUG } 06:33:30 DEBUG } 06:33:30 INFO [http_cmd]: http status code OK 06:33:30 DEBUG --- status code --- 06:33:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:30 DEBUG --- http response --- 06:33:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:33:30 DEBUG ----- output ----- 06:33:30 DEBUG { 06:33:30 DEBUG "status": "success", 06:33:30 DEBUG "data": { 06:33:30 DEBUG "resultType": "vector", 06:33:30 DEBUG "result": [ 06:33:30 DEBUG { 06:33:30 DEBUG "metric": { 06:33:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:33:30 DEBUG }, 06:33:30 DEBUG "value": [ 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG [...] 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:33:30 DEBUG }, 06:33:30 DEBUG "value": [ 06:33:30 DEBUG 1712727210, 06:33:30 DEBUG "1443.7666666666667" 06:33:30 DEBUG ] 06:33:30 DEBUG } 06:33:30 DEBUG ] 06:33:30 DEBUG } 06:33:30 DEBUG } 06:33:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727210" 06:33:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727210" 06:33:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727210" 06:33:30 INFO [http_cmd]: http status code OK 06:33:30 DEBUG --- status code --- 06:33:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:30 DEBUG --- http response --- 06:33:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:33:30 DEBUG ----- output ----- 06:33:30 DEBUG { 06:33:30 DEBUG "status": "success", 06:33:30 DEBUG "data": { 06:33:30 DEBUG "resultType": "vector", 06:33:30 DEBUG "result": [ 06:33:30 DEBUG { 06:33:30 DEBUG "metric": { 06:33:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:33:30 DEBUG }, 06:33:30 DEBUG "value": [ 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG [...] 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:33:30 DEBUG }, 06:33:30 DEBUG "value": [ 06:33:30 DEBUG 1712727210, 06:33:30 DEBUG "1443.5666666666666" 06:33:30 DEBUG ] 06:33:30 DEBUG } 06:33:30 DEBUG ] 06:33:30 DEBUG } 06:33:30 DEBUG } 06:33:30 INFO [http_cmd]: http status code OK 06:33:30 DEBUG --- status code --- 06:33:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:30 DEBUG --- http response --- 06:33:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:33:30 DEBUG ----- output ----- 06:33:30 DEBUG { 06:33:30 DEBUG "status": "success", 06:33:30 DEBUG "data": { 06:33:30 DEBUG "resultType": "vector", 06:33:30 DEBUG "result": [ 06:33:30 DEBUG { 06:33:30 DEBUG "metric": { 06:33:30 DEBUG "pod": "ds-idrepo-0" 06:33:30 DEBUG }, 06:33:30 DEBUG "value": [ 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG [...] 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG "pod": "ds-idrepo-1" 06:33:30 DEBUG }, 06:33:30 DEBUG "value": [ 06:33:30 DEBUG 1712727210, 06:33:30 DEBUG "0" 06:33:30 DEBUG ] 06:33:30 DEBUG } 06:33:30 DEBUG ] 06:33:30 DEBUG } 06:33:30 DEBUG } 06:33:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727210" 06:33:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727210" 06:33:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727210" 06:33:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727210" 06:33:30 INFO [http_cmd]: http status code OK 06:33:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727210" 06:33:30 DEBUG --- status code --- 06:33:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:30 DEBUG --- http response --- 06:33:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:33:30 DEBUG ----- output ----- 06:33:30 DEBUG { 06:33:30 DEBUG "status": "success", 06:33:30 DEBUG "data": { 06:33:30 DEBUG "resultType": "vector", 06:33:30 DEBUG "result": [ 06:33:30 DEBUG { 06:33:30 DEBUG "metric": { 06:33:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:33:30 DEBUG }, 06:33:30 DEBUG "value": [ 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG [...] 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:33:30 DEBUG }, 06:33:30 DEBUG "value": [ 06:33:30 DEBUG 1712727210, 06:33:30 DEBUG "0" 06:33:30 DEBUG ] 06:33:30 DEBUG } 06:33:30 DEBUG ] 06:33:30 DEBUG } 06:33:30 DEBUG } 06:33:30 INFO [http_cmd]: http status code OK 06:33:30 INFO [http_cmd]: http status code OK 06:33:30 DEBUG --- status code --- 06:33:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:30 INFO [http_cmd]: http status code OK 06:33:30 DEBUG --- status code --- 06:33:30 DEBUG --- http response --- 06:33:30 DEBUG --- status code --- 06:33:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:30 DEBUG --- http response --- 06:33:30 INFO [http_cmd]: http status code OK 06:33:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:30 DEBUG --- http response --- 06:33:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:33:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:33:30 DEBUG --- status code --- 06:33:30 INFO [http_cmd]: http status code OK 06:33:30 DEBUG --- status code --- 06:33:30 DEBUG ----- output ----- 06:33:30 DEBUG ----- output ----- 06:33:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:33:30 DEBUG { 06:33:30 DEBUG { 06:33:30 DEBUG --- http response --- 06:33:30 DEBUG --- http response --- 06:33:30 DEBUG ----- output ----- 06:33:30 DEBUG { 06:33:30 DEBUG "status": "success", 06:33:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:33:30 DEBUG "status": "success", 06:33:30 DEBUG "status": "success", 06:33:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:33:30 DEBUG ----- output ----- 06:33:30 DEBUG ----- output ----- 06:33:30 DEBUG "data": { 06:33:30 DEBUG "data": { 06:33:30 DEBUG { 06:33:30 DEBUG "data": { 06:33:30 DEBUG { 06:33:30 DEBUG "resultType": "vector", 06:33:30 DEBUG "resultType": "vector", 06:33:30 DEBUG "status": "success", 06:33:30 DEBUG "resultType": "vector", 06:33:30 DEBUG "status": "success", 06:33:30 DEBUG "result": [ 06:33:30 DEBUG { 06:33:30 DEBUG "data": { 06:33:30 DEBUG "result": [ 06:33:30 DEBUG { 06:33:30 DEBUG "metric": { 06:33:30 DEBUG "result": [ 06:33:30 DEBUG "resultType": "vector", 06:33:30 DEBUG "metric": { 06:33:30 DEBUG "data": { 06:33:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:33:30 DEBUG { 06:33:30 DEBUG "result": [ 06:33:30 DEBUG "instance": "10.142.0.25:9100" 06:33:30 DEBUG "resultType": "vector", 06:33:30 DEBUG }, 06:33:30 DEBUG "metric": { 06:33:30 DEBUG { 06:33:30 DEBUG }, 06:33:30 DEBUG "result": [ 06:33:30 DEBUG "value": [ 06:33:30 DEBUG "instance": "10.142.0.25:9100" 06:33:30 DEBUG "metric": { 06:33:30 DEBUG "value": [ 06:33:30 DEBUG { 06:33:30 DEBUG 06:33:30 DEBUG }, 06:33:30 DEBUG "pod": "ds-idrepo-0" 06:33:30 DEBUG 06:33:30 DEBUG "metric": { 06:33:30 DEBUG 06:33:30 DEBUG "value": [ 06:33:30 DEBUG }, 06:33:30 DEBUG 06:33:30 DEBUG "pod": "ds-idrepo-0" 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG "value": [ 06:33:30 DEBUG 06:33:30 DEBUG }, 06:33:30 DEBUG [...] 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG [...] 06:33:30 DEBUG "value": [ 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG [...] 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG 06:33:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:33:30 DEBUG [...] 06:33:30 DEBUG 06:33:30 DEBUG "instance": "10.142.0.77:9100" 06:33:30 DEBUG [...] 06:33:30 DEBUG }, 06:33:30 DEBUG 06:33:30 DEBUG "instance": "10.142.0.77:9100" 06:33:30 DEBUG }, 06:33:30 DEBUG 06:33:30 DEBUG "value": [ 06:33:30 DEBUG 06:33:30 DEBUG }, 06:33:30 DEBUG "value": [ 06:33:30 DEBUG 06:33:30 DEBUG 1712727210, 06:33:30 DEBUG 06:33:30 DEBUG "value": [ 06:33:30 DEBUG 1712727210, 06:33:30 DEBUG 06:33:30 DEBUG "257581.5" 06:33:30 DEBUG "pod": "ds-cts-2" 06:33:30 DEBUG 1712727210, 06:33:30 DEBUG "0.09519999999999947" 06:33:30 DEBUG "pod": "ds-cts-2" 06:33:30 DEBUG ] 06:33:30 DEBUG }, 06:33:30 DEBUG "408794668032" 06:33:30 DEBUG ] 06:33:30 DEBUG }, 06:33:30 DEBUG } 06:33:30 DEBUG "value": [ 06:33:30 DEBUG ] 06:33:30 DEBUG } 06:33:30 DEBUG "value": [ 06:33:30 DEBUG ] 06:33:30 DEBUG 1712727210, 06:33:30 DEBUG ] 06:33:30 DEBUG } 06:33:30 DEBUG 1712727210, 06:33:30 DEBUG } 06:33:30 DEBUG "29.7" 06:33:30 DEBUG } 06:33:30 DEBUG } 06:33:30 DEBUG "0" 06:33:30 DEBUG } 06:33:30 DEBUG ] 06:33:30 DEBUG ] 06:33:30 DEBUG ] 06:33:30 DEBUG } 06:33:30 DEBUG } 06:33:30 DEBUG } 06:33:30 DEBUG ] 06:33:30 DEBUG } 06:33:30 DEBUG ] 06:33:30 DEBUG } 06:33:30 DEBUG } 06:33:30 DEBUG } 06:33:30 DEBUG } 06:33:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727210" 06:33:31 INFO [http_cmd]: http status code OK 06:33:31 DEBUG --- status code --- 06:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:31 DEBUG --- http response --- 06:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:33:31 DEBUG ----- output ----- 06:33:31 DEBUG { 06:33:31 DEBUG "status": "success", 06:33:31 DEBUG "data": { 06:33:31 DEBUG "resultType": "vector", 06:33:31 DEBUG "result": [ 06:33:31 DEBUG { 06:33:31 DEBUG "metric": { 06:33:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:33:31 DEBUG }, 06:33:31 DEBUG "value": [ 06:33:31 DEBUG 06:33:31 DEBUG 06:33:31 DEBUG 06:33:31 DEBUG [...] 06:33:31 DEBUG 06:33:31 DEBUG 06:33:31 DEBUG 06:33:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:33:31 DEBUG }, 06:33:31 DEBUG "value": [ 06:33:31 DEBUG 1712727210, 06:33:31 DEBUG "281987628.64654267" 06:33:31 DEBUG ] 06:33:31 DEBUG } 06:33:31 DEBUG ] 06:33:31 DEBUG } 06:33:31 DEBUG } 06:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712727211" 06:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712727211" 06:33:31 INFO [http_cmd]: http status code OK 06:33:31 INFO [http_cmd]: http status code OK 06:33:31 DEBUG --- status code --- 06:33:31 DEBUG --- status code --- 06:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:31 DEBUG --- http response --- 06:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:33:31 DEBUG ----- output ----- 06:33:31 DEBUG { 06:33:31 DEBUG "status": "success", 06:33:31 DEBUG "data": { 06:33:31 DEBUG --- http response --- 06:33:31 DEBUG "resultType": "vector", 06:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:33:31 DEBUG "result": [ 06:33:31 DEBUG ----- output ----- 06:33:31 DEBUG { 06:33:31 DEBUG { 06:33:31 DEBUG "metric": { 06:33:31 DEBUG "status": "success", 06:33:31 DEBUG "pod": "ds-idrepo-0" 06:33:31 DEBUG "data": { 06:33:31 DEBUG }, 06:33:31 DEBUG "resultType": "vector", 06:33:31 DEBUG "value": [ 06:33:31 DEBUG "result": [ 06:33:31 DEBUG 06:33:31 DEBUG { 06:33:31 DEBUG "metric": { 06:33:31 DEBUG 06:33:31 DEBUG 06:33:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:33:31 DEBUG }, 06:33:31 DEBUG "value": [ 06:33:31 DEBUG 06:33:31 DEBUG 06:33:31 DEBUG [...] 06:33:31 DEBUG 06:33:31 DEBUG 06:33:31 DEBUG [...] 06:33:31 DEBUG 06:33:31 DEBUG 06:33:31 DEBUG 06:33:31 DEBUG 06:33:31 DEBUG "pod": "ds-cts-2" 06:33:31 DEBUG 06:33:31 DEBUG }, 06:33:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:33:31 DEBUG "value": [ 06:33:31 DEBUG }, 06:33:31 DEBUG 1712727211, 06:33:31 DEBUG "value": [ 06:33:31 DEBUG "0" 06:33:31 DEBUG 1712727211, 06:33:31 DEBUG ] 06:33:31 DEBUG "4.410859970369137" 06:33:31 DEBUG } 06:33:31 DEBUG ] 06:33:31 DEBUG ] 06:33:31 DEBUG } 06:33:31 DEBUG } 06:33:31 DEBUG } 06:33:31 DEBUG ] 06:33:31 DEBUG } 06:33:31 DEBUG } 06:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712727211" 06:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727211" 06:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727211" 06:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727211" 06:33:31 INFO [http_cmd]: http status code OK 06:33:31 DEBUG --- status code --- 06:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:31 DEBUG --- http response --- 06:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:33:31 DEBUG ----- output ----- 06:33:31 DEBUG { 06:33:31 DEBUG "status": "success", 06:33:31 DEBUG "data": { 06:33:31 DEBUG "resultType": "vector", 06:33:31 DEBUG "result": [ 06:33:31 DEBUG { 06:33:31 DEBUG "metric": { 06:33:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:33:31 DEBUG }, 06:33:31 DEBUG "value": [ 06:33:31 DEBUG 06:33:31 DEBUG 06:33:31 DEBUG 06:33:31 DEBUG [...] 06:33:31 DEBUG 06:33:31 DEBUG 06:33:31 DEBUG 06:33:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:33:31 DEBUG }, 06:33:31 DEBUG "value": [ 06:33:31 DEBUG 1712727211, 06:33:31 DEBUG "0.051" 06:33:31 DEBUG ] 06:33:31 DEBUG } 06:33:31 DEBUG ] 06:33:31 DEBUG } 06:33:31 DEBUG } 06:33:31 INFO [http_cmd]: http status code OK 06:33:31 DEBUG --- status code --- 06:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:31 DEBUG --- http response --- 06:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:33:31 INFO [http_cmd]: http status code OK 06:33:31 DEBUG ----- output ----- 06:33:31 DEBUG --- status code --- 06:33:31 DEBUG { 06:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:31 DEBUG "status": "success", 06:33:31 DEBUG --- http response --- 06:33:31 DEBUG "data": { 06:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:33:31 DEBUG "resultType": "vector", 06:33:31 DEBUG ----- output ----- 06:33:31 DEBUG "result": [ 06:33:31 DEBUG { 06:33:31 DEBUG { 06:33:31 DEBUG "metric": { 06:33:31 DEBUG "status": "success", 06:33:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:33:31 DEBUG "data": { 06:33:31 DEBUG }, 06:33:31 DEBUG "resultType": "vector", 06:33:31 DEBUG "value": [ 06:33:31 DEBUG "result": [ 06:33:31 DEBUG 06:33:31 DEBUG { 06:33:31 DEBUG 06:33:31 DEBUG "metric": { 06:33:31 DEBUG 06:33:31 DEBUG "pod": "ds-idrepo-2" 06:33:31 DEBUG [...] 06:33:31 DEBUG }, 06:33:31 DEBUG 06:33:31 DEBUG "value": [ 06:33:31 DEBUG 06:33:31 DEBUG 06:33:31 DEBUG 06:33:31 DEBUG 06:33:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:33:31 DEBUG 06:33:31 DEBUG }, 06:33:31 DEBUG [...] 06:33:31 INFO [http_cmd]: http status code OK 06:33:31 DEBUG "value": [ 06:33:31 DEBUG 06:33:31 DEBUG --- status code --- 06:33:31 DEBUG 1712727211, 06:33:31 DEBUG 06:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:31 DEBUG "1443.5333333333333" 06:33:31 DEBUG 06:33:31 DEBUG --- http response --- 06:33:31 DEBUG ] 06:33:31 DEBUG "pod": "ds-cts-2" 06:33:31 DEBUG }, 06:33:31 DEBUG } 06:33:31 DEBUG ] 06:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:33:31 DEBUG "value": [ 06:33:31 DEBUG } 06:33:31 DEBUG ----- output ----- 06:33:31 DEBUG 1712727211, 06:33:31 DEBUG } 06:33:31 DEBUG { 06:33:31 DEBUG "status": "success", 06:33:31 DEBUG "data": { 06:33:31 DEBUG "resultType": "vector", 06:33:31 DEBUG "result": [ 06:33:31 DEBUG { 06:33:31 DEBUG "metric": { 06:33:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:33:31 DEBUG }, 06:33:31 DEBUG "value": [ 06:33:31 DEBUG 06:33:31 DEBUG 06:33:31 DEBUG 06:33:31 DEBUG [...] 06:33:31 DEBUG 06:33:31 DEBUG 06:33:31 DEBUG 06:33:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:33:31 DEBUG }, 06:33:31 DEBUG "value": [ 06:33:31 DEBUG 1712727211, 06:33:31 DEBUG "8.355528048335387" 06:33:31 DEBUG ] 06:33:31 DEBUG } 06:33:31 DEBUG ] 06:33:31 DEBUG } 06:33:31 DEBUG } 06:33:31 DEBUG "0.000023439444444444447" 06:33:31 DEBUG ] 06:33:31 DEBUG } 06:33:31 DEBUG ] 06:33:31 DEBUG } 06:33:31 DEBUG } 06:33:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727211" 06:33:31 INFO [http_cmd]: http status code OK 06:33:31 DEBUG --- status code --- 06:33:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:31 DEBUG --- http response --- 06:33:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:33:31 DEBUG ----- output ----- 06:33:31 DEBUG { 06:33:31 DEBUG "status": "success", 06:33:31 DEBUG "data": { 06:33:31 DEBUG "resultType": "vector", 06:33:31 DEBUG "result": [ 06:33:31 DEBUG { 06:33:31 DEBUG "metric": { 06:33:31 DEBUG "pod": "ds-idrepo-0" 06:33:31 DEBUG }, 06:33:31 DEBUG "value": [ 06:33:31 DEBUG 06:33:31 DEBUG 06:33:31 DEBUG 06:33:31 DEBUG [...] 06:33:31 DEBUG 06:33:31 DEBUG 06:33:31 DEBUG 06:33:31 DEBUG "pod": "ds-idrepo-1" 06:33:31 DEBUG }, 06:33:31 DEBUG "value": [ 06:33:31 DEBUG 1712727211, 06:33:31 DEBUG "0" 06:33:31 DEBUG ] 06:33:31 DEBUG } 06:33:31 DEBUG ] 06:33:31 DEBUG } 06:33:31 DEBUG } 06:33:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727212" 06:33:32 INFO [http_cmd]: http status code OK 06:33:32 DEBUG --- status code --- 06:33:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:32 DEBUG --- http response --- 06:33:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:33:32 DEBUG ----- output ----- 06:33:32 DEBUG { 06:33:32 DEBUG "status": "success", 06:33:32 DEBUG "data": { 06:33:32 DEBUG "resultType": "vector", 06:33:32 DEBUG "result": [ 06:33:32 DEBUG { 06:33:32 DEBUG "metric": { 06:33:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:33:32 DEBUG }, 06:33:32 DEBUG "value": [ 06:33:32 DEBUG 06:33:32 DEBUG 06:33:32 DEBUG 06:33:32 DEBUG [...] 06:33:32 DEBUG 06:33:32 DEBUG 06:33:32 DEBUG 06:33:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:33:32 DEBUG }, 06:33:32 DEBUG "value": [ 06:33:32 DEBUG 1712727212, 06:33:32 DEBUG "1447.1333333333332" 06:33:32 DEBUG ] 06:33:32 DEBUG } 06:33:32 DEBUG ] 06:33:32 DEBUG } 06:33:32 DEBUG } 06:33:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727212" 06:33:32 INFO [http_cmd]: http status code OK 06:33:32 DEBUG --- status code --- 06:33:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:32 DEBUG --- http response --- 06:33:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:33:32 DEBUG ----- output ----- 06:33:32 DEBUG { 06:33:32 DEBUG "status": "success", 06:33:32 DEBUG "data": { 06:33:32 DEBUG "resultType": "vector", 06:33:32 DEBUG "result": [ 06:33:32 DEBUG { 06:33:32 DEBUG "metric": { 06:33:32 DEBUG "pod": "ds-idrepo-0" 06:33:32 DEBUG }, 06:33:32 DEBUG "value": [ 06:33:32 DEBUG 06:33:32 DEBUG 06:33:32 DEBUG 06:33:32 DEBUG [...] 06:33:32 DEBUG 06:33:32 DEBUG 06:33:32 DEBUG 06:33:32 DEBUG "pod": "ds-idrepo-1" 06:33:32 DEBUG }, 06:33:32 DEBUG "value": [ 06:33:32 DEBUG 1712727212, 06:33:32 DEBUG "0" 06:33:32 DEBUG ] 06:33:32 DEBUG } 06:33:32 DEBUG ] 06:33:32 DEBUG } 06:33:32 DEBUG } 06:33:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727212" 06:33:32 INFO [http_cmd]: http status code OK 06:33:32 DEBUG --- status code --- 06:33:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:32 DEBUG --- http response --- 06:33:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:33:32 DEBUG ----- output ----- 06:33:32 DEBUG { 06:33:32 DEBUG "status": "success", 06:33:32 DEBUG "data": { 06:33:32 DEBUG "resultType": "vector", 06:33:32 DEBUG "result": [ 06:33:32 DEBUG { 06:33:32 DEBUG "metric": { 06:33:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:33:32 DEBUG }, 06:33:32 DEBUG "value": [ 06:33:32 DEBUG 06:33:32 DEBUG 06:33:32 DEBUG 06:33:32 DEBUG [...] 06:33:32 DEBUG 06:33:32 DEBUG 06:33:32 DEBUG 06:33:32 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:33:32 DEBUG }, 06:33:32 DEBUG "value": [ 06:33:32 DEBUG 1712727212, 06:33:32 DEBUG "0" 06:33:32 DEBUG ] 06:33:32 DEBUG } 06:33:32 DEBUG ] 06:33:32 DEBUG } 06:33:32 DEBUG } 06:33:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712727212" 06:33:32 INFO [http_cmd]: http status code OK 06:33:32 DEBUG --- status code --- 06:33:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:32 DEBUG --- http response --- 06:33:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:33:32 DEBUG ----- output ----- 06:33:32 DEBUG { 06:33:32 DEBUG "status": "success", 06:33:32 DEBUG "data": { 06:33:32 DEBUG "resultType": "vector", 06:33:32 DEBUG "result": [ 06:33:32 DEBUG { 06:33:32 DEBUG "metric": { 06:33:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:33:32 DEBUG }, 06:33:32 DEBUG "value": [ 06:33:32 DEBUG 06:33:32 DEBUG 06:33:32 DEBUG 06:33:32 DEBUG [...] 06:33:32 DEBUG 06:33:32 DEBUG 06:33:32 DEBUG 06:33:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:33:32 DEBUG }, 06:33:32 DEBUG "value": [ 06:33:32 DEBUG 1712727212, 06:33:32 DEBUG "2486722560" 06:33:32 DEBUG ] 06:33:32 DEBUG } 06:33:32 DEBUG ] 06:33:32 DEBUG } 06:33:32 DEBUG } 06:33:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727212" 06:33:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727212" 06:33:32 INFO [http_cmd]: http status code OK 06:33:32 DEBUG --- status code --- 06:33:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:32 DEBUG --- http response --- 06:33:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:33:32 DEBUG ----- output ----- 06:33:32 DEBUG { 06:33:32 DEBUG "status": "success", 06:33:32 DEBUG "data": { 06:33:32 DEBUG "resultType": "vector", 06:33:32 DEBUG "result": [ 06:33:32 DEBUG { 06:33:32 DEBUG "metric": { 06:33:32 DEBUG "instance": "10.142.0.25:9100" 06:33:32 DEBUG }, 06:33:32 DEBUG "value": [ 06:33:32 DEBUG 06:33:32 DEBUG 06:33:32 DEBUG 06:33:32 DEBUG [...] 06:33:32 DEBUG 06:33:32 DEBUG 06:33:32 DEBUG 06:33:32 DEBUG "instance": "10.142.0.77:9100" 06:33:32 DEBUG }, 06:33:32 DEBUG "value": [ 06:33:32 DEBUG 1712727212, 06:33:32 DEBUG "42510451.099999994" 06:33:32 DEBUG ] 06:33:32 DEBUG } 06:33:32 DEBUG ] 06:33:32 DEBUG } 06:33:32 DEBUG } 06:33:32 INFO [http_cmd]: http status code OK 06:33:32 DEBUG --- status code --- 06:33:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:32 DEBUG --- http response --- 06:33:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:33:32 DEBUG ----- output ----- 06:33:32 DEBUG { 06:33:32 DEBUG "status": "success", 06:33:32 DEBUG "data": { 06:33:32 DEBUG "resultType": "vector", 06:33:32 DEBUG "result": [ 06:33:32 DEBUG { 06:33:32 DEBUG "metric": { 06:33:32 DEBUG "instance": "10.142.0.25:9100" 06:33:32 DEBUG }, 06:33:32 DEBUG "value": [ 06:33:32 DEBUG 06:33:32 DEBUG 06:33:32 DEBUG 06:33:32 DEBUG [...] 06:33:32 DEBUG 06:33:32 DEBUG 06:33:32 DEBUG 06:33:32 DEBUG "instance": "10.142.0.77:9100" 06:33:32 DEBUG }, 06:33:32 DEBUG "value": [ 06:33:32 DEBUG 1712727212, 06:33:32 DEBUG "0.035999999999999706" 06:33:32 DEBUG ] 06:33:32 DEBUG } 06:33:32 DEBUG ] 06:33:32 DEBUG } 06:33:32 DEBUG } 06:33:51 INFO 06:33:51 INFO [loop_until]: kubectl --namespace=xlou top pods 06:33:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:33:52 INFO [loop_until]: OK (rc = 0) 06:33:52 DEBUG --- stdout --- 06:33:52 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 11009m 8992Mi am-654c4fcbcf-9scrj 9926m 8458Mi am-654c4fcbcf-v5qb5 8264m 8489Mi ds-cts-0 8812m 13391Mi ds-cts-1 8318m 12682Mi ds-cts-2 7407m 12977Mi ds-idrepo-0 10573m 5623Mi ds-idrepo-1 371m 5672Mi ds-idrepo-2 367m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1191Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 8m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4270m 2511Mi 06:33:52 DEBUG --- stderr --- 06:33:52 DEBUG 06:34:00 INFO 06:34:00 INFO [loop_until]: kubectl --namespace=xlou top node 06:34:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:34:00 INFO [loop_until]: OK (rc = 0) 06:34:00 DEBUG --- stdout --- 06:34:00 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 95m 0% 2306Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9190m 57% 9363Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 79m 0% 3259Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 14226m 89% 6822Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10964m 68% 9339Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 12057m 75% 9815Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 79m 0% 854Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10888m 68% 6307Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8582m 54% 14104Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 7489m 47% 13609Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 443m 2% 6395Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8503m 53% 13540Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 407m 2% 6361Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4515m 28% 6198Mi 10% 06:34:00 DEBUG --- stderr --- 06:34:00 DEBUG 06:34:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712727269" 06:34:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727269" 06:34:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727269" 06:34:29 ERROR [http_cmd]: ERROR 06:34:29 INFO [http_cmd]: http status code OK 06:34:29 ERROR --- status code --- 06:34:29 DEBUG --- status code --- 06:34:29 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:34:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:29 ERROR --- http response --- 06:34:29 DEBUG --- http response --- 06:34:29 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:34:29 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:34:29 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:34:29 INFO [http_cmd]: http status code OK 06:34:29 DEBUG --- status code --- 06:34:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:29 DEBUG --- http response --- 06:34:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:34:29 DEBUG ----- output ----- 06:34:29 DEBUG { 06:34:29 DEBUG "status": "success", 06:34:29 DEBUG "data": { 06:34:29 DEBUG "resultType": "vector", 06:34:29 DEBUG "result": [ 06:34:29 DEBUG { 06:34:29 DEBUG "metric": { 06:34:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:34:29 DEBUG }, 06:34:29 DEBUG "value": [ 06:34:29 DEBUG 06:34:29 DEBUG 06:34:29 DEBUG 06:34:29 DEBUG [...] 06:34:29 DEBUG 06:34:29 DEBUG 06:34:29 DEBUG 06:34:29 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:34:29 DEBUG }, 06:34:29 DEBUG "value": [ 06:34:29 DEBUG 1712727269, 06:34:29 DEBUG "0" 06:34:29 DEBUG ] 06:34:29 DEBUG } 06:34:29 DEBUG ] 06:34:29 DEBUG } 06:34:29 DEBUG } 06:34:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727270" 06:34:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727270" 06:34:30 INFO [http_cmd]: http status code OK 06:34:30 DEBUG --- status code --- 06:34:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:30 DEBUG --- http response --- 06:34:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:34:30 DEBUG ----- output ----- 06:34:30 DEBUG { 06:34:30 DEBUG "status": "success", 06:34:30 DEBUG "data": { 06:34:30 DEBUG "resultType": "vector", 06:34:30 DEBUG "result": [ 06:34:30 DEBUG { 06:34:30 DEBUG "metric": { 06:34:30 DEBUG "instance": "10.142.0.25:9100" 06:34:30 DEBUG }, 06:34:30 DEBUG "value": [ 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG [...] 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG "instance": "10.142.0.77:9100" 06:34:30 DEBUG }, 06:34:30 DEBUG "value": [ 06:34:30 DEBUG 1712727270, 06:34:30 DEBUG "42569870.733333334" 06:34:30 DEBUG ] 06:34:30 DEBUG } 06:34:30 DEBUG ] 06:34:30 DEBUG } 06:34:30 DEBUG } 06:34:30 INFO [http_cmd]: http status code OK 06:34:30 DEBUG --- status code --- 06:34:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:30 DEBUG --- http response --- 06:34:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:34:30 DEBUG ----- output ----- 06:34:30 DEBUG { 06:34:30 DEBUG "status": "success", 06:34:30 DEBUG "data": { 06:34:30 DEBUG "resultType": "vector", 06:34:30 DEBUG "result": [ 06:34:30 DEBUG { 06:34:30 DEBUG "metric": { 06:34:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:34:30 DEBUG }, 06:34:30 DEBUG "value": [ 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG [...] 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:34:30 DEBUG }, 06:34:30 DEBUG "value": [ 06:34:30 DEBUG 1712727270, 06:34:30 DEBUG "1399.5666666666666" 06:34:30 DEBUG ] 06:34:30 DEBUG } 06:34:30 DEBUG ] 06:34:30 DEBUG } 06:34:30 DEBUG } 06:34:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727270" 06:34:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727270" 06:34:30 INFO [http_cmd]: http status code OK 06:34:30 DEBUG --- status code --- 06:34:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:30 DEBUG --- http response --- 06:34:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:34:30 DEBUG ----- output ----- 06:34:30 DEBUG { 06:34:30 DEBUG "status": "success", 06:34:30 DEBUG "data": { 06:34:30 DEBUG "resultType": "vector", 06:34:30 DEBUG "result": [ 06:34:30 DEBUG { 06:34:30 DEBUG "metric": { 06:34:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:34:30 DEBUG }, 06:34:30 DEBUG "value": [ 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG [...] 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:34:30 DEBUG }, 06:34:30 DEBUG "value": [ 06:34:30 DEBUG 1712727270, 06:34:30 DEBUG "1399.6333333333334" 06:34:30 DEBUG ] 06:34:30 DEBUG } 06:34:30 DEBUG ] 06:34:30 DEBUG } 06:34:30 DEBUG } 06:34:30 INFO [http_cmd]: http status code OK 06:34:30 DEBUG --- status code --- 06:34:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:30 DEBUG --- http response --- 06:34:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:34:30 DEBUG ----- output ----- 06:34:30 DEBUG { 06:34:30 DEBUG "status": "success", 06:34:30 DEBUG "data": { 06:34:30 DEBUG "resultType": "vector", 06:34:30 DEBUG "result": [ 06:34:30 DEBUG { 06:34:30 DEBUG "metric": { 06:34:30 DEBUG "pod": "ds-idrepo-0" 06:34:30 DEBUG }, 06:34:30 DEBUG "value": [ 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG [...] 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG "pod": "ds-idrepo-1" 06:34:30 DEBUG }, 06:34:30 DEBUG "value": [ 06:34:30 DEBUG 1712727270, 06:34:30 DEBUG "0" 06:34:30 DEBUG ] 06:34:30 DEBUG } 06:34:30 DEBUG ] 06:34:30 DEBUG } 06:34:30 DEBUG } 06:34:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727270" 06:34:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727270" 06:34:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727270" 06:34:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727270" 06:34:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727270" 06:34:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727270" 06:34:30 INFO [http_cmd]: http status code OK 06:34:30 DEBUG --- status code --- 06:34:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:30 DEBUG --- http response --- 06:34:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:34:30 DEBUG ----- output ----- 06:34:30 DEBUG { 06:34:30 DEBUG "status": "success", 06:34:30 DEBUG "data": { 06:34:30 DEBUG "resultType": "vector", 06:34:30 DEBUG "result": [ 06:34:30 DEBUG { 06:34:30 DEBUG "metric": { 06:34:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:34:30 DEBUG }, 06:34:30 DEBUG "value": [ 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG [...] 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:34:30 DEBUG }, 06:34:30 DEBUG "value": [ 06:34:30 DEBUG 1712727270, 06:34:30 DEBUG "0" 06:34:30 INFO [http_cmd]: http status code OK 06:34:30 DEBUG ] 06:34:30 DEBUG } 06:34:30 DEBUG ] 06:34:30 DEBUG } 06:34:30 DEBUG } 06:34:30 DEBUG --- status code --- 06:34:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:30 INFO [http_cmd]: http status code OK 06:34:30 INFO [http_cmd]: http status code OK 06:34:30 DEBUG --- status code --- 06:34:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:30 DEBUG --- status code --- 06:34:30 INFO [http_cmd]: http status code OK 06:34:30 DEBUG --- http response --- 06:34:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:34:30 DEBUG ----- output ----- 06:34:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:30 DEBUG --- http response --- 06:34:30 DEBUG { 06:34:30 DEBUG "status": "success", 06:34:30 DEBUG "data": { 06:34:30 DEBUG --- status code --- 06:34:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:30 DEBUG "resultType": "vector", 06:34:30 DEBUG "result": [ 06:34:30 DEBUG { 06:34:30 DEBUG --- http response --- 06:34:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:34:30 DEBUG ----- output ----- 06:34:30 INFO [http_cmd]: http status code OK 06:34:30 DEBUG --- status code --- 06:34:30 DEBUG { 06:34:30 DEBUG "metric": { 06:34:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:34:30 DEBUG ----- output ----- 06:34:30 DEBUG --- http response --- 06:34:30 DEBUG "status": "success", 06:34:30 DEBUG "pod": "ds-idrepo-0" 06:34:30 DEBUG { 06:34:30 DEBUG "status": "success", 06:34:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:34:30 DEBUG "data": { 06:34:30 DEBUG }, 06:34:30 DEBUG "data": { 06:34:30 DEBUG "resultType": "vector", 06:34:30 DEBUG ----- output ----- 06:34:30 DEBUG "resultType": "vector", 06:34:30 DEBUG "value": [ 06:34:30 DEBUG 06:34:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:30 DEBUG { 06:34:30 DEBUG "result": [ 06:34:30 DEBUG { 06:34:30 DEBUG "metric": { 06:34:30 DEBUG "pod": "ds-idrepo-0" 06:34:30 DEBUG "status": "success", 06:34:30 DEBUG }, 06:34:30 DEBUG "value": [ 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG [...] 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG "pod": "ds-cts-2" 06:34:30 DEBUG }, 06:34:30 DEBUG "value": [ 06:34:30 DEBUG 1712727270, 06:34:30 DEBUG "29.833333333333332" 06:34:30 DEBUG ] 06:34:30 DEBUG "data": { 06:34:30 DEBUG "resultType": "vector", 06:34:30 DEBUG "result": [ 06:34:30 DEBUG { 06:34:30 DEBUG "metric": { 06:34:30 DEBUG "instance": "10.142.0.25:9100" 06:34:30 DEBUG "result": [ 06:34:30 DEBUG }, 06:34:30 DEBUG "value": [ 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG { 06:34:30 DEBUG "metric": { 06:34:30 DEBUG "instance": "10.142.0.25:9100" 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG [...] 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG [...] 06:34:30 DEBUG }, 06:34:30 DEBUG "value": [ 06:34:30 DEBUG } 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG ] 06:34:30 DEBUG 06:34:30 DEBUG --- http response --- 06:34:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:34:30 DEBUG 06:34:30 DEBUG } 06:34:30 DEBUG } 06:34:30 DEBUG ----- output ----- 06:34:30 DEBUG { 06:34:30 DEBUG "status": "success", 06:34:30 DEBUG "data": { 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG "instance": "10.142.0.77:9100" 06:34:30 DEBUG }, 06:34:30 DEBUG [...] 06:34:30 DEBUG "value": [ 06:34:30 DEBUG "resultType": "vector", 06:34:30 DEBUG "pod": "ds-cts-2" 06:34:30 DEBUG 06:34:30 DEBUG 1712727270, 06:34:30 DEBUG "result": [ 06:34:30 DEBUG }, 06:34:30 DEBUG "value": [ 06:34:30 DEBUG 1712727270, 06:34:30 DEBUG { 06:34:30 DEBUG "metric": { 06:34:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:34:30 DEBUG "0.1617666666666634" 06:34:30 DEBUG ] 06:34:30 DEBUG }, 06:34:30 DEBUG "value": [ 06:34:30 DEBUG } 06:34:30 DEBUG ] 06:34:30 DEBUG } 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG "instance": "10.142.0.77:9100" 06:34:30 DEBUG "0" 06:34:30 DEBUG ] 06:34:30 DEBUG } 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG ] 06:34:30 DEBUG } 06:34:30 DEBUG 06:34:30 DEBUG [...] 06:34:30 DEBUG 06:34:30 DEBUG } 06:34:30 DEBUG 06:34:30 DEBUG 06:34:30 DEBUG } 06:34:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:34:30 DEBUG }, 06:34:30 DEBUG "value": [ 06:34:30 DEBUG 1712727270, 06:34:30 DEBUG "257407" 06:34:30 DEBUG ] 06:34:30 DEBUG } 06:34:30 DEBUG ] 06:34:30 DEBUG } 06:34:30 DEBUG } 06:34:30 DEBUG }, 06:34:30 DEBUG "value": [ 06:34:30 DEBUG 1712727270, 06:34:30 DEBUG "410789395456" 06:34:30 DEBUG ] 06:34:30 DEBUG } 06:34:30 DEBUG ] 06:34:30 DEBUG } 06:34:30 DEBUG } 06:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727271" 06:34:31 INFO [http_cmd]: http status code OK 06:34:31 DEBUG --- status code --- 06:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:31 DEBUG --- http response --- 06:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:34:31 DEBUG ----- output ----- 06:34:31 DEBUG { 06:34:31 DEBUG "status": "success", 06:34:31 DEBUG "data": { 06:34:31 DEBUG "resultType": "vector", 06:34:31 DEBUG "result": [ 06:34:31 DEBUG { 06:34:31 DEBUG "metric": { 06:34:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:34:31 DEBUG }, 06:34:31 DEBUG "value": [ 06:34:31 DEBUG 06:34:31 DEBUG 06:34:31 DEBUG 06:34:31 DEBUG [...] 06:34:31 DEBUG 06:34:31 DEBUG 06:34:31 DEBUG 06:34:31 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:34:31 DEBUG }, 06:34:31 DEBUG "value": [ 06:34:31 DEBUG 1712727271, 06:34:31 DEBUG "123694756.30966535" 06:34:31 DEBUG ] 06:34:31 DEBUG } 06:34:31 DEBUG ] 06:34:31 DEBUG } 06:34:31 DEBUG } 06:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712727271" 06:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712727271" 06:34:31 INFO [http_cmd]: http status code OK 06:34:31 DEBUG --- status code --- 06:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:31 DEBUG --- http response --- 06:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:34:31 DEBUG ----- output ----- 06:34:31 DEBUG { 06:34:31 DEBUG "status": "success", 06:34:31 DEBUG "data": { 06:34:31 DEBUG "resultType": "vector", 06:34:31 DEBUG "result": [ 06:34:31 DEBUG { 06:34:31 DEBUG "metric": { 06:34:31 DEBUG "pod": "ds-idrepo-0" 06:34:31 DEBUG }, 06:34:31 DEBUG "value": [ 06:34:31 DEBUG 06:34:31 DEBUG 06:34:31 DEBUG 06:34:31 DEBUG [...] 06:34:31 DEBUG 06:34:31 DEBUG 06:34:31 DEBUG 06:34:31 DEBUG "pod": "ds-cts-2" 06:34:31 DEBUG }, 06:34:31 DEBUG "value": [ 06:34:31 DEBUG 1712727271, 06:34:31 DEBUG "0" 06:34:31 DEBUG ] 06:34:31 DEBUG } 06:34:31 DEBUG ] 06:34:31 DEBUG } 06:34:31 DEBUG } 06:34:31 INFO [http_cmd]: http status code OK 06:34:31 DEBUG --- status code --- 06:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:31 DEBUG --- http response --- 06:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:34:31 DEBUG ----- output ----- 06:34:31 DEBUG { 06:34:31 DEBUG "status": "success", 06:34:31 DEBUG "data": { 06:34:31 DEBUG "resultType": "vector", 06:34:31 DEBUG "result": [ 06:34:31 DEBUG { 06:34:31 DEBUG "metric": { 06:34:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:34:31 DEBUG }, 06:34:31 DEBUG "value": [ 06:34:31 DEBUG 06:34:31 DEBUG 06:34:31 DEBUG 06:34:31 DEBUG [...] 06:34:31 DEBUG 06:34:31 DEBUG 06:34:31 DEBUG 06:34:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:34:31 DEBUG }, 06:34:31 DEBUG "value": [ 06:34:31 DEBUG 1712727271, 06:34:31 DEBUG "4.449052466355197" 06:34:31 DEBUG ] 06:34:31 DEBUG } 06:34:31 DEBUG ] 06:34:31 DEBUG } 06:34:31 DEBUG } 06:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712727271" 06:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727271" 06:34:31 INFO [http_cmd]: http status code OK 06:34:31 DEBUG --- status code --- 06:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727271" 06:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727271" 06:34:31 DEBUG --- http response --- 06:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:34:31 DEBUG ----- output ----- 06:34:31 DEBUG { 06:34:31 DEBUG "status": "success", 06:34:31 DEBUG "data": { 06:34:31 DEBUG "resultType": "vector", 06:34:31 DEBUG "result": [ 06:34:31 DEBUG { 06:34:31 DEBUG "metric": { 06:34:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:34:31 DEBUG }, 06:34:31 DEBUG "value": [ 06:34:31 DEBUG 06:34:31 DEBUG 06:34:31 DEBUG 06:34:31 DEBUG [...] 06:34:31 DEBUG 06:34:31 DEBUG 06:34:31 DEBUG 06:34:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:34:31 DEBUG }, 06:34:31 DEBUG "value": [ 06:34:31 DEBUG 1712727271, 06:34:31 DEBUG "0.051" 06:34:31 DEBUG ] 06:34:31 DEBUG } 06:34:31 DEBUG ] 06:34:31 DEBUG } 06:34:31 DEBUG } 06:34:31 INFO [http_cmd]: http status code OK 06:34:31 DEBUG --- status code --- 06:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:31 DEBUG --- http response --- 06:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:34:31 DEBUG ----- output ----- 06:34:31 DEBUG { 06:34:31 DEBUG "status": "success", 06:34:31 DEBUG "data": { 06:34:31 DEBUG "resultType": "vector", 06:34:31 DEBUG "result": [ 06:34:31 DEBUG { 06:34:31 DEBUG "metric": { 06:34:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:34:31 DEBUG }, 06:34:31 DEBUG "value": [ 06:34:31 DEBUG 06:34:31 DEBUG 06:34:31 DEBUG 06:34:31 DEBUG [...] 06:34:31 DEBUG 06:34:31 DEBUG 06:34:31 DEBUG 06:34:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:34:31 DEBUG }, 06:34:31 DEBUG "value": [ 06:34:31 DEBUG 1712727271, 06:34:31 DEBUG "1399.7333333333333" 06:34:31 DEBUG ] 06:34:31 DEBUG } 06:34:31 DEBUG ] 06:34:31 DEBUG } 06:34:31 DEBUG } 06:34:31 INFO [http_cmd]: http status code OK 06:34:31 INFO [http_cmd]: http status code OK 06:34:31 DEBUG --- status code --- 06:34:31 DEBUG --- status code --- 06:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:31 DEBUG --- http response --- 06:34:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:34:31 DEBUG ----- output ----- 06:34:31 DEBUG --- http response --- 06:34:31 DEBUG { 06:34:31 DEBUG "status": "success", 06:34:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:34:31 DEBUG "data": { 06:34:31 DEBUG ----- output ----- 06:34:31 DEBUG "resultType": "vector", 06:34:31 DEBUG { 06:34:31 DEBUG "result": [ 06:34:31 DEBUG "status": "success", 06:34:31 DEBUG { 06:34:31 DEBUG "data": { 06:34:31 DEBUG "metric": { 06:34:31 DEBUG "resultType": "vector", 06:34:31 DEBUG "pod": "ds-idrepo-2" 06:34:31 DEBUG "result": [ 06:34:31 DEBUG }, 06:34:31 DEBUG { 06:34:31 DEBUG "value": [ 06:34:31 DEBUG "metric": { 06:34:31 DEBUG 06:34:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:34:31 DEBUG 06:34:31 DEBUG }, 06:34:31 DEBUG 06:34:31 DEBUG "value": [ 06:34:31 DEBUG [...] 06:34:31 DEBUG 06:34:31 DEBUG 06:34:31 DEBUG 06:34:31 DEBUG 06:34:31 DEBUG 06:34:31 DEBUG 06:34:31 DEBUG [...] 06:34:31 DEBUG "pod": "ds-cts-2" 06:34:31 DEBUG 06:34:31 DEBUG }, 06:34:31 DEBUG 06:34:31 DEBUG "value": [ 06:34:31 DEBUG 06:34:31 DEBUG 1712727271, 06:34:31 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:34:31 DEBUG }, 06:34:31 DEBUG "0.00007032333333333334" 06:34:31 DEBUG "value": [ 06:34:31 DEBUG ] 06:34:31 DEBUG 1712727271, 06:34:31 DEBUG } 06:34:31 DEBUG "0.0055254468498860485" 06:34:31 DEBUG ] 06:34:31 DEBUG ] 06:34:31 DEBUG } 06:34:31 DEBUG } 06:34:31 DEBUG } 06:34:31 DEBUG ] 06:34:31 DEBUG } 06:34:31 DEBUG } 06:34:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727271" 06:34:32 INFO [http_cmd]: http status code OK 06:34:32 DEBUG --- status code --- 06:34:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:32 DEBUG --- http response --- 06:34:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:34:32 DEBUG ----- output ----- 06:34:32 DEBUG { 06:34:32 DEBUG "status": "success", 06:34:32 DEBUG "data": { 06:34:32 DEBUG "resultType": "vector", 06:34:32 DEBUG "result": [ 06:34:32 DEBUG { 06:34:32 DEBUG "metric": { 06:34:32 DEBUG "pod": "ds-idrepo-0" 06:34:32 DEBUG }, 06:34:32 DEBUG "value": [ 06:34:32 DEBUG 06:34:32 DEBUG 06:34:32 DEBUG 06:34:32 DEBUG [...] 06:34:32 DEBUG 06:34:32 DEBUG 06:34:32 DEBUG 06:34:32 DEBUG "pod": "ds-idrepo-1" 06:34:32 DEBUG }, 06:34:32 DEBUG "value": [ 06:34:32 DEBUG 1712727271, 06:34:32 DEBUG "0" 06:34:32 DEBUG ] 06:34:32 DEBUG } 06:34:32 DEBUG ] 06:34:32 DEBUG } 06:34:32 DEBUG } 06:34:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727272" 06:34:32 INFO [http_cmd]: http status code OK 06:34:32 DEBUG --- status code --- 06:34:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:32 DEBUG --- http response --- 06:34:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:34:32 DEBUG ----- output ----- 06:34:32 DEBUG { 06:34:32 DEBUG "status": "success", 06:34:32 DEBUG "data": { 06:34:32 DEBUG "resultType": "vector", 06:34:32 DEBUG "result": [ 06:34:32 DEBUG { 06:34:32 DEBUG "metric": { 06:34:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:34:32 DEBUG }, 06:34:32 DEBUG "value": [ 06:34:32 DEBUG 06:34:32 DEBUG 06:34:32 DEBUG 06:34:32 DEBUG [...] 06:34:32 DEBUG 06:34:32 DEBUG 06:34:32 DEBUG 06:34:32 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:34:32 DEBUG }, 06:34:32 DEBUG "value": [ 06:34:32 DEBUG 1712727272, 06:34:32 DEBUG "1390.9666666666667" 06:34:32 DEBUG ] 06:34:32 DEBUG } 06:34:32 DEBUG ] 06:34:32 DEBUG } 06:34:32 DEBUG } 06:34:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727272" 06:34:32 INFO [http_cmd]: http status code OK 06:34:32 DEBUG --- status code --- 06:34:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:32 DEBUG --- http response --- 06:34:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:34:32 DEBUG ----- output ----- 06:34:32 DEBUG { 06:34:32 DEBUG "status": "success", 06:34:32 DEBUG "data": { 06:34:32 DEBUG "resultType": "vector", 06:34:32 DEBUG "result": [ 06:34:32 DEBUG { 06:34:32 DEBUG "metric": { 06:34:32 DEBUG "pod": "ds-idrepo-0" 06:34:32 DEBUG }, 06:34:32 DEBUG "value": [ 06:34:32 DEBUG 06:34:32 DEBUG 06:34:32 DEBUG 06:34:32 DEBUG [...] 06:34:32 DEBUG 06:34:32 DEBUG 06:34:32 DEBUG 06:34:32 DEBUG "pod": "ds-idrepo-1" 06:34:32 DEBUG }, 06:34:32 DEBUG "value": [ 06:34:32 DEBUG 1712727272, 06:34:32 DEBUG "0" 06:34:32 DEBUG ] 06:34:32 DEBUG } 06:34:32 DEBUG ] 06:34:32 DEBUG } 06:34:32 DEBUG } 06:34:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727272" 06:34:32 INFO [http_cmd]: http status code OK 06:34:32 DEBUG --- status code --- 06:34:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:32 DEBUG --- http response --- 06:34:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:34:32 DEBUG ----- output ----- 06:34:32 DEBUG { 06:34:32 DEBUG "status": "success", 06:34:32 DEBUG "data": { 06:34:32 DEBUG "resultType": "vector", 06:34:32 DEBUG "result": [ 06:34:32 DEBUG { 06:34:32 DEBUG "metric": { 06:34:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:34:32 DEBUG }, 06:34:32 DEBUG "value": [ 06:34:32 DEBUG 06:34:32 DEBUG 06:34:32 DEBUG 06:34:32 DEBUG [...] 06:34:32 DEBUG 06:34:32 DEBUG 06:34:32 DEBUG 06:34:32 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:34:32 DEBUG }, 06:34:32 DEBUG "value": [ 06:34:32 DEBUG 1712727272, 06:34:32 DEBUG "0" 06:34:32 DEBUG ] 06:34:32 DEBUG } 06:34:32 DEBUG ] 06:34:32 DEBUG } 06:34:32 DEBUG } 06:34:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712727272" 06:34:32 INFO [http_cmd]: http status code OK 06:34:32 DEBUG --- status code --- 06:34:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:32 DEBUG --- http response --- 06:34:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:34:32 DEBUG ----- output ----- 06:34:32 DEBUG { 06:34:32 DEBUG "status": "success", 06:34:32 DEBUG "data": { 06:34:32 DEBUG "resultType": "vector", 06:34:32 DEBUG "result": [ 06:34:32 DEBUG { 06:34:32 DEBUG "metric": { 06:34:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:34:32 DEBUG }, 06:34:32 DEBUG "value": [ 06:34:32 DEBUG 06:34:32 DEBUG 06:34:32 DEBUG 06:34:32 DEBUG [...] 06:34:32 DEBUG 06:34:32 DEBUG 06:34:32 DEBUG 06:34:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:34:32 DEBUG }, 06:34:32 DEBUG "value": [ 06:34:32 DEBUG 1712727272, 06:34:32 DEBUG "2488291328" 06:34:32 DEBUG ] 06:34:32 DEBUG } 06:34:32 DEBUG ] 06:34:32 DEBUG } 06:34:32 DEBUG } 06:34:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727272" 06:34:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727272" 06:34:32 INFO [http_cmd]: http status code OK 06:34:32 DEBUG --- status code --- 06:34:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:32 DEBUG --- http response --- 06:34:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:34:32 DEBUG ----- output ----- 06:34:32 DEBUG { 06:34:32 DEBUG "status": "success", 06:34:32 DEBUG "data": { 06:34:32 DEBUG "resultType": "vector", 06:34:32 DEBUG "result": [ 06:34:32 DEBUG { 06:34:32 DEBUG "metric": { 06:34:32 DEBUG "instance": "10.142.0.25:9100" 06:34:32 DEBUG }, 06:34:32 DEBUG "value": [ 06:34:32 DEBUG 06:34:32 DEBUG 06:34:32 DEBUG 06:34:32 DEBUG [...] 06:34:32 DEBUG 06:34:32 DEBUG 06:34:32 DEBUG 06:34:32 DEBUG "instance": "10.142.0.77:9100" 06:34:32 DEBUG }, 06:34:32 DEBUG "value": [ 06:34:32 DEBUG 1712727272, 06:34:32 DEBUG "42572001.766666666" 06:34:32 DEBUG ] 06:34:32 DEBUG } 06:34:32 DEBUG ] 06:34:32 DEBUG } 06:34:32 DEBUG } 06:34:32 INFO [http_cmd]: http status code OK 06:34:32 DEBUG --- status code --- 06:34:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:32 DEBUG --- http response --- 06:34:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:34:32 DEBUG ----- output ----- 06:34:32 DEBUG { 06:34:32 DEBUG "status": "success", 06:34:32 DEBUG "data": { 06:34:32 DEBUG "resultType": "vector", 06:34:32 DEBUG "result": [ 06:34:32 DEBUG { 06:34:32 DEBUG "metric": { 06:34:32 DEBUG "instance": "10.142.0.25:9100" 06:34:32 DEBUG }, 06:34:32 DEBUG "value": [ 06:34:32 DEBUG 06:34:32 DEBUG 06:34:32 DEBUG 06:34:32 DEBUG [...] 06:34:32 DEBUG 06:34:32 DEBUG 06:34:32 DEBUG 06:34:32 DEBUG "instance": "10.142.0.77:9100" 06:34:32 DEBUG }, 06:34:32 DEBUG "value": [ 06:34:32 DEBUG 1712727272, 06:34:32 DEBUG "0.05766666666666609" 06:34:32 DEBUG ] 06:34:32 DEBUG } 06:34:32 DEBUG ] 06:34:32 DEBUG } 06:34:32 DEBUG } 06:34:52 INFO 06:34:52 INFO [loop_until]: kubectl --namespace=xlou top pods 06:34:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:34:52 INFO [loop_until]: OK (rc = 0) 06:34:52 DEBUG --- stdout --- 06:34:52 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 11029m 9019Mi am-654c4fcbcf-9scrj 10120m 8459Mi am-654c4fcbcf-v5qb5 8446m 8492Mi ds-cts-0 8237m 12975Mi ds-cts-1 8480m 13396Mi ds-cts-2 7516m 13297Mi ds-idrepo-0 11023m 5634Mi ds-idrepo-1 351m 5672Mi ds-idrepo-2 375m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1191Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 3m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4387m 2591Mi 06:34:52 DEBUG --- stderr --- 06:34:52 DEBUG 06:35:00 INFO 06:35:00 INFO [loop_until]: kubectl --namespace=xlou top node 06:35:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:35:00 INFO [loop_until]: OK (rc = 0) 06:35:00 DEBUG --- stdout --- 06:35:00 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 96m 0% 2309Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9278m 58% 9363Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 81m 0% 3255Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13782m 86% 6890Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10969m 69% 9340Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11936m 75% 9834Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 80m 0% 857Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10941m 68% 6318Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8349m 52% 13659Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 7461m 46% 13170Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 475m 2% 6396Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8442m 53% 14106Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 409m 2% 6362Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4390m 27% 6177Mi 10% 06:35:00 DEBUG --- stderr --- 06:35:00 DEBUG 06:35:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712727329" 06:35:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727329" 06:35:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727329" 06:35:29 ERROR [http_cmd]: ERROR 06:35:29 INFO [http_cmd]: http status code OK 06:35:29 ERROR --- status code --- 06:35:29 INFO [http_cmd]: http status code OK 06:35:29 DEBUG --- status code --- 06:35:29 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:35:29 DEBUG --- status code --- 06:35:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:29 ERROR --- http response --- 06:35:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:29 DEBUG --- http response --- 06:35:29 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:35:29 DEBUG --- http response --- 06:35:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:35:29 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:35:29 DEBUG ----- output ----- 06:35:29 DEBUG { 06:35:29 DEBUG "status": "success", 06:35:29 DEBUG "data": { 06:35:29 DEBUG "resultType": "vector", 06:35:29 DEBUG "result": [ 06:35:29 DEBUG { 06:35:29 DEBUG "metric": { 06:35:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:35:29 DEBUG }, 06:35:29 DEBUG "value": [ 06:35:29 DEBUG 06:35:29 DEBUG 06:35:29 DEBUG 06:35:29 DEBUG [...] 06:35:29 DEBUG 06:35:29 DEBUG 06:35:29 DEBUG 06:35:29 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:35:29 DEBUG }, 06:35:29 DEBUG "value": [ 06:35:29 DEBUG 1712727329, 06:35:29 DEBUG "0" 06:35:29 DEBUG ] 06:35:29 DEBUG } 06:35:29 DEBUG ] 06:35:29 DEBUG } 06:35:29 DEBUG } 06:35:29 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:35:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727330" 06:35:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727330" 06:35:30 INFO [http_cmd]: http status code OK 06:35:30 DEBUG --- status code --- 06:35:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:30 DEBUG --- http response --- 06:35:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:35:30 DEBUG ----- output ----- 06:35:30 DEBUG { 06:35:30 DEBUG "status": "success", 06:35:30 DEBUG "data": { 06:35:30 DEBUG "resultType": "vector", 06:35:30 DEBUG "result": [ 06:35:30 DEBUG { 06:35:30 DEBUG "metric": { 06:35:30 DEBUG "instance": "10.142.0.25:9100" 06:35:30 DEBUG }, 06:35:30 DEBUG "value": [ 06:35:30 DEBUG 06:35:30 DEBUG 06:35:30 DEBUG 06:35:30 DEBUG [...] 06:35:30 DEBUG 06:35:30 DEBUG 06:35:30 DEBUG 06:35:30 DEBUG "instance": "10.142.0.77:9100" 06:35:30 DEBUG }, 06:35:30 DEBUG "value": [ 06:35:30 DEBUG 1712727330, 06:35:30 DEBUG "42686945.266666666" 06:35:30 DEBUG ] 06:35:30 DEBUG } 06:35:30 DEBUG ] 06:35:30 DEBUG } 06:35:30 DEBUG } 06:35:30 INFO [http_cmd]: http status code OK 06:35:30 DEBUG --- status code --- 06:35:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:30 DEBUG --- http response --- 06:35:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:35:30 DEBUG ----- output ----- 06:35:30 DEBUG { 06:35:30 DEBUG "status": "success", 06:35:30 DEBUG "data": { 06:35:30 DEBUG "resultType": "vector", 06:35:30 DEBUG "result": [ 06:35:30 DEBUG { 06:35:30 DEBUG "metric": { 06:35:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:35:30 DEBUG }, 06:35:30 DEBUG "value": [ 06:35:30 DEBUG 06:35:30 DEBUG 06:35:30 DEBUG 06:35:30 DEBUG [...] 06:35:30 DEBUG 06:35:30 DEBUG 06:35:30 DEBUG 06:35:30 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:35:30 DEBUG }, 06:35:30 DEBUG "value": [ 06:35:30 DEBUG 1712727330, 06:35:30 DEBUG "1402.3666666666666" 06:35:30 DEBUG ] 06:35:30 DEBUG } 06:35:30 DEBUG ] 06:35:30 DEBUG } 06:35:30 DEBUG } 06:35:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727330" 06:35:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727330" 06:35:30 INFO [http_cmd]: http status code OK 06:35:30 INFO [http_cmd]: http status code OK 06:35:30 DEBUG --- status code --- 06:35:30 DEBUG --- status code --- 06:35:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:30 DEBUG --- http response --- 06:35:30 DEBUG --- http response --- 06:35:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:35:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:35:30 DEBUG ----- output ----- 06:35:30 DEBUG ----- output ----- 06:35:30 DEBUG { 06:35:30 DEBUG { 06:35:30 DEBUG "status": "success", 06:35:30 DEBUG "status": "success", 06:35:30 DEBUG "data": { 06:35:30 DEBUG "resultType": "vector", 06:35:30 DEBUG "result": [ 06:35:30 DEBUG { 06:35:30 DEBUG "data": { 06:35:30 DEBUG "metric": { 06:35:30 DEBUG "resultType": "vector", 06:35:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:35:30 DEBUG "result": [ 06:35:30 DEBUG }, 06:35:30 DEBUG "value": [ 06:35:30 DEBUG 06:35:30 DEBUG 06:35:30 DEBUG 06:35:30 DEBUG [...] 06:35:30 DEBUG { 06:35:30 DEBUG 06:35:30 DEBUG 06:35:30 DEBUG "metric": { 06:35:30 DEBUG 06:35:30 DEBUG "pod": "ds-idrepo-0" 06:35:30 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:35:30 DEBUG }, 06:35:30 DEBUG }, 06:35:30 DEBUG "value": [ 06:35:30 DEBUG 06:35:30 DEBUG 06:35:30 DEBUG 06:35:30 DEBUG [...] 06:35:30 DEBUG "value": [ 06:35:30 DEBUG 06:35:30 DEBUG 1712727330, 06:35:30 DEBUG 06:35:30 DEBUG "1397" 06:35:30 DEBUG 06:35:30 DEBUG ] 06:35:30 DEBUG "pod": "ds-idrepo-1" 06:35:30 DEBUG }, 06:35:30 DEBUG } 06:35:30 DEBUG "value": [ 06:35:30 DEBUG ] 06:35:30 DEBUG 1712727330, 06:35:30 DEBUG } 06:35:30 DEBUG "0" 06:35:30 DEBUG } 06:35:30 DEBUG ] 06:35:30 DEBUG } 06:35:30 DEBUG ] 06:35:30 DEBUG } 06:35:30 DEBUG } 06:35:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727330" 06:35:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727330" 06:35:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727330" 06:35:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727330" 06:35:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727330" 06:35:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727330" 06:35:30 INFO [http_cmd]: http status code OK 06:35:30 DEBUG --- status code --- 06:35:30 INFO [http_cmd]: http status code OK 06:35:30 INFO [http_cmd]: http status code OK 06:35:30 INFO [http_cmd]: http status code OK 06:35:30 INFO [http_cmd]: http status code OK 06:35:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:30 DEBUG --- status code --- 06:35:30 DEBUG --- status code --- 06:35:30 DEBUG --- status code --- 06:35:30 DEBUG --- status code --- 06:35:30 DEBUG --- http response --- 06:35:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:35:30 DEBUG --- http response --- 06:35:30 DEBUG --- http response --- 06:35:30 DEBUG --- http response --- 06:35:30 DEBUG --- http response --- 06:35:30 DEBUG ----- output ----- 06:35:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:35:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:35:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:35:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:35:30 DEBUG { 06:35:30 DEBUG ----- output ----- 06:35:30 DEBUG ----- output ----- 06:35:30 DEBUG ----- output ----- 06:35:30 DEBUG ----- output ----- 06:35:30 DEBUG "status": "success", 06:35:30 DEBUG "data": { 06:35:30 DEBUG { 06:35:30 DEBUG { 06:35:30 DEBUG "status": "success", 06:35:30 DEBUG { 06:35:30 DEBUG "status": "success", 06:35:30 DEBUG "status": "success", 06:35:30 DEBUG "data": { 06:35:30 DEBUG "resultType": "vector", 06:35:30 DEBUG "resultType": "vector", 06:35:30 INFO [http_cmd]: http status code OK 06:35:30 DEBUG "data": { 06:35:30 DEBUG "data": { 06:35:30 DEBUG { 06:35:30 DEBUG "result": [ 06:35:30 DEBUG "result": [ 06:35:30 DEBUG --- status code --- 06:35:30 DEBUG "resultType": "vector", 06:35:30 DEBUG "resultType": "vector", 06:35:30 DEBUG "status": "success", 06:35:30 DEBUG "data": { 06:35:30 DEBUG { 06:35:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:30 DEBUG "result": [ 06:35:30 DEBUG "result": [ 06:35:30 DEBUG { 06:35:30 DEBUG "metric": { 06:35:30 DEBUG "metric": { 06:35:30 DEBUG --- http response --- 06:35:30 DEBUG { 06:35:30 DEBUG { 06:35:30 DEBUG "resultType": "vector", 06:35:30 DEBUG "result": [ 06:35:30 DEBUG "instance": "10.142.0.25:9100" 06:35:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:35:30 DEBUG "metric": { 06:35:30 DEBUG "metric": { 06:35:30 DEBUG "pod": "ds-idrepo-0" 06:35:30 DEBUG { 06:35:30 DEBUG }, 06:35:30 DEBUG ----- output ----- 06:35:30 DEBUG "pod": "ds-idrepo-0" 06:35:30 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:35:30 DEBUG }, 06:35:30 DEBUG "metric": { 06:35:30 DEBUG "value": [ 06:35:30 DEBUG { 06:35:30 DEBUG }, 06:35:30 DEBUG }, 06:35:30 DEBUG "value": [ 06:35:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:35:30 DEBUG 06:35:30 DEBUG "status": "success", 06:35:30 DEBUG "value": [ 06:35:30 DEBUG "value": [ 06:35:30 DEBUG 06:35:30 DEBUG }, 06:35:30 DEBUG "value": [ 06:35:30 DEBUG "data": { 06:35:30 DEBUG "resultType": "vector", 06:35:30 DEBUG "result": [ 06:35:30 DEBUG { 06:35:30 DEBUG "metric": { 06:35:30 DEBUG 06:35:30 DEBUG 06:35:30 DEBUG "instance": "10.142.0.25:9100" 06:35:30 DEBUG 06:35:30 DEBUG 06:35:30 DEBUG [...] 06:35:30 DEBUG 06:35:30 DEBUG [...] 06:35:30 DEBUG 06:35:30 DEBUG 06:35:30 DEBUG 06:35:30 DEBUG 06:35:30 DEBUG }, 06:35:30 DEBUG "value": [ 06:35:30 DEBUG 06:35:30 DEBUG 06:35:30 DEBUG [...] 06:35:30 DEBUG 06:35:30 DEBUG 06:35:30 DEBUG 06:35:30 DEBUG [...] 06:35:30 DEBUG 06:35:30 DEBUG 06:35:30 DEBUG 06:35:30 DEBUG "pod": "ds-cts-2" 06:35:30 DEBUG 06:35:30 DEBUG 06:35:30 DEBUG 06:35:30 DEBUG 06:35:30 DEBUG 06:35:30 DEBUG }, 06:35:30 DEBUG 06:35:30 DEBUG "instance": "10.142.0.77:9100" 06:35:30 DEBUG }, 06:35:30 DEBUG "value": [ 06:35:30 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:35:30 DEBUG "value": [ 06:35:30 DEBUG [...] 06:35:30 DEBUG 06:35:30 DEBUG 1712727330, 06:35:30 DEBUG 06:35:30 DEBUG }, 06:35:30 DEBUG 1712727330, 06:35:30 DEBUG 06:35:30 DEBUG 06:35:30 DEBUG "0.0386333333333333" 06:35:30 DEBUG 06:35:30 DEBUG "value": [ 06:35:30 DEBUG "29.53333333333333" 06:35:30 DEBUG 06:35:30 DEBUG [...] 06:35:30 DEBUG ] 06:35:30 DEBUG 06:35:30 DEBUG 1712727330, 06:35:30 DEBUG ] 06:35:30 DEBUG 06:35:30 DEBUG 06:35:30 DEBUG } 06:35:30 DEBUG "pod": "ds-cts-2" 06:35:30 DEBUG "251216" 06:35:30 DEBUG } 06:35:30 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:35:30 DEBUG 06:35:30 DEBUG ] 06:35:30 DEBUG }, 06:35:30 DEBUG ] 06:35:30 DEBUG ] 06:35:30 DEBUG }, 06:35:30 DEBUG 06:35:30 DEBUG } 06:35:30 DEBUG "value": [ 06:35:30 DEBUG } 06:35:30 DEBUG } 06:35:30 DEBUG "value": [ 06:35:30 DEBUG "instance": "10.142.0.77:9100" 06:35:30 DEBUG } 06:35:30 DEBUG 1712727330, 06:35:30 DEBUG "0" 06:35:30 DEBUG } 06:35:30 DEBUG 1712727330, 06:35:30 DEBUG }, 06:35:30 DEBUG ] 06:35:30 DEBUG ] 06:35:30 DEBUG "0" 06:35:30 DEBUG "value": [ 06:35:30 DEBUG } 06:35:30 DEBUG } 06:35:30 DEBUG ] 06:35:30 DEBUG 1712727330, 06:35:30 DEBUG } 06:35:30 DEBUG } 06:35:30 DEBUG ] 06:35:30 DEBUG ] 06:35:30 DEBUG } 06:35:30 DEBUG } 06:35:30 DEBUG } 06:35:30 DEBUG } 06:35:30 DEBUG "412865942528" 06:35:31 DEBUG ] 06:35:31 DEBUG } 06:35:31 DEBUG ] 06:35:31 DEBUG } 06:35:31 DEBUG } 06:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727331" 06:35:31 INFO [http_cmd]: http status code OK 06:35:31 DEBUG --- status code --- 06:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:31 DEBUG --- http response --- 06:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:35:31 DEBUG ----- output ----- 06:35:31 DEBUG { 06:35:31 DEBUG "status": "success", 06:35:31 DEBUG "data": { 06:35:31 DEBUG "resultType": "vector", 06:35:31 DEBUG "result": [ 06:35:31 DEBUG { 06:35:31 DEBUG "metric": { 06:35:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:35:31 DEBUG }, 06:35:31 DEBUG "value": [ 06:35:31 DEBUG 06:35:31 DEBUG 06:35:31 DEBUG 06:35:31 DEBUG [...] 06:35:31 DEBUG 06:35:31 DEBUG 06:35:31 DEBUG 06:35:31 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:35:31 DEBUG }, 06:35:31 DEBUG "value": [ 06:35:31 DEBUG 1712727331, 06:35:31 DEBUG "4732.41521995593" 06:35:31 DEBUG ] 06:35:31 DEBUG } 06:35:31 DEBUG ] 06:35:31 DEBUG } 06:35:31 DEBUG } 06:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712727331" 06:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712727331" 06:35:31 INFO [http_cmd]: http status code OK 06:35:31 DEBUG --- status code --- 06:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:31 DEBUG --- http response --- 06:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:35:31 DEBUG ----- output ----- 06:35:31 DEBUG { 06:35:31 DEBUG "status": "success", 06:35:31 DEBUG "data": { 06:35:31 DEBUG "resultType": "vector", 06:35:31 DEBUG "result": [ 06:35:31 DEBUG { 06:35:31 DEBUG "metric": { 06:35:31 DEBUG "pod": "ds-idrepo-0" 06:35:31 DEBUG }, 06:35:31 DEBUG "value": [ 06:35:31 DEBUG 06:35:31 DEBUG 06:35:31 DEBUG 06:35:31 DEBUG [...] 06:35:31 DEBUG 06:35:31 DEBUG 06:35:31 DEBUG 06:35:31 DEBUG "pod": "ds-cts-2" 06:35:31 DEBUG }, 06:35:31 DEBUG "value": [ 06:35:31 DEBUG 1712727331, 06:35:31 DEBUG "0" 06:35:31 DEBUG ] 06:35:31 DEBUG } 06:35:31 DEBUG ] 06:35:31 DEBUG } 06:35:31 DEBUG } 06:35:31 INFO [http_cmd]: http status code OK 06:35:31 DEBUG --- status code --- 06:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:31 DEBUG --- http response --- 06:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:35:31 DEBUG ----- output ----- 06:35:31 DEBUG { 06:35:31 DEBUG "status": "success", 06:35:31 DEBUG "data": { 06:35:31 DEBUG "resultType": "vector", 06:35:31 DEBUG "result": [ 06:35:31 DEBUG { 06:35:31 DEBUG "metric": { 06:35:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:35:31 DEBUG }, 06:35:31 DEBUG "value": [ 06:35:31 DEBUG 06:35:31 DEBUG 06:35:31 DEBUG 06:35:31 DEBUG [...] 06:35:31 DEBUG 06:35:31 DEBUG 06:35:31 DEBUG 06:35:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:35:31 DEBUG }, 06:35:31 DEBUG "value": [ 06:35:31 DEBUG 1712727331, 06:35:31 DEBUG "4.417155562795495" 06:35:31 DEBUG ] 06:35:31 DEBUG } 06:35:31 DEBUG ] 06:35:31 DEBUG } 06:35:31 DEBUG } 06:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712727331" 06:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727331" 06:35:31 INFO [http_cmd]: http status code OK 06:35:31 DEBUG --- status code --- 06:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:31 DEBUG --- http response --- 06:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727331" 06:35:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727331" 06:35:31 DEBUG ----- output ----- 06:35:31 INFO [http_cmd]: http status code OK 06:35:31 DEBUG --- status code --- 06:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:31 DEBUG --- http response --- 06:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:35:31 DEBUG ----- output ----- 06:35:31 DEBUG { 06:35:31 DEBUG "status": "success", 06:35:31 DEBUG "data": { 06:35:31 DEBUG "resultType": "vector", 06:35:31 DEBUG "result": [ 06:35:31 DEBUG { 06:35:31 DEBUG "metric": { 06:35:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:35:31 DEBUG }, 06:35:31 DEBUG "value": [ 06:35:31 DEBUG 06:35:31 DEBUG 06:35:31 DEBUG { 06:35:31 DEBUG "status": "success", 06:35:31 DEBUG 06:35:31 DEBUG [...] 06:35:31 DEBUG 06:35:31 DEBUG 06:35:31 DEBUG "data": { 06:35:31 DEBUG "resultType": "vector", 06:35:31 DEBUG "result": [ 06:35:31 DEBUG { 06:35:31 DEBUG "metric": { 06:35:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:35:31 DEBUG 06:35:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:35:31 DEBUG }, 06:35:31 DEBUG "value": [ 06:35:31 DEBUG }, 06:35:31 DEBUG "value": [ 06:35:31 DEBUG 06:35:31 DEBUG 06:35:31 DEBUG 1712727331, 06:35:31 DEBUG "1402.9" 06:35:31 DEBUG ] 06:35:31 DEBUG } 06:35:31 DEBUG ] 06:35:31 DEBUG } 06:35:31 DEBUG } 06:35:31 DEBUG 06:35:31 DEBUG [...] 06:35:31 DEBUG 06:35:31 DEBUG 06:35:31 DEBUG 06:35:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:35:31 DEBUG }, 06:35:31 DEBUG "value": [ 06:35:31 DEBUG 1712727331, 06:35:31 DEBUG "0.053" 06:35:31 DEBUG ] 06:35:31 DEBUG } 06:35:31 DEBUG ] 06:35:31 DEBUG } 06:35:31 DEBUG } 06:35:31 INFO [http_cmd]: http status code OK 06:35:31 DEBUG --- status code --- 06:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:31 DEBUG --- http response --- 06:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:35:31 DEBUG ----- output ----- 06:35:31 DEBUG { 06:35:31 DEBUG "status": "success", 06:35:31 DEBUG "data": { 06:35:31 DEBUG "resultType": "vector", 06:35:31 DEBUG "result": [ 06:35:31 DEBUG { 06:35:31 DEBUG "metric": { 06:35:31 DEBUG "pod": "ds-idrepo-2" 06:35:31 DEBUG }, 06:35:31 DEBUG "value": [ 06:35:31 DEBUG 06:35:31 DEBUG 06:35:31 DEBUG 06:35:31 DEBUG [...] 06:35:31 DEBUG 06:35:31 DEBUG 06:35:31 DEBUG 06:35:31 DEBUG "pod": "ds-cts-2" 06:35:31 DEBUG }, 06:35:31 DEBUG "value": [ 06:35:31 DEBUG 1712727331, 06:35:31 DEBUG "0.0003" 06:35:31 DEBUG ] 06:35:31 DEBUG } 06:35:31 DEBUG ] 06:35:31 DEBUG } 06:35:31 DEBUG } 06:35:31 INFO [http_cmd]: http status code OK 06:35:31 DEBUG --- status code --- 06:35:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:31 DEBUG --- http response --- 06:35:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:35:31 DEBUG ----- output ----- 06:35:31 DEBUG { 06:35:31 DEBUG "status": "success", 06:35:31 DEBUG "data": { 06:35:31 DEBUG "resultType": "vector", 06:35:31 DEBUG "result": [ 06:35:31 DEBUG { 06:35:31 DEBUG "metric": { 06:35:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:35:31 DEBUG }, 06:35:31 DEBUG "value": [ 06:35:31 DEBUG 06:35:31 DEBUG 06:35:31 DEBUG 06:35:31 DEBUG [...] 06:35:31 DEBUG 06:35:31 DEBUG 06:35:31 DEBUG 06:35:31 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:35:31 DEBUG }, 06:35:31 DEBUG "value": [ 06:35:31 DEBUG 1712727331, 06:35:31 DEBUG "0.005923304358240256" 06:35:31 DEBUG ] 06:35:31 DEBUG } 06:35:31 DEBUG ] 06:35:31 DEBUG } 06:35:31 DEBUG } 06:35:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727332" 06:35:32 INFO [http_cmd]: http status code OK 06:35:32 DEBUG --- status code --- 06:35:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:32 DEBUG --- http response --- 06:35:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:35:32 DEBUG ----- output ----- 06:35:32 DEBUG { 06:35:32 DEBUG "status": "success", 06:35:32 DEBUG "data": { 06:35:32 DEBUG "resultType": "vector", 06:35:32 DEBUG "result": [ 06:35:32 DEBUG { 06:35:32 DEBUG "metric": { 06:35:32 DEBUG "pod": "ds-idrepo-0" 06:35:32 DEBUG }, 06:35:32 DEBUG "value": [ 06:35:32 DEBUG 06:35:32 DEBUG 06:35:32 DEBUG 06:35:32 DEBUG [...] 06:35:32 DEBUG 06:35:32 DEBUG 06:35:32 DEBUG 06:35:32 DEBUG "pod": "ds-idrepo-1" 06:35:32 DEBUG }, 06:35:32 DEBUG "value": [ 06:35:32 DEBUG 1712727332, 06:35:32 DEBUG "0" 06:35:32 DEBUG ] 06:35:32 DEBUG } 06:35:32 DEBUG ] 06:35:32 DEBUG } 06:35:32 DEBUG } 06:35:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727332" 06:35:32 INFO [http_cmd]: http status code OK 06:35:32 DEBUG --- status code --- 06:35:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:32 DEBUG --- http response --- 06:35:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:35:32 DEBUG ----- output ----- 06:35:32 DEBUG { 06:35:32 DEBUG "status": "success", 06:35:32 DEBUG "data": { 06:35:32 DEBUG "resultType": "vector", 06:35:32 DEBUG "result": [ 06:35:32 DEBUG { 06:35:32 DEBUG "metric": { 06:35:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:35:32 DEBUG }, 06:35:32 DEBUG "value": [ 06:35:32 DEBUG 06:35:32 DEBUG 06:35:32 DEBUG 06:35:32 DEBUG [...] 06:35:32 DEBUG 06:35:32 DEBUG 06:35:32 DEBUG 06:35:32 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:35:32 DEBUG }, 06:35:32 DEBUG "value": [ 06:35:32 DEBUG 1712727332, 06:35:32 DEBUG "1389.3" 06:35:32 DEBUG ] 06:35:32 DEBUG } 06:35:32 DEBUG ] 06:35:32 DEBUG } 06:35:32 DEBUG } 06:35:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727332" 06:35:32 INFO [http_cmd]: http status code OK 06:35:32 DEBUG --- status code --- 06:35:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:32 DEBUG --- http response --- 06:35:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:35:32 DEBUG ----- output ----- 06:35:32 DEBUG { 06:35:32 DEBUG "status": "success", 06:35:32 DEBUG "data": { 06:35:32 DEBUG "resultType": "vector", 06:35:32 DEBUG "result": [ 06:35:32 DEBUG { 06:35:32 DEBUG "metric": { 06:35:32 DEBUG "pod": "ds-idrepo-0" 06:35:32 DEBUG }, 06:35:32 DEBUG "value": [ 06:35:32 DEBUG 06:35:32 DEBUG 06:35:32 DEBUG 06:35:32 DEBUG [...] 06:35:32 DEBUG 06:35:32 DEBUG 06:35:32 DEBUG 06:35:32 DEBUG "pod": "ds-idrepo-1" 06:35:32 DEBUG }, 06:35:32 DEBUG "value": [ 06:35:32 DEBUG 1712727332, 06:35:32 DEBUG "0" 06:35:32 DEBUG ] 06:35:32 DEBUG } 06:35:32 DEBUG ] 06:35:32 DEBUG } 06:35:32 DEBUG } 06:35:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727332" 06:35:32 INFO [http_cmd]: http status code OK 06:35:32 DEBUG --- status code --- 06:35:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:32 DEBUG --- http response --- 06:35:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:35:32 DEBUG ----- output ----- 06:35:32 DEBUG { 06:35:32 DEBUG "status": "success", 06:35:32 DEBUG "data": { 06:35:32 DEBUG "resultType": "vector", 06:35:32 DEBUG "result": [ 06:35:32 DEBUG { 06:35:32 DEBUG "metric": { 06:35:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:35:32 DEBUG }, 06:35:32 DEBUG "value": [ 06:35:32 DEBUG 06:35:32 DEBUG 06:35:32 DEBUG 06:35:32 DEBUG [...] 06:35:32 DEBUG 06:35:32 DEBUG 06:35:32 DEBUG 06:35:32 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:35:32 DEBUG }, 06:35:32 DEBUG "value": [ 06:35:32 DEBUG 1712727332, 06:35:32 DEBUG "0" 06:35:32 DEBUG ] 06:35:32 DEBUG } 06:35:32 DEBUG ] 06:35:32 DEBUG } 06:35:32 DEBUG } 06:35:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712727332" 06:35:32 INFO [http_cmd]: http status code OK 06:35:32 DEBUG --- status code --- 06:35:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:32 DEBUG --- http response --- 06:35:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:35:32 DEBUG ----- output ----- 06:35:32 DEBUG { 06:35:32 DEBUG "status": "success", 06:35:32 DEBUG "data": { 06:35:32 DEBUG "resultType": "vector", 06:35:32 DEBUG "result": [ 06:35:32 DEBUG { 06:35:32 DEBUG "metric": { 06:35:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:35:32 DEBUG }, 06:35:32 DEBUG "value": [ 06:35:32 DEBUG 06:35:32 DEBUG 06:35:32 DEBUG 06:35:32 DEBUG [...] 06:35:32 DEBUG 06:35:32 DEBUG 06:35:32 DEBUG 06:35:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:35:32 DEBUG }, 06:35:32 DEBUG "value": [ 06:35:32 DEBUG 1712727332, 06:35:32 DEBUG "2486669312" 06:35:32 DEBUG ] 06:35:32 DEBUG } 06:35:32 DEBUG ] 06:35:32 DEBUG } 06:35:32 DEBUG } 06:35:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727332" 06:35:32 INFO [http_cmd]: http status code OK 06:35:32 DEBUG --- status code --- 06:35:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727332" 06:35:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:32 DEBUG --- http response --- 06:35:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:35:32 DEBUG ----- output ----- 06:35:32 DEBUG { 06:35:32 DEBUG "status": "success", 06:35:32 DEBUG "data": { 06:35:32 DEBUG "resultType": "vector", 06:35:32 DEBUG "result": [ 06:35:32 DEBUG { 06:35:32 DEBUG "metric": { 06:35:32 DEBUG "instance": "10.142.0.25:9100" 06:35:32 DEBUG }, 06:35:32 DEBUG "value": [ 06:35:32 DEBUG 06:35:32 DEBUG 06:35:32 DEBUG 06:35:32 DEBUG [...] 06:35:32 DEBUG 06:35:32 DEBUG 06:35:32 DEBUG 06:35:32 DEBUG "instance": "10.142.0.77:9100" 06:35:32 DEBUG }, 06:35:32 DEBUG "value": [ 06:35:32 DEBUG 1712727332, 06:35:32 DEBUG "42688902.166666664" 06:35:32 DEBUG ] 06:35:32 DEBUG } 06:35:32 DEBUG ] 06:35:32 DEBUG } 06:35:32 DEBUG } 06:35:32 INFO [http_cmd]: http status code OK 06:35:32 DEBUG --- status code --- 06:35:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:32 DEBUG --- http response --- 06:35:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:35:32 DEBUG ----- output ----- 06:35:32 DEBUG { 06:35:32 DEBUG "status": "success", 06:35:32 DEBUG "data": { 06:35:32 DEBUG "resultType": "vector", 06:35:32 DEBUG "result": [ 06:35:32 DEBUG { 06:35:32 DEBUG "metric": { 06:35:32 DEBUG "instance": "10.142.0.25:9100" 06:35:32 DEBUG }, 06:35:32 DEBUG "value": [ 06:35:32 DEBUG 06:35:32 DEBUG 06:35:32 DEBUG 06:35:32 DEBUG [...] 06:35:32 DEBUG 06:35:32 DEBUG 06:35:32 DEBUG 06:35:32 DEBUG "instance": "10.142.0.77:9100" 06:35:32 DEBUG }, 06:35:32 DEBUG "value": [ 06:35:32 DEBUG 1712727332, 06:35:32 DEBUG "0.017333333333333673" 06:35:32 DEBUG ] 06:35:32 DEBUG } 06:35:32 DEBUG ] 06:35:32 DEBUG } 06:35:32 DEBUG } 06:35:52 INFO 06:35:52 INFO [loop_until]: kubectl --namespace=xlou top pods 06:35:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:35:52 INFO [loop_until]: OK (rc = 0) 06:35:52 DEBUG --- stdout --- 06:35:52 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 11010m 9020Mi am-654c4fcbcf-9scrj 10116m 8460Mi am-654c4fcbcf-v5qb5 8420m 8489Mi ds-cts-0 8283m 13393Mi ds-cts-1 8327m 13022Mi ds-cts-2 7365m 13076Mi ds-idrepo-0 10659m 5643Mi ds-idrepo-1 355m 5672Mi ds-idrepo-2 401m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1191Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 8m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4289m 2631Mi 06:35:52 DEBUG --- stderr --- 06:35:52 DEBUG 06:36:00 INFO 06:36:00 INFO [loop_until]: kubectl --namespace=xlou top node 06:36:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:36:00 INFO [loop_until]: OK (rc = 0) 06:36:00 DEBUG --- stdout --- 06:36:00 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 96m 0% 2308Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9269m 58% 9366Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 80m 0% 3257Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 14187m 89% 6888Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10965m 69% 9339Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11964m 75% 9842Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 82m 0% 857Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10595m 66% 6329Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8375m 52% 14091Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 7585m 47% 13892Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 434m 2% 6395Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8480m 53% 13626Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 414m 2% 6359Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4505m 28% 6213Mi 10% 06:36:00 DEBUG --- stderr --- 06:36:00 DEBUG 06:36:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712727389" 06:36:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727389" 06:36:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727389" 06:36:29 ERROR [http_cmd]: ERROR 06:36:29 ERROR --- status code --- 06:36:29 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:36:29 ERROR --- http response --- 06:36:29 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:36:29 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:36:29 INFO [http_cmd]: http status code OK 06:36:29 DEBUG --- status code --- 06:36:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:29 DEBUG --- http response --- 06:36:29 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:36:29 INFO [http_cmd]: http status code OK 06:36:29 DEBUG --- status code --- 06:36:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:29 DEBUG --- http response --- 06:36:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:36:29 DEBUG ----- output ----- 06:36:29 DEBUG { 06:36:29 DEBUG "status": "success", 06:36:29 DEBUG "data": { 06:36:29 DEBUG "resultType": "vector", 06:36:29 DEBUG "result": [ 06:36:29 DEBUG { 06:36:29 DEBUG "metric": { 06:36:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:36:29 DEBUG }, 06:36:29 DEBUG "value": [ 06:36:29 DEBUG 06:36:29 DEBUG 06:36:29 DEBUG 06:36:29 DEBUG [...] 06:36:29 DEBUG 06:36:29 DEBUG 06:36:29 DEBUG 06:36:29 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:36:29 DEBUG }, 06:36:29 DEBUG "value": [ 06:36:29 DEBUG 1712727389, 06:36:29 DEBUG "0" 06:36:29 DEBUG ] 06:36:29 DEBUG } 06:36:29 DEBUG ] 06:36:29 DEBUG } 06:36:29 DEBUG } 06:36:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727390" 06:36:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727390" 06:36:30 INFO [http_cmd]: http status code OK 06:36:30 DEBUG --- status code --- 06:36:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:30 DEBUG --- http response --- 06:36:30 INFO [http_cmd]: http status code OK 06:36:30 DEBUG --- status code --- 06:36:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:30 DEBUG --- http response --- 06:36:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:36:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:36:30 DEBUG ----- output ----- 06:36:30 DEBUG ----- output ----- 06:36:30 DEBUG { 06:36:30 DEBUG { 06:36:30 DEBUG "status": "success", 06:36:30 DEBUG "status": "success", 06:36:30 DEBUG "data": { 06:36:30 DEBUG "data": { 06:36:30 DEBUG "resultType": "vector", 06:36:30 DEBUG "resultType": "vector", 06:36:30 DEBUG "result": [ 06:36:30 DEBUG "result": [ 06:36:30 DEBUG { 06:36:30 DEBUG { 06:36:30 DEBUG "metric": { 06:36:30 DEBUG "metric": { 06:36:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:36:30 DEBUG "instance": "10.142.0.25:9100" 06:36:30 DEBUG }, 06:36:30 DEBUG }, 06:36:30 DEBUG "value": [ 06:36:30 DEBUG "value": [ 06:36:30 DEBUG 06:36:30 DEBUG 06:36:30 DEBUG 06:36:30 DEBUG 06:36:30 DEBUG 06:36:30 DEBUG 06:36:30 DEBUG [...] 06:36:30 DEBUG [...] 06:36:30 DEBUG 06:36:30 DEBUG 06:36:30 DEBUG 06:36:30 DEBUG 06:36:30 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:36:30 DEBUG }, 06:36:30 DEBUG "value": [ 06:36:30 DEBUG 1712727390, 06:36:30 DEBUG "1381.9333333333334" 06:36:30 DEBUG ] 06:36:30 DEBUG } 06:36:30 DEBUG ] 06:36:30 DEBUG } 06:36:30 DEBUG } 06:36:30 DEBUG 06:36:30 DEBUG 06:36:30 DEBUG "instance": "10.142.0.77:9100" 06:36:30 DEBUG }, 06:36:30 DEBUG "value": [ 06:36:30 DEBUG 1712727390, 06:36:30 DEBUG "42484736.766666666" 06:36:30 DEBUG ] 06:36:30 DEBUG } 06:36:30 DEBUG ] 06:36:30 DEBUG } 06:36:30 DEBUG } 06:36:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727390" 06:36:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727390" 06:36:31 INFO [http_cmd]: http status code OK 06:36:31 DEBUG --- status code --- 06:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:31 DEBUG --- http response --- 06:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:36:31 DEBUG ----- output ----- 06:36:31 DEBUG { 06:36:31 DEBUG "status": "success", 06:36:31 DEBUG "data": { 06:36:31 DEBUG "resultType": "vector", 06:36:31 DEBUG "result": [ 06:36:31 DEBUG { 06:36:31 DEBUG "metric": { 06:36:31 DEBUG "pod": "ds-idrepo-0" 06:36:31 DEBUG }, 06:36:31 DEBUG "value": [ 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG [...] 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG "pod": "ds-idrepo-1" 06:36:31 DEBUG }, 06:36:31 DEBUG "value": [ 06:36:31 DEBUG 1712727390, 06:36:31 DEBUG "0" 06:36:31 DEBUG ] 06:36:31 DEBUG } 06:36:31 DEBUG ] 06:36:31 DEBUG } 06:36:31 DEBUG } 06:36:31 INFO [http_cmd]: http status code OK 06:36:31 DEBUG --- status code --- 06:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:31 DEBUG --- http response --- 06:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:36:31 DEBUG ----- output ----- 06:36:31 DEBUG { 06:36:31 DEBUG "status": "success", 06:36:31 DEBUG "data": { 06:36:31 DEBUG "resultType": "vector", 06:36:31 DEBUG "result": [ 06:36:31 DEBUG { 06:36:31 DEBUG "metric": { 06:36:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:36:31 DEBUG }, 06:36:31 DEBUG "value": [ 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG [...] 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:36:31 DEBUG }, 06:36:31 DEBUG "value": [ 06:36:31 DEBUG 1712727390, 06:36:31 DEBUG "1389.3333333333333" 06:36:31 DEBUG ] 06:36:31 DEBUG } 06:36:31 DEBUG ] 06:36:31 DEBUG } 06:36:31 DEBUG } 06:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727391" 06:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727391" 06:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727391" 06:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727391" 06:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727391" 06:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727391" 06:36:31 INFO [http_cmd]: http status code OK 06:36:31 DEBUG --- status code --- 06:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:31 DEBUG --- http response --- 06:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:36:31 DEBUG ----- output ----- 06:36:31 DEBUG { 06:36:31 DEBUG "status": "success", 06:36:31 DEBUG "data": { 06:36:31 DEBUG "resultType": "vector", 06:36:31 DEBUG "result": [ 06:36:31 DEBUG { 06:36:31 DEBUG "metric": { 06:36:31 DEBUG "pod": "ds-idrepo-0" 06:36:31 DEBUG }, 06:36:31 DEBUG "value": [ 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG [...] 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG "pod": "ds-cts-2" 06:36:31 DEBUG }, 06:36:31 DEBUG "value": [ 06:36:31 DEBUG 1712727391, 06:36:31 INFO [http_cmd]: http status code OK 06:36:31 INFO [http_cmd]: http status code OK 06:36:31 DEBUG "0" 06:36:31 INFO [http_cmd]: http status code OK 06:36:31 INFO [http_cmd]: http status code OK 06:36:31 DEBUG --- status code --- 06:36:31 DEBUG --- status code --- 06:36:31 INFO [http_cmd]: http status code OK 06:36:31 DEBUG ] 06:36:31 DEBUG --- status code --- 06:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:31 DEBUG --- http response --- 06:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:31 DEBUG --- status code --- 06:36:31 DEBUG } 06:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:31 DEBUG --- status code --- 06:36:31 DEBUG --- http response --- 06:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:36:31 DEBUG ----- output ----- 06:36:31 DEBUG { 06:36:31 DEBUG --- http response --- 06:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:31 DEBUG --- http response --- 06:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:36:31 DEBUG ] 06:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:36:31 DEBUG ----- output ----- 06:36:31 DEBUG ----- output ----- 06:36:31 DEBUG { 06:36:31 DEBUG "status": "success", 06:36:31 DEBUG } 06:36:31 DEBUG { 06:36:31 DEBUG "status": "success", 06:36:31 DEBUG "data": { 06:36:31 DEBUG "resultType": "vector", 06:36:31 DEBUG "result": [ 06:36:31 DEBUG } 06:36:31 DEBUG "data": { 06:36:31 DEBUG "resultType": "vector", 06:36:31 DEBUG { 06:36:31 DEBUG "metric": { 06:36:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:36:31 DEBUG "result": [ 06:36:31 DEBUG { 06:36:31 DEBUG }, 06:36:31 DEBUG "value": [ 06:36:31 DEBUG 06:36:31 DEBUG "metric": { 06:36:31 DEBUG "instance": "10.142.0.25:9100" 06:36:31 DEBUG }, 06:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:31 DEBUG --- http response --- 06:36:31 DEBUG "value": [ 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG [...] 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:36:31 DEBUG ----- output ----- 06:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:36:31 DEBUG "instance": "10.142.0.77:9100" 06:36:31 DEBUG [...] 06:36:31 DEBUG { 06:36:31 DEBUG "status": "success", 06:36:31 DEBUG ----- output ----- 06:36:31 DEBUG }, 06:36:31 DEBUG 06:36:31 DEBUG "data": { 06:36:31 DEBUG "resultType": "vector", 06:36:31 DEBUG { 06:36:31 DEBUG "value": [ 06:36:31 DEBUG 06:36:31 DEBUG "result": [ 06:36:31 DEBUG { 06:36:31 DEBUG "status": "success", 06:36:31 DEBUG 1712727391, 06:36:31 DEBUG 06:36:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:36:31 DEBUG "status": "success", 06:36:31 DEBUG "data": { 06:36:31 DEBUG "415060767744" 06:36:31 DEBUG }, 06:36:31 DEBUG "value": [ 06:36:31 DEBUG "data": { 06:36:31 DEBUG "resultType": "vector", 06:36:31 DEBUG ] 06:36:31 DEBUG 1712727391, 06:36:31 DEBUG "250781" 06:36:31 DEBUG "resultType": "vector", 06:36:31 DEBUG "result": [ 06:36:31 DEBUG } 06:36:31 DEBUG ] 06:36:31 DEBUG } 06:36:31 DEBUG "result": [ 06:36:31 DEBUG { 06:36:31 DEBUG ] 06:36:31 DEBUG ] 06:36:31 DEBUG } 06:36:31 DEBUG { 06:36:31 DEBUG "metric": { 06:36:31 DEBUG } 06:36:31 DEBUG } 06:36:31 DEBUG "metric": { 06:36:31 DEBUG "metric": { 06:36:31 DEBUG "instance": "10.142.0.25:9100" 06:36:31 DEBUG } 06:36:31 DEBUG "pod": "ds-idrepo-0" 06:36:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:36:31 DEBUG }, 06:36:31 DEBUG "value": [ 06:36:31 DEBUG }, 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG "value": [ 06:36:31 DEBUG 06:36:31 DEBUG [...] 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG "instance": "10.142.0.77:9100" 06:36:31 DEBUG 06:36:31 DEBUG }, 06:36:31 DEBUG "value": [ 06:36:31 DEBUG [...] 06:36:31 DEBUG 1712727391, 06:36:31 DEBUG "0.15613333333333634" 06:36:31 DEBUG 06:36:31 DEBUG ] 06:36:31 DEBUG } 06:36:31 DEBUG 06:36:31 DEBUG ] 06:36:31 DEBUG } 06:36:31 DEBUG 06:36:31 DEBUG } 06:36:31 DEBUG }, 06:36:31 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:36:31 DEBUG "value": [ 06:36:31 DEBUG }, 06:36:31 DEBUG 06:36:31 DEBUG "value": [ 06:36:31 DEBUG 06:36:31 DEBUG 1712727391, 06:36:31 DEBUG 06:36:31 DEBUG "0" 06:36:31 DEBUG [...] 06:36:31 DEBUG ] 06:36:31 DEBUG 06:36:31 DEBUG } 06:36:31 DEBUG 06:36:31 DEBUG ] 06:36:31 DEBUG 06:36:31 DEBUG } 06:36:31 DEBUG "pod": "ds-cts-2" 06:36:31 DEBUG }, 06:36:31 DEBUG "value": [ 06:36:31 DEBUG 1712727391, 06:36:31 DEBUG "29.766666666666666" 06:36:31 DEBUG ] 06:36:31 DEBUG } 06:36:31 DEBUG ] 06:36:31 DEBUG } 06:36:31 DEBUG } 06:36:31 DEBUG } 06:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727391" 06:36:31 INFO [http_cmd]: http status code OK 06:36:31 DEBUG --- status code --- 06:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:31 DEBUG --- http response --- 06:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:36:31 DEBUG ----- output ----- 06:36:31 DEBUG { 06:36:31 DEBUG "status": "success", 06:36:31 DEBUG "data": { 06:36:31 DEBUG "resultType": "vector", 06:36:31 DEBUG "result": [ 06:36:31 DEBUG { 06:36:31 DEBUG "metric": { 06:36:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:36:31 DEBUG }, 06:36:31 DEBUG "value": [ 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG [...] 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:36:31 DEBUG }, 06:36:31 DEBUG "value": [ 06:36:31 DEBUG 1712727391, 06:36:31 DEBUG "0" 06:36:31 DEBUG ] 06:36:31 DEBUG } 06:36:31 DEBUG ] 06:36:31 DEBUG } 06:36:31 DEBUG } 06:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712727391" 06:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712727391" 06:36:31 INFO [http_cmd]: http status code OK 06:36:31 DEBUG --- status code --- 06:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:31 DEBUG --- http response --- 06:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:36:31 DEBUG ----- output ----- 06:36:31 DEBUG { 06:36:31 DEBUG "status": "success", 06:36:31 DEBUG "data": { 06:36:31 DEBUG "resultType": "vector", 06:36:31 DEBUG "result": [ 06:36:31 DEBUG { 06:36:31 DEBUG "metric": { 06:36:31 DEBUG "pod": "ds-idrepo-0" 06:36:31 DEBUG }, 06:36:31 DEBUG "value": [ 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG [...] 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG "pod": "ds-cts-2" 06:36:31 DEBUG }, 06:36:31 DEBUG "value": [ 06:36:31 DEBUG 1712727391, 06:36:31 DEBUG "0" 06:36:31 DEBUG ] 06:36:31 DEBUG } 06:36:31 DEBUG ] 06:36:31 DEBUG } 06:36:31 DEBUG } 06:36:31 INFO [http_cmd]: http status code OK 06:36:31 DEBUG --- status code --- 06:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:31 DEBUG --- http response --- 06:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:36:31 DEBUG ----- output ----- 06:36:31 DEBUG { 06:36:31 DEBUG "status": "success", 06:36:31 DEBUG "data": { 06:36:31 DEBUG "resultType": "vector", 06:36:31 DEBUG "result": [ 06:36:31 DEBUG { 06:36:31 DEBUG "metric": { 06:36:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:36:31 DEBUG }, 06:36:31 DEBUG "value": [ 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG [...] 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:36:31 DEBUG }, 06:36:31 DEBUG "value": [ 06:36:31 DEBUG 1712727391, 06:36:31 DEBUG "4.406450717196362" 06:36:31 DEBUG ] 06:36:31 DEBUG } 06:36:31 DEBUG ] 06:36:31 DEBUG } 06:36:31 DEBUG } 06:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727391" 06:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712727391" 06:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727391" 06:36:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727391" 06:36:31 INFO [http_cmd]: http status code OK 06:36:31 INFO [http_cmd]: http status code OK 06:36:31 DEBUG --- status code --- 06:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:31 DEBUG --- status code --- 06:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:31 DEBUG --- http response --- 06:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:36:31 DEBUG ----- output ----- 06:36:31 DEBUG { 06:36:31 DEBUG "status": "success", 06:36:31 DEBUG "data": { 06:36:31 DEBUG "resultType": "vector", 06:36:31 DEBUG "result": [ 06:36:31 DEBUG { 06:36:31 DEBUG "metric": { 06:36:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:36:31 DEBUG }, 06:36:31 DEBUG "value": [ 06:36:31 DEBUG 06:36:31 DEBUG --- http response --- 06:36:31 INFO [http_cmd]: http status code OK 06:36:31 DEBUG --- status code --- 06:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:31 DEBUG --- http response --- 06:36:31 DEBUG 06:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:36:31 DEBUG ----- output ----- 06:36:31 DEBUG { 06:36:31 DEBUG "status": "success", 06:36:31 DEBUG "data": { 06:36:31 DEBUG "resultType": "vector", 06:36:31 DEBUG "result": [ 06:36:31 DEBUG 06:36:31 DEBUG { 06:36:31 DEBUG "metric": { 06:36:31 DEBUG [...] 06:36:31 DEBUG "pod": "ds-idrepo-2" 06:36:31 DEBUG ----- output ----- 06:36:31 DEBUG { 06:36:31 DEBUG }, 06:36:31 DEBUG "value": [ 06:36:31 DEBUG "status": "success", 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG "data": { 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG "resultType": "vector", 06:36:31 DEBUG 06:36:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:36:31 DEBUG "result": [ 06:36:31 DEBUG 06:36:31 DEBUG [...] 06:36:31 DEBUG { 06:36:31 DEBUG "metric": { 06:36:31 DEBUG }, 06:36:31 DEBUG "value": [ 06:36:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG }, 06:36:31 DEBUG 06:36:31 DEBUG "pod": "ds-cts-2" 06:36:31 DEBUG "value": [ 06:36:31 DEBUG 1712727391, 06:36:31 DEBUG "0.053" 06:36:31 DEBUG 06:36:31 DEBUG ] 06:36:31 DEBUG }, 06:36:31 DEBUG 06:36:31 INFO [http_cmd]: http status code OK 06:36:31 DEBUG --- status code --- 06:36:31 DEBUG } 06:36:31 DEBUG 06:36:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:31 DEBUG --- http response --- 06:36:31 DEBUG ] 06:36:31 DEBUG [...] 06:36:31 DEBUG "value": [ 06:36:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:36:31 DEBUG } 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG } 06:36:31 DEBUG 1712727391, 06:36:31 DEBUG "0.00006666666666666667" 06:36:31 DEBUG ] 06:36:31 DEBUG ----- output ----- 06:36:31 DEBUG { 06:36:31 DEBUG "status": "success", 06:36:31 DEBUG "data": { 06:36:31 DEBUG "resultType": "vector", 06:36:31 DEBUG "result": [ 06:36:31 DEBUG { 06:36:31 DEBUG "metric": { 06:36:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:36:31 DEBUG }, 06:36:31 DEBUG "value": [ 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG [...] 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG 06:36:31 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:36:31 DEBUG }, 06:36:31 DEBUG "value": [ 06:36:31 DEBUG 1712727391, 06:36:31 DEBUG "0" 06:36:31 DEBUG ] 06:36:31 DEBUG } 06:36:31 DEBUG ] 06:36:31 DEBUG } 06:36:31 DEBUG } 06:36:31 DEBUG } 06:36:31 DEBUG ] 06:36:31 DEBUG } 06:36:31 DEBUG } 06:36:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:36:31 DEBUG }, 06:36:31 DEBUG "value": [ 06:36:31 DEBUG 1712727391, 06:36:31 DEBUG "1381.1333333333332" 06:36:31 DEBUG ] 06:36:31 DEBUG } 06:36:31 DEBUG ] 06:36:31 DEBUG } 06:36:31 DEBUG } 06:36:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727392" 06:36:32 INFO [http_cmd]: http status code OK 06:36:32 DEBUG --- status code --- 06:36:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:32 DEBUG --- http response --- 06:36:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:36:32 DEBUG ----- output ----- 06:36:32 DEBUG { 06:36:32 DEBUG "status": "success", 06:36:32 DEBUG "data": { 06:36:32 DEBUG "resultType": "vector", 06:36:32 DEBUG "result": [ 06:36:32 DEBUG { 06:36:32 DEBUG "metric": { 06:36:32 DEBUG "pod": "ds-idrepo-0" 06:36:32 DEBUG }, 06:36:32 DEBUG "value": [ 06:36:32 DEBUG 06:36:32 DEBUG 06:36:32 DEBUG 06:36:32 DEBUG [...] 06:36:32 DEBUG 06:36:32 DEBUG 06:36:32 DEBUG 06:36:32 DEBUG "pod": "ds-idrepo-1" 06:36:32 DEBUG }, 06:36:32 DEBUG "value": [ 06:36:32 DEBUG 1712727392, 06:36:32 DEBUG "0" 06:36:32 DEBUG ] 06:36:32 DEBUG } 06:36:32 DEBUG ] 06:36:32 DEBUG } 06:36:32 DEBUG } 06:36:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727392" 06:36:32 INFO [http_cmd]: http status code OK 06:36:32 DEBUG --- status code --- 06:36:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:32 DEBUG --- http response --- 06:36:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:36:32 DEBUG ----- output ----- 06:36:32 DEBUG { 06:36:32 DEBUG "status": "success", 06:36:32 DEBUG "data": { 06:36:32 DEBUG "resultType": "vector", 06:36:32 DEBUG "result": [ 06:36:32 DEBUG { 06:36:32 DEBUG "metric": { 06:36:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:36:32 DEBUG }, 06:36:32 DEBUG "value": [ 06:36:32 DEBUG 06:36:32 DEBUG 06:36:32 DEBUG 06:36:32 DEBUG [...] 06:36:32 DEBUG 06:36:32 DEBUG 06:36:32 DEBUG 06:36:32 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:36:32 DEBUG }, 06:36:32 DEBUG "value": [ 06:36:32 DEBUG 1712727392, 06:36:32 DEBUG "1382.1" 06:36:32 DEBUG ] 06:36:32 DEBUG } 06:36:32 DEBUG ] 06:36:32 DEBUG } 06:36:32 DEBUG } 06:36:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727392" 06:36:32 INFO [http_cmd]: http status code OK 06:36:32 DEBUG --- status code --- 06:36:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:32 DEBUG --- http response --- 06:36:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:36:32 DEBUG ----- output ----- 06:36:32 DEBUG { 06:36:32 DEBUG "status": "success", 06:36:32 DEBUG "data": { 06:36:32 DEBUG "resultType": "vector", 06:36:32 DEBUG "result": [ 06:36:32 DEBUG { 06:36:32 DEBUG "metric": { 06:36:32 DEBUG "pod": "ds-idrepo-0" 06:36:32 DEBUG }, 06:36:32 DEBUG "value": [ 06:36:32 DEBUG 06:36:32 DEBUG 06:36:32 DEBUG 06:36:32 DEBUG [...] 06:36:32 DEBUG 06:36:32 DEBUG 06:36:32 DEBUG 06:36:32 DEBUG "pod": "ds-idrepo-1" 06:36:32 DEBUG }, 06:36:32 DEBUG "value": [ 06:36:32 DEBUG 1712727392, 06:36:32 DEBUG "0" 06:36:32 DEBUG ] 06:36:32 DEBUG } 06:36:32 DEBUG ] 06:36:32 DEBUG } 06:36:32 DEBUG } 06:36:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727392" 06:36:32 INFO [http_cmd]: http status code OK 06:36:32 DEBUG --- status code --- 06:36:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:32 DEBUG --- http response --- 06:36:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:36:32 DEBUG ----- output ----- 06:36:32 DEBUG { 06:36:32 DEBUG "status": "success", 06:36:32 DEBUG "data": { 06:36:32 DEBUG "resultType": "vector", 06:36:32 DEBUG "result": [ 06:36:32 DEBUG { 06:36:32 DEBUG "metric": { 06:36:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:36:32 DEBUG }, 06:36:32 DEBUG "value": [ 06:36:32 DEBUG 06:36:32 DEBUG 06:36:32 DEBUG 06:36:32 DEBUG [...] 06:36:32 DEBUG 06:36:32 DEBUG 06:36:32 DEBUG 06:36:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:36:32 DEBUG }, 06:36:32 DEBUG "value": [ 06:36:32 DEBUG 1712727392, 06:36:32 DEBUG "0" 06:36:32 DEBUG ] 06:36:32 DEBUG } 06:36:32 DEBUG ] 06:36:32 DEBUG } 06:36:32 DEBUG } 06:36:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712727392" 06:36:32 INFO [http_cmd]: http status code OK 06:36:32 DEBUG --- status code --- 06:36:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:32 DEBUG --- http response --- 06:36:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:36:32 DEBUG ----- output ----- 06:36:32 DEBUG { 06:36:32 DEBUG "status": "success", 06:36:32 DEBUG "data": { 06:36:32 DEBUG "resultType": "vector", 06:36:32 DEBUG "result": [ 06:36:32 DEBUG { 06:36:32 DEBUG "metric": { 06:36:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:36:32 DEBUG }, 06:36:32 DEBUG "value": [ 06:36:32 DEBUG 06:36:32 DEBUG 06:36:32 DEBUG 06:36:32 DEBUG [...] 06:36:32 DEBUG 06:36:32 DEBUG 06:36:32 DEBUG 06:36:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:36:32 DEBUG }, 06:36:32 DEBUG "value": [ 06:36:32 DEBUG 1712727392, 06:36:32 DEBUG "2487545856" 06:36:32 DEBUG ] 06:36:32 DEBUG } 06:36:32 DEBUG ] 06:36:32 DEBUG } 06:36:32 DEBUG } 06:36:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727392" 06:36:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727392" 06:36:32 INFO [http_cmd]: http status code OK 06:36:32 DEBUG --- status code --- 06:36:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:32 DEBUG --- http response --- 06:36:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:36:32 DEBUG ----- output ----- 06:36:32 DEBUG { 06:36:32 DEBUG "status": "success", 06:36:32 DEBUG "data": { 06:36:32 DEBUG "resultType": "vector", 06:36:32 DEBUG "result": [ 06:36:32 DEBUG { 06:36:32 DEBUG "metric": { 06:36:32 DEBUG "instance": "10.142.0.25:9100" 06:36:32 DEBUG }, 06:36:32 DEBUG "value": [ 06:36:32 DEBUG 06:36:32 DEBUG 06:36:32 DEBUG 06:36:32 DEBUG [...] 06:36:32 DEBUG 06:36:32 DEBUG 06:36:32 DEBUG 06:36:32 DEBUG "instance": "10.142.0.77:9100" 06:36:32 DEBUG }, 06:36:32 DEBUG "value": [ 06:36:32 DEBUG 1712727392, 06:36:32 DEBUG "42486499.666666664" 06:36:32 DEBUG ] 06:36:32 DEBUG } 06:36:32 DEBUG ] 06:36:32 DEBUG } 06:36:32 DEBUG } 06:36:32 INFO [http_cmd]: http status code OK 06:36:32 DEBUG --- status code --- 06:36:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:32 DEBUG --- http response --- 06:36:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:36:32 DEBUG ----- output ----- 06:36:32 DEBUG { 06:36:32 DEBUG "status": "success", 06:36:32 DEBUG "data": { 06:36:32 DEBUG "resultType": "vector", 06:36:32 DEBUG "result": [ 06:36:32 DEBUG { 06:36:32 DEBUG "metric": { 06:36:32 DEBUG "instance": "10.142.0.25:9100" 06:36:32 DEBUG }, 06:36:32 DEBUG "value": [ 06:36:32 DEBUG 06:36:32 DEBUG 06:36:32 DEBUG 06:36:32 DEBUG [...] 06:36:32 DEBUG 06:36:32 DEBUG 06:36:32 DEBUG 06:36:32 DEBUG "instance": "10.142.0.77:9100" 06:36:32 DEBUG }, 06:36:32 DEBUG "value": [ 06:36:32 DEBUG 1712727392, 06:36:32 DEBUG "0.0449999999999998" 06:36:32 DEBUG ] 06:36:32 DEBUG } 06:36:32 DEBUG ] 06:36:32 DEBUG } 06:36:32 DEBUG } 06:36:52 INFO 06:36:52 INFO [loop_until]: kubectl --namespace=xlou top pods 06:36:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:36:52 INFO [loop_until]: OK (rc = 0) 06:36:52 DEBUG --- stdout --- 06:36:52 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10746m 9055Mi am-654c4fcbcf-9scrj 9890m 8457Mi am-654c4fcbcf-v5qb5 8272m 8491Mi ds-cts-0 7935m 13392Mi ds-cts-1 8527m 12540Mi ds-cts-2 7532m 12593Mi ds-idrepo-0 10655m 5634Mi ds-idrepo-1 357m 5672Mi ds-idrepo-2 420m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1191Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 11m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4379m 2627Mi 06:36:52 DEBUG --- stderr --- 06:36:52 DEBUG 06:37:00 INFO 06:37:00 INFO [loop_until]: kubectl --namespace=xlou top node 06:37:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:37:01 INFO [loop_until]: OK (rc = 0) 06:37:01 DEBUG --- stdout --- 06:37:01 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 96m 0% 2310Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9280m 58% 9362Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 83m 0% 3259Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 14179m 89% 6883Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10939m 68% 9339Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11977m 75% 9851Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 81m 0% 858Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10898m 68% 6319Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8467m 53% 13170Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 7699m 48% 13344Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 481m 3% 6394Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8379m 52% 13213Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 429m 2% 6361Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4517m 28% 6208Mi 10% 06:37:01 DEBUG --- stderr --- 06:37:01 DEBUG 06:37:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712727449" 06:37:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727449" 06:37:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727449" 06:37:29 INFO [http_cmd]: http status code OK 06:37:29 ERROR [http_cmd]: ERROR 06:37:29 DEBUG --- status code --- 06:37:29 INFO [http_cmd]: http status code OK 06:37:29 ERROR --- status code --- 06:37:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:29 DEBUG --- status code --- 06:37:29 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:37:29 DEBUG --- http response --- 06:37:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:29 ERROR --- http response --- 06:37:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:37:29 DEBUG --- http response --- 06:37:29 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:37:29 DEBUG ----- output ----- 06:37:29 DEBUG { 06:37:29 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:37:29 DEBUG "status": "success", 06:37:29 DEBUG "data": { 06:37:29 DEBUG "resultType": "vector", 06:37:29 DEBUG "result": [ 06:37:29 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:37:29 DEBUG { 06:37:29 DEBUG "metric": { 06:37:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:37:29 DEBUG }, 06:37:29 DEBUG "value": [ 06:37:29 DEBUG 06:37:29 DEBUG 06:37:29 DEBUG 06:37:29 DEBUG [...] 06:37:29 DEBUG 06:37:29 DEBUG 06:37:29 DEBUG 06:37:29 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:37:29 DEBUG }, 06:37:29 DEBUG "value": [ 06:37:29 DEBUG 1712727449, 06:37:29 DEBUG "0" 06:37:29 DEBUG ] 06:37:29 DEBUG } 06:37:29 DEBUG ] 06:37:29 DEBUG } 06:37:29 DEBUG } 06:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727451" 06:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727451" 06:37:31 INFO [http_cmd]: http status code OK 06:37:31 INFO [http_cmd]: http status code OK 06:37:31 DEBUG --- status code --- 06:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:31 DEBUG --- http response --- 06:37:31 DEBUG --- status code --- 06:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:31 DEBUG ----- output ----- 06:37:31 DEBUG --- http response --- 06:37:31 DEBUG { 06:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:37:31 DEBUG "status": "success", 06:37:31 DEBUG ----- output ----- 06:37:31 DEBUG "data": { 06:37:31 DEBUG { 06:37:31 DEBUG "resultType": "vector", 06:37:31 DEBUG "status": "success", 06:37:31 DEBUG "result": [ 06:37:31 DEBUG "data": { 06:37:31 DEBUG "resultType": "vector", 06:37:31 DEBUG { 06:37:31 DEBUG "result": [ 06:37:31 DEBUG "metric": { 06:37:31 DEBUG { 06:37:31 DEBUG "instance": "10.142.0.25:9100" 06:37:31 DEBUG "metric": { 06:37:31 DEBUG }, 06:37:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:37:31 DEBUG "value": [ 06:37:31 DEBUG }, 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG "value": [ 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG [...] 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG [...] 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG "instance": "10.142.0.77:9100" 06:37:31 DEBUG 06:37:31 DEBUG }, 06:37:31 DEBUG 06:37:31 DEBUG "value": [ 06:37:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:37:31 DEBUG 1712727451, 06:37:31 DEBUG }, 06:37:31 DEBUG "42784442.599999994" 06:37:31 DEBUG "value": [ 06:37:31 DEBUG ] 06:37:31 DEBUG 1712727451, 06:37:31 DEBUG } 06:37:31 DEBUG "1401.2333333333333" 06:37:31 DEBUG ] 06:37:31 DEBUG ] 06:37:31 DEBUG } 06:37:31 DEBUG } 06:37:31 DEBUG } 06:37:31 DEBUG ] 06:37:31 DEBUG } 06:37:31 DEBUG } 06:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727451" 06:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727451" 06:37:31 INFO [http_cmd]: http status code OK 06:37:31 DEBUG --- status code --- 06:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:31 DEBUG --- http response --- 06:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:37:31 DEBUG ----- output ----- 06:37:31 DEBUG { 06:37:31 DEBUG "status": "success", 06:37:31 DEBUG "data": { 06:37:31 DEBUG "resultType": "vector", 06:37:31 DEBUG "result": [ 06:37:31 DEBUG { 06:37:31 DEBUG "metric": { 06:37:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:37:31 DEBUG }, 06:37:31 DEBUG "value": [ 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG [...] 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:37:31 DEBUG }, 06:37:31 DEBUG "value": [ 06:37:31 DEBUG 1712727451, 06:37:31 DEBUG "1408.7" 06:37:31 DEBUG ] 06:37:31 DEBUG } 06:37:31 DEBUG ] 06:37:31 DEBUG } 06:37:31 DEBUG } 06:37:31 INFO [http_cmd]: http status code OK 06:37:31 DEBUG --- status code --- 06:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:31 DEBUG --- http response --- 06:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:37:31 DEBUG ----- output ----- 06:37:31 DEBUG { 06:37:31 DEBUG "status": "success", 06:37:31 DEBUG "data": { 06:37:31 DEBUG "resultType": "vector", 06:37:31 DEBUG "result": [ 06:37:31 DEBUG { 06:37:31 DEBUG "metric": { 06:37:31 DEBUG "pod": "ds-idrepo-0" 06:37:31 DEBUG }, 06:37:31 DEBUG "value": [ 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG [...] 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG "pod": "ds-idrepo-1" 06:37:31 DEBUG }, 06:37:31 DEBUG "value": [ 06:37:31 DEBUG 1712727451, 06:37:31 DEBUG "0" 06:37:31 DEBUG ] 06:37:31 DEBUG } 06:37:31 DEBUG ] 06:37:31 DEBUG } 06:37:31 DEBUG } 06:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727451" 06:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727451" 06:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727451" 06:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727451" 06:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727451" 06:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727451" 06:37:31 INFO [http_cmd]: http status code OK 06:37:31 DEBUG --- status code --- 06:37:31 INFO [http_cmd]: http status code OK 06:37:31 DEBUG --- status code --- 06:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:31 DEBUG --- http response --- 06:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:37:31 DEBUG ----- output ----- 06:37:31 DEBUG { 06:37:31 DEBUG "status": "success", 06:37:31 DEBUG "data": { 06:37:31 DEBUG "resultType": "vector", 06:37:31 DEBUG "result": [ 06:37:31 DEBUG { 06:37:31 DEBUG "metric": { 06:37:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:37:31 DEBUG }, 06:37:31 DEBUG "value": [ 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG [...] 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:37:31 DEBUG }, 06:37:31 DEBUG "value": [ 06:37:31 DEBUG 1712727451, 06:37:31 DEBUG "250807" 06:37:31 DEBUG ] 06:37:31 DEBUG } 06:37:31 DEBUG ] 06:37:31 DEBUG } 06:37:31 DEBUG } 06:37:31 INFO [http_cmd]: http status code OK 06:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:31 INFO [http_cmd]: http status code OK 06:37:31 DEBUG --- status code --- 06:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:31 DEBUG --- http response --- 06:37:31 DEBUG --- http response --- 06:37:31 INFO [http_cmd]: http status code OK 06:37:31 DEBUG --- status code --- 06:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:37:31 DEBUG ----- output ----- 06:37:31 DEBUG { 06:37:31 DEBUG ----- output ----- 06:37:31 DEBUG "status": "success", 06:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:31 DEBUG --- http response --- 06:37:31 DEBUG { 06:37:31 DEBUG "status": "success", 06:37:31 DEBUG --- status code --- 06:37:31 DEBUG "data": { 06:37:31 DEBUG "resultType": "vector", 06:37:31 DEBUG "result": [ 06:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:37:31 DEBUG { 06:37:31 DEBUG "metric": { 06:37:31 DEBUG ----- output ----- 06:37:31 DEBUG "instance": "10.142.0.25:9100" 06:37:31 DEBUG }, 06:37:31 DEBUG { 06:37:31 DEBUG "status": "success", 06:37:31 DEBUG "data": { 06:37:31 DEBUG "resultType": "vector", 06:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:31 DEBUG "result": [ 06:37:31 DEBUG { 06:37:31 DEBUG "data": { 06:37:31 DEBUG "resultType": "vector", 06:37:31 DEBUG "metric": { 06:37:31 DEBUG "result": [ 06:37:31 DEBUG { 06:37:31 DEBUG "instance": "10.142.0.25:9100" 06:37:31 DEBUG "metric": { 06:37:31 DEBUG "pod": "ds-idrepo-0" 06:37:31 DEBUG }, 06:37:31 DEBUG }, 06:37:31 DEBUG "value": [ 06:37:31 DEBUG "value": [ 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG [...] 06:37:31 DEBUG 06:37:31 DEBUG "value": [ 06:37:31 DEBUG --- http response --- 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 INFO [http_cmd]: http status code OK 06:37:31 DEBUG 06:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:37:31 DEBUG [...] 06:37:31 DEBUG 06:37:31 DEBUG --- status code --- 06:37:31 DEBUG 06:37:31 DEBUG ----- output ----- 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:31 DEBUG 06:37:31 DEBUG { 06:37:31 DEBUG 06:37:31 DEBUG "pod": "ds-cts-2" 06:37:31 DEBUG --- http response --- 06:37:31 DEBUG [...] 06:37:31 DEBUG "status": "success", 06:37:31 DEBUG 06:37:31 DEBUG }, 06:37:31 DEBUG "value": [ 06:37:31 DEBUG 1712727451, 06:37:31 DEBUG 06:37:31 DEBUG "instance": "10.142.0.77:9100" 06:37:31 DEBUG "29.366666666666667" 06:37:31 DEBUG ] 06:37:31 DEBUG } 06:37:31 DEBUG 06:37:31 DEBUG }, 06:37:31 DEBUG "value": [ 06:37:31 DEBUG 1712727451, 06:37:31 DEBUG "0.041333333333331626" 06:37:31 DEBUG 06:37:31 DEBUG ] 06:37:31 DEBUG } 06:37:31 DEBUG ] 06:37:31 DEBUG } 06:37:31 DEBUG "instance": "10.142.0.77:9100" 06:37:31 DEBUG } 06:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:37:31 DEBUG ----- output ----- 06:37:31 DEBUG { 06:37:31 DEBUG }, 06:37:31 DEBUG "status": "success", 06:37:31 DEBUG "data": { 06:37:31 DEBUG "resultType": "vector", 06:37:31 DEBUG "value": [ 06:37:31 DEBUG "result": [ 06:37:31 DEBUG { 06:37:31 DEBUG "metric": { 06:37:31 DEBUG 1712727451, 06:37:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:37:31 DEBUG }, 06:37:31 DEBUG "value": [ 06:37:31 DEBUG "417083581440" 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG ] 06:37:31 DEBUG [...] 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG } 06:37:31 DEBUG 06:37:31 DEBUG "pod": "ds-idrepo-0" 06:37:31 DEBUG }, 06:37:31 DEBUG ] 06:37:31 DEBUG } 06:37:31 DEBUG } 06:37:31 DEBUG "value": [ 06:37:31 DEBUG "data": { 06:37:31 DEBUG "resultType": "vector", 06:37:31 DEBUG "result": [ 06:37:31 DEBUG { 06:37:31 DEBUG "metric": { 06:37:31 DEBUG 1712727451, 06:37:31 DEBUG "0" 06:37:31 DEBUG "pod": "ds-idrepo-0" 06:37:31 DEBUG ] 06:37:31 DEBUG } 06:37:31 DEBUG }, 06:37:31 DEBUG "value": [ 06:37:31 DEBUG ] 06:37:31 DEBUG } 06:37:31 DEBUG 06:37:31 DEBUG } 06:37:31 DEBUG ] 06:37:31 DEBUG 06:37:31 DEBUG } 06:37:31 DEBUG 06:37:31 DEBUG } 06:37:31 DEBUG [...] 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG "pod": "ds-cts-2" 06:37:31 DEBUG }, 06:37:31 DEBUG "value": [ 06:37:31 DEBUG 1712727451, 06:37:31 DEBUG "0" 06:37:31 DEBUG ] 06:37:31 DEBUG } 06:37:31 DEBUG ] 06:37:31 DEBUG } 06:37:31 DEBUG } 06:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727451" 06:37:31 INFO [http_cmd]: http status code OK 06:37:31 DEBUG --- status code --- 06:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:31 DEBUG --- http response --- 06:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:37:31 DEBUG ----- output ----- 06:37:31 DEBUG { 06:37:31 DEBUG "status": "success", 06:37:31 DEBUG "data": { 06:37:31 DEBUG "resultType": "vector", 06:37:31 DEBUG "result": [ 06:37:31 DEBUG { 06:37:31 DEBUG "metric": { 06:37:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:37:31 DEBUG }, 06:37:31 DEBUG "value": [ 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG [...] 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG "pod": "ds-idrepo-0" 06:37:31 DEBUG }, 06:37:31 DEBUG "value": [ 06:37:31 DEBUG 1712727451, 06:37:31 DEBUG "167160548.7511668" 06:37:31 DEBUG ] 06:37:31 DEBUG } 06:37:31 DEBUG ] 06:37:31 DEBUG } 06:37:31 DEBUG } 06:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712727451" 06:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712727451" 06:37:31 INFO [http_cmd]: http status code OK 06:37:31 DEBUG --- status code --- 06:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:31 DEBUG --- http response --- 06:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:37:31 DEBUG ----- output ----- 06:37:31 DEBUG { 06:37:31 DEBUG "status": "success", 06:37:31 DEBUG "data": { 06:37:31 DEBUG "resultType": "vector", 06:37:31 DEBUG "result": [ 06:37:31 DEBUG { 06:37:31 DEBUG "metric": { 06:37:31 DEBUG "pod": "ds-idrepo-0" 06:37:31 DEBUG }, 06:37:31 DEBUG "value": [ 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG [...] 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG "pod": "ds-cts-2" 06:37:31 DEBUG }, 06:37:31 DEBUG "value": [ 06:37:31 DEBUG 1712727451, 06:37:31 DEBUG "0" 06:37:31 DEBUG ] 06:37:31 DEBUG } 06:37:31 DEBUG ] 06:37:31 DEBUG } 06:37:31 DEBUG } 06:37:31 INFO [http_cmd]: http status code OK 06:37:31 DEBUG --- status code --- 06:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:31 DEBUG --- http response --- 06:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:37:31 DEBUG ----- output ----- 06:37:31 DEBUG { 06:37:31 DEBUG "status": "success", 06:37:31 DEBUG "data": { 06:37:31 DEBUG "resultType": "vector", 06:37:31 DEBUG "result": [ 06:37:31 DEBUG { 06:37:31 DEBUG "metric": { 06:37:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:37:31 DEBUG }, 06:37:31 DEBUG "value": [ 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG [...] 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:37:31 DEBUG }, 06:37:31 DEBUG "value": [ 06:37:31 DEBUG 1712727451, 06:37:31 DEBUG "4.381005713872672" 06:37:31 DEBUG ] 06:37:31 DEBUG } 06:37:31 DEBUG ] 06:37:31 DEBUG } 06:37:31 DEBUG } 06:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712727451" 06:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727451" 06:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727451" 06:37:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727451" 06:37:31 INFO [http_cmd]: http status code OK 06:37:31 DEBUG --- status code --- 06:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:31 DEBUG --- http response --- 06:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:37:31 DEBUG ----- output ----- 06:37:31 DEBUG { 06:37:31 DEBUG "status": "success", 06:37:31 DEBUG "data": { 06:37:31 DEBUG "resultType": "vector", 06:37:31 DEBUG "result": [ 06:37:31 DEBUG { 06:37:31 DEBUG "metric": { 06:37:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:37:31 DEBUG }, 06:37:31 DEBUG "value": [ 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG [...] 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:37:31 DEBUG }, 06:37:31 DEBUG "value": [ 06:37:31 DEBUG 1712727451, 06:37:31 DEBUG "0.053" 06:37:31 DEBUG ] 06:37:31 DEBUG } 06:37:31 DEBUG ] 06:37:31 DEBUG } 06:37:31 DEBUG } 06:37:31 INFO [http_cmd]: http status code OK 06:37:31 INFO [http_cmd]: http status code OK 06:37:31 DEBUG --- status code --- 06:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:31 DEBUG --- http response --- 06:37:31 DEBUG --- status code --- 06:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:31 DEBUG --- http response --- 06:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:37:31 INFO [http_cmd]: http status code OK 06:37:31 DEBUG --- status code --- 06:37:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:31 DEBUG ----- output ----- 06:37:31 DEBUG { 06:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:37:31 DEBUG --- http response --- 06:37:31 DEBUG "status": "success", 06:37:31 DEBUG ----- output ----- 06:37:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:37:31 DEBUG "data": { 06:37:31 DEBUG { 06:37:31 DEBUG ----- output ----- 06:37:31 DEBUG "resultType": "vector", 06:37:31 DEBUG "status": "success", 06:37:31 DEBUG { 06:37:31 DEBUG "result": [ 06:37:31 DEBUG "data": { 06:37:31 DEBUG "status": "success", 06:37:31 DEBUG "data": { 06:37:31 DEBUG "resultType": "vector", 06:37:31 DEBUG "resultType": "vector", 06:37:31 DEBUG "result": [ 06:37:31 DEBUG "result": [ 06:37:31 DEBUG { 06:37:31 DEBUG "metric": { 06:37:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:37:31 DEBUG { 06:37:31 DEBUG }, 06:37:31 DEBUG "value": [ 06:37:31 DEBUG 06:37:31 DEBUG { 06:37:31 DEBUG "metric": { 06:37:31 DEBUG 06:37:31 DEBUG "metric": { 06:37:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:37:31 DEBUG }, 06:37:31 DEBUG "pod": "ds-idrepo-2" 06:37:31 DEBUG 06:37:31 DEBUG [...] 06:37:31 DEBUG }, 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG "value": [ 06:37:31 DEBUG 06:37:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:37:31 DEBUG 06:37:31 DEBUG }, 06:37:31 DEBUG "value": [ 06:37:31 DEBUG 06:37:31 DEBUG 1712727451, 06:37:31 DEBUG "1400.9666666666667" 06:37:31 DEBUG 06:37:31 DEBUG ] 06:37:31 DEBUG } 06:37:31 DEBUG [...] 06:37:31 DEBUG ] 06:37:31 DEBUG "value": [ 06:37:31 DEBUG 06:37:31 DEBUG } 06:37:31 DEBUG } 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG 06:37:31 DEBUG "pod": "ds-cts-2" 06:37:31 DEBUG 06:37:31 DEBUG }, 06:37:31 DEBUG [...] 06:37:31 DEBUG "value": [ 06:37:31 DEBUG 06:37:31 DEBUG 1712727451, 06:37:31 DEBUG 06:37:31 DEBUG "0.000023441111111111114" 06:37:31 DEBUG 06:37:31 DEBUG ] 06:37:31 DEBUG "pod": "ds-idrepo-0" 06:37:31 DEBUG } 06:37:31 DEBUG }, 06:37:31 DEBUG ] 06:37:31 DEBUG "value": [ 06:37:31 DEBUG } 06:37:31 DEBUG 1712727451, 06:37:31 DEBUG } 06:37:31 DEBUG "10.805357534183853" 06:37:31 DEBUG ] 06:37:31 DEBUG } 06:37:31 DEBUG ] 06:37:31 DEBUG } 06:37:31 DEBUG } 06:37:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727452" 06:37:32 INFO [http_cmd]: http status code OK 06:37:32 DEBUG --- status code --- 06:37:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:32 DEBUG --- http response --- 06:37:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:37:32 DEBUG ----- output ----- 06:37:32 DEBUG { 06:37:32 DEBUG "status": "success", 06:37:32 DEBUG "data": { 06:37:32 DEBUG "resultType": "vector", 06:37:32 DEBUG "result": [ 06:37:32 DEBUG { 06:37:32 DEBUG "metric": { 06:37:32 DEBUG "pod": "ds-idrepo-0" 06:37:32 DEBUG }, 06:37:32 DEBUG "value": [ 06:37:32 DEBUG 06:37:32 DEBUG 06:37:32 DEBUG 06:37:32 DEBUG [...] 06:37:32 DEBUG 06:37:32 DEBUG 06:37:32 DEBUG 06:37:32 DEBUG "pod": "ds-idrepo-1" 06:37:32 DEBUG }, 06:37:32 DEBUG "value": [ 06:37:32 DEBUG 1712727452, 06:37:32 DEBUG "0" 06:37:32 DEBUG ] 06:37:32 DEBUG } 06:37:32 DEBUG ] 06:37:32 DEBUG } 06:37:32 DEBUG } 06:37:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727452" 06:37:32 INFO [http_cmd]: http status code OK 06:37:32 DEBUG --- status code --- 06:37:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:32 DEBUG --- http response --- 06:37:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:37:32 DEBUG ----- output ----- 06:37:32 DEBUG { 06:37:32 DEBUG "status": "success", 06:37:32 DEBUG "data": { 06:37:32 DEBUG "resultType": "vector", 06:37:32 DEBUG "result": [ 06:37:32 DEBUG { 06:37:32 DEBUG "metric": { 06:37:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:37:32 DEBUG }, 06:37:32 DEBUG "value": [ 06:37:32 DEBUG 06:37:32 DEBUG 06:37:32 DEBUG 06:37:32 DEBUG [...] 06:37:32 DEBUG 06:37:32 DEBUG 06:37:32 DEBUG 06:37:32 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:37:32 DEBUG }, 06:37:32 DEBUG "value": [ 06:37:32 DEBUG 1712727452, 06:37:32 DEBUG "1399.5666666666666" 06:37:32 DEBUG ] 06:37:32 DEBUG } 06:37:32 DEBUG ] 06:37:32 DEBUG } 06:37:32 DEBUG } 06:37:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727452" 06:37:32 INFO [http_cmd]: http status code OK 06:37:32 DEBUG --- status code --- 06:37:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:32 DEBUG --- http response --- 06:37:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:37:32 DEBUG ----- output ----- 06:37:32 DEBUG { 06:37:32 DEBUG "status": "success", 06:37:32 DEBUG "data": { 06:37:32 DEBUG "resultType": "vector", 06:37:32 DEBUG "result": [ 06:37:32 DEBUG { 06:37:32 DEBUG "metric": { 06:37:32 DEBUG "pod": "ds-idrepo-0" 06:37:32 DEBUG }, 06:37:32 DEBUG "value": [ 06:37:32 DEBUG 06:37:32 DEBUG 06:37:32 DEBUG 06:37:32 DEBUG [...] 06:37:32 DEBUG 06:37:32 DEBUG 06:37:32 DEBUG 06:37:32 DEBUG "pod": "ds-idrepo-1" 06:37:32 DEBUG }, 06:37:32 DEBUG "value": [ 06:37:32 DEBUG 1712727452, 06:37:32 DEBUG "0" 06:37:32 DEBUG ] 06:37:32 DEBUG } 06:37:32 DEBUG ] 06:37:32 DEBUG } 06:37:32 DEBUG } 06:37:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727452" 06:37:32 INFO [http_cmd]: http status code OK 06:37:32 DEBUG --- status code --- 06:37:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:32 DEBUG --- http response --- 06:37:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:37:32 DEBUG ----- output ----- 06:37:32 DEBUG { 06:37:32 DEBUG "status": "success", 06:37:32 DEBUG "data": { 06:37:32 DEBUG "resultType": "vector", 06:37:32 DEBUG "result": [ 06:37:32 DEBUG { 06:37:32 DEBUG "metric": { 06:37:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:37:32 DEBUG }, 06:37:32 DEBUG "value": [ 06:37:32 DEBUG 06:37:32 DEBUG 06:37:32 DEBUG 06:37:32 DEBUG [...] 06:37:32 DEBUG 06:37:32 DEBUG 06:37:32 DEBUG 06:37:32 DEBUG "pod": "ds-idrepo-0" 06:37:32 DEBUG }, 06:37:32 DEBUG "value": [ 06:37:32 DEBUG 1712727452, 06:37:32 DEBUG "3591.661245155998" 06:37:32 DEBUG ] 06:37:32 DEBUG } 06:37:32 DEBUG ] 06:37:32 DEBUG } 06:37:32 DEBUG } 06:37:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712727452" 06:37:32 INFO [http_cmd]: http status code OK 06:37:32 DEBUG --- status code --- 06:37:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:32 DEBUG --- http response --- 06:37:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:37:32 DEBUG ----- output ----- 06:37:32 DEBUG { 06:37:32 DEBUG "status": "success", 06:37:32 DEBUG "data": { 06:37:32 DEBUG "resultType": "vector", 06:37:32 DEBUG "result": [ 06:37:32 DEBUG { 06:37:32 DEBUG "metric": { 06:37:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:37:32 DEBUG }, 06:37:32 DEBUG "value": [ 06:37:32 DEBUG 06:37:32 DEBUG 06:37:32 DEBUG 06:37:32 DEBUG [...] 06:37:32 DEBUG 06:37:32 DEBUG 06:37:32 DEBUG 06:37:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:37:32 DEBUG }, 06:37:32 DEBUG "value": [ 06:37:32 DEBUG 1712727452, 06:37:32 DEBUG "2489987072" 06:37:32 DEBUG ] 06:37:32 DEBUG } 06:37:32 DEBUG ] 06:37:32 DEBUG } 06:37:32 DEBUG } 06:37:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727452" 06:37:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727452" 06:37:32 INFO [http_cmd]: http status code OK 06:37:32 DEBUG --- status code --- 06:37:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:32 DEBUG --- http response --- 06:37:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:37:32 DEBUG ----- output ----- 06:37:32 DEBUG { 06:37:32 DEBUG "status": "success", 06:37:32 DEBUG "data": { 06:37:32 DEBUG "resultType": "vector", 06:37:32 DEBUG "result": [ 06:37:32 DEBUG { 06:37:32 DEBUG "metric": { 06:37:32 DEBUG "instance": "10.142.0.25:9100" 06:37:32 DEBUG }, 06:37:32 DEBUG "value": [ 06:37:32 DEBUG 06:37:32 DEBUG 06:37:32 DEBUG 06:37:32 DEBUG [...] 06:37:32 DEBUG 06:37:32 DEBUG 06:37:32 DEBUG 06:37:32 DEBUG "instance": "10.142.0.77:9100" 06:37:32 DEBUG }, 06:37:32 DEBUG "value": [ 06:37:32 DEBUG 1712727452, 06:37:32 DEBUG "42787862.33333333" 06:37:32 DEBUG ] 06:37:32 DEBUG } 06:37:32 DEBUG ] 06:37:32 DEBUG } 06:37:32 DEBUG } 06:37:32 INFO [http_cmd]: http status code OK 06:37:32 DEBUG --- status code --- 06:37:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:32 DEBUG --- http response --- 06:37:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:37:32 DEBUG ----- output ----- 06:37:32 DEBUG { 06:37:32 DEBUG "status": "success", 06:37:32 DEBUG "data": { 06:37:32 DEBUG "resultType": "vector", 06:37:32 DEBUG "result": [ 06:37:32 DEBUG { 06:37:32 DEBUG "metric": { 06:37:32 DEBUG "instance": "10.142.0.25:9100" 06:37:32 DEBUG }, 06:37:32 DEBUG "value": [ 06:37:32 DEBUG 06:37:32 DEBUG 06:37:32 DEBUG 06:37:32 DEBUG [...] 06:37:32 DEBUG 06:37:32 DEBUG 06:37:32 DEBUG 06:37:32 DEBUG "instance": "10.142.0.77:9100" 06:37:32 DEBUG }, 06:37:32 DEBUG "value": [ 06:37:32 DEBUG 1712727452, 06:37:32 DEBUG "0.0179999999999995" 06:37:32 DEBUG ] 06:37:32 DEBUG } 06:37:32 DEBUG ] 06:37:32 DEBUG } 06:37:32 DEBUG } 06:37:52 INFO 06:37:52 INFO [loop_until]: kubectl --namespace=xlou top pods 06:37:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:37:52 INFO [loop_until]: OK (rc = 0) 06:37:52 DEBUG --- stdout --- 06:37:52 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10606m 9068Mi am-654c4fcbcf-9scrj 10268m 8461Mi am-654c4fcbcf-v5qb5 8502m 8488Mi ds-cts-0 8074m 12961Mi ds-cts-1 8327m 13210Mi ds-cts-2 7778m 13394Mi ds-idrepo-0 10762m 5634Mi ds-idrepo-1 408m 5672Mi ds-idrepo-2 377m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1191Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 6m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4447m 2630Mi 06:37:52 DEBUG --- stderr --- 06:37:52 DEBUG 06:38:01 INFO 06:38:01 INFO [loop_until]: kubectl --namespace=xlou top node 06:38:01 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:38:01 INFO [loop_until]: OK (rc = 0) 06:38:01 DEBUG --- stdout --- 06:38:01 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 94m 0% 2310Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9391m 59% 9363Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 78m 0% 3260Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13860m 87% 6906Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 11001m 69% 9339Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11512m 72% 9857Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 78m 0% 855Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10958m 68% 6320Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8614m 54% 13783Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 7794m 49% 14084Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 426m 2% 6393Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8632m 54% 13946Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 470m 2% 6359Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4513m 28% 6213Mi 10% 06:38:01 DEBUG --- stderr --- 06:38:01 DEBUG 06:38:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727509" 06:38:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727509" 06:38:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712727509" 06:38:29 INFO [http_cmd]: http status code OK 06:38:29 INFO [http_cmd]: http status code OK 06:38:29 DEBUG --- status code --- 06:38:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:29 DEBUG --- http response --- 06:38:29 DEBUG --- status code --- 06:38:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:38:29 DEBUG ----- output ----- 06:38:29 DEBUG { 06:38:29 DEBUG "status": "success", 06:38:29 DEBUG "data": { 06:38:29 DEBUG "resultType": "vector", 06:38:29 DEBUG "result": [ 06:38:29 DEBUG { 06:38:29 DEBUG "metric": { 06:38:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:38:29 DEBUG }, 06:38:29 DEBUG "value": [ 06:38:29 DEBUG 06:38:29 DEBUG 06:38:29 DEBUG 06:38:29 DEBUG [...] 06:38:29 DEBUG 06:38:29 DEBUG 06:38:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:29 DEBUG 06:38:29 DEBUG --- http response --- 06:38:29 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:38:29 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:38:29 DEBUG }, 06:38:29 DEBUG "value": [ 06:38:29 DEBUG 1712727509, 06:38:29 DEBUG "0" 06:38:29 DEBUG ] 06:38:29 DEBUG } 06:38:29 DEBUG ] 06:38:29 DEBUG } 06:38:29 DEBUG } 06:38:29 ERROR [http_cmd]: ERROR 06:38:29 ERROR --- status code --- 06:38:29 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:38:29 ERROR --- http response --- 06:38:29 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:38:29 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727511" 06:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727511" 06:38:31 INFO [http_cmd]: http status code OK 06:38:31 DEBUG --- status code --- 06:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:31 DEBUG --- http response --- 06:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:38:31 DEBUG ----- output ----- 06:38:31 DEBUG { 06:38:31 DEBUG "status": "success", 06:38:31 DEBUG "data": { 06:38:31 DEBUG "resultType": "vector", 06:38:31 DEBUG "result": [ 06:38:31 INFO [http_cmd]: http status code OK 06:38:31 DEBUG { 06:38:31 DEBUG --- status code --- 06:38:31 DEBUG "metric": { 06:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:38:31 DEBUG }, 06:38:31 DEBUG "value": [ 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG [...] 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:38:31 DEBUG }, 06:38:31 DEBUG "value": [ 06:38:31 DEBUG 1712727511, 06:38:31 DEBUG "1411.3666666666666" 06:38:31 DEBUG ] 06:38:31 DEBUG } 06:38:31 DEBUG ] 06:38:31 DEBUG } 06:38:31 DEBUG } 06:38:31 DEBUG --- http response --- 06:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:38:31 DEBUG ----- output ----- 06:38:31 DEBUG { 06:38:31 DEBUG "status": "success", 06:38:31 DEBUG "data": { 06:38:31 DEBUG "resultType": "vector", 06:38:31 DEBUG "result": [ 06:38:31 DEBUG { 06:38:31 DEBUG "metric": { 06:38:31 DEBUG "instance": "10.142.0.25:9100" 06:38:31 DEBUG }, 06:38:31 DEBUG "value": [ 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG [...] 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG "instance": "10.142.0.77:9100" 06:38:31 DEBUG }, 06:38:31 DEBUG "value": [ 06:38:31 DEBUG 1712727511, 06:38:31 DEBUG "42755348.266666666" 06:38:31 DEBUG ] 06:38:31 DEBUG } 06:38:31 DEBUG ] 06:38:31 DEBUG } 06:38:31 DEBUG } 06:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727511" 06:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727511" 06:38:31 INFO [http_cmd]: http status code OK 06:38:31 DEBUG --- status code --- 06:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:31 DEBUG --- http response --- 06:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:38:31 DEBUG ----- output ----- 06:38:31 DEBUG { 06:38:31 DEBUG "status": "success", 06:38:31 DEBUG "data": { 06:38:31 DEBUG "resultType": "vector", 06:38:31 DEBUG "result": [ 06:38:31 DEBUG { 06:38:31 DEBUG "metric": { 06:38:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:38:31 DEBUG }, 06:38:31 DEBUG "value": [ 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG [...] 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:38:31 INFO [http_cmd]: http status code OK 06:38:31 DEBUG }, 06:38:31 DEBUG --- status code --- 06:38:31 DEBUG "value": [ 06:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:31 DEBUG 1712727511, 06:38:31 DEBUG --- http response --- 06:38:31 DEBUG "1407.5666666666666" 06:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:38:31 DEBUG ] 06:38:31 DEBUG ----- output ----- 06:38:31 DEBUG { 06:38:31 DEBUG } 06:38:31 DEBUG ] 06:38:31 DEBUG "status": "success", 06:38:31 DEBUG } 06:38:31 DEBUG } 06:38:31 DEBUG "data": { 06:38:31 DEBUG "resultType": "vector", 06:38:31 DEBUG "result": [ 06:38:31 DEBUG { 06:38:31 DEBUG "metric": { 06:38:31 DEBUG "pod": "ds-idrepo-0" 06:38:31 DEBUG }, 06:38:31 DEBUG "value": [ 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG [...] 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG "pod": "ds-idrepo-1" 06:38:31 DEBUG }, 06:38:31 DEBUG "value": [ 06:38:31 DEBUG 1712727511, 06:38:31 DEBUG "0" 06:38:31 DEBUG ] 06:38:31 DEBUG } 06:38:31 DEBUG ] 06:38:31 DEBUG } 06:38:31 DEBUG } 06:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727511" 06:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727511" 06:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727511" 06:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727511" 06:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727511" 06:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727511" 06:38:31 INFO [http_cmd]: http status code OK 06:38:31 DEBUG --- status code --- 06:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:31 DEBUG --- http response --- 06:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:38:31 DEBUG ----- output ----- 06:38:31 INFO [http_cmd]: http status code OK 06:38:31 INFO [http_cmd]: http status code OK 06:38:31 DEBUG --- status code --- 06:38:31 DEBUG --- status code --- 06:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:31 DEBUG { 06:38:31 DEBUG "status": "success", 06:38:31 DEBUG --- http response --- 06:38:31 DEBUG "data": { 06:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:31 DEBUG --- http response --- 06:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:38:31 DEBUG ----- output ----- 06:38:31 DEBUG "resultType": "vector", 06:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:38:31 INFO [http_cmd]: http status code OK 06:38:31 DEBUG { 06:38:31 DEBUG "result": [ 06:38:31 DEBUG ----- output ----- 06:38:31 DEBUG { 06:38:31 DEBUG --- status code --- 06:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:31 DEBUG --- http response --- 06:38:31 DEBUG "status": "success", 06:38:31 DEBUG "data": { 06:38:31 DEBUG "status": "success", 06:38:31 DEBUG { 06:38:31 DEBUG "resultType": "vector", 06:38:31 DEBUG "result": [ 06:38:31 DEBUG { 06:38:31 INFO [http_cmd]: http status code OK 06:38:31 DEBUG "metric": { 06:38:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:38:31 DEBUG }, 06:38:31 DEBUG "metric": { 06:38:31 DEBUG "instance": "10.142.0.25:9100" 06:38:31 DEBUG }, 06:38:31 DEBUG "value": [ 06:38:31 DEBUG 06:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:38:31 DEBUG ----- output ----- 06:38:31 DEBUG "value": [ 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 INFO [http_cmd]: http status code OK 06:38:31 DEBUG "data": { 06:38:31 DEBUG "resultType": "vector", 06:38:31 DEBUG 06:38:31 DEBUG --- status code --- 06:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:31 DEBUG --- status code --- 06:38:31 DEBUG { 06:38:31 DEBUG "status": "success", 06:38:31 DEBUG 06:38:31 DEBUG --- http response --- 06:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:31 DEBUG --- http response --- 06:38:31 DEBUG "result": [ 06:38:31 DEBUG 06:38:31 DEBUG ----- output ----- 06:38:31 DEBUG { 06:38:31 DEBUG "data": { 06:38:31 DEBUG "resultType": "vector", 06:38:31 DEBUG { 06:38:31 DEBUG [...] 06:38:31 DEBUG 06:38:31 DEBUG "status": "success", 06:38:31 DEBUG "result": [ 06:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:38:31 DEBUG ----- output ----- 06:38:31 DEBUG { 06:38:31 DEBUG 06:38:31 DEBUG "data": { 06:38:31 DEBUG { 06:38:31 DEBUG "status": "success", 06:38:31 DEBUG "data": { 06:38:31 DEBUG "resultType": "vector", 06:38:31 DEBUG 06:38:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:38:31 DEBUG }, 06:38:31 DEBUG "value": [ 06:38:31 DEBUG "metric": { 06:38:31 DEBUG "instance": "10.142.0.25:9100" 06:38:31 DEBUG }, 06:38:31 DEBUG "value": [ 06:38:31 DEBUG 1712727511, 06:38:31 DEBUG "metric": { 06:38:31 DEBUG "resultType": "vector", 06:38:31 DEBUG "result": [ 06:38:31 DEBUG { 06:38:31 DEBUG "metric": { 06:38:31 DEBUG "251511.5" 06:38:31 DEBUG "pod": "ds-idrepo-0" 06:38:31 DEBUG "pod": "ds-idrepo-0" 06:38:31 DEBUG "result": [ 06:38:31 DEBUG { 06:38:31 DEBUG "metric": { 06:38:31 DEBUG ] 06:38:31 DEBUG }, 06:38:31 DEBUG }, 06:38:31 DEBUG "value": [ 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG } 06:38:31 DEBUG "value": [ 06:38:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:38:31 DEBUG }, 06:38:31 DEBUG "value": [ 06:38:31 DEBUG 06:38:31 DEBUG ] 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG [...] 06:38:31 DEBUG 06:38:31 DEBUG } 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG [...] 06:38:31 DEBUG } 06:38:31 DEBUG [...] 06:38:31 DEBUG [...] 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG "pod": "ds-cts-2" 06:38:31 DEBUG }, 06:38:31 DEBUG "value": [ 06:38:31 DEBUG 1712727511, 06:38:31 DEBUG "29.797020297970203" 06:38:31 DEBUG ] 06:38:31 DEBUG } 06:38:31 DEBUG ] 06:38:31 DEBUG } 06:38:31 DEBUG } 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:38:31 DEBUG }, 06:38:31 DEBUG "value": [ 06:38:31 DEBUG 1712727511, 06:38:31 DEBUG "0" 06:38:31 DEBUG ] 06:38:31 DEBUG } 06:38:31 DEBUG ] 06:38:31 DEBUG } 06:38:31 DEBUG } 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG "pod": "ds-cts-2" 06:38:31 DEBUG }, 06:38:31 DEBUG "value": [ 06:38:31 DEBUG 1712727511, 06:38:31 DEBUG "0" 06:38:31 DEBUG ] 06:38:31 DEBUG } 06:38:31 DEBUG ] 06:38:31 DEBUG } 06:38:31 DEBUG } 06:38:31 DEBUG [...] 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG "instance": "10.142.0.77:9100" 06:38:31 DEBUG }, 06:38:31 DEBUG "value": [ 06:38:31 DEBUG 1712727511, 06:38:31 DEBUG "419258061824" 06:38:31 DEBUG ] 06:38:31 DEBUG } 06:38:31 DEBUG ] 06:38:31 DEBUG } 06:38:31 DEBUG } 06:38:31 DEBUG 06:38:31 DEBUG "instance": "10.142.0.77:9100" 06:38:31 DEBUG }, 06:38:31 DEBUG "value": [ 06:38:31 DEBUG 1712727511, 06:38:31 DEBUG "0.15793333333332987" 06:38:31 DEBUG ] 06:38:31 DEBUG } 06:38:31 DEBUG ] 06:38:31 DEBUG } 06:38:31 DEBUG } 06:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727511" 06:38:31 INFO [http_cmd]: http status code OK 06:38:31 DEBUG --- status code --- 06:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:31 DEBUG --- http response --- 06:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:38:31 DEBUG ----- output ----- 06:38:31 DEBUG { 06:38:31 DEBUG "status": "success", 06:38:31 DEBUG "data": { 06:38:31 DEBUG "resultType": "vector", 06:38:31 DEBUG "result": [ 06:38:31 DEBUG { 06:38:31 DEBUG "metric": { 06:38:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:38:31 DEBUG }, 06:38:31 DEBUG "value": [ 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG [...] 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:38:31 DEBUG }, 06:38:31 DEBUG "value": [ 06:38:31 DEBUG 1712727511, 06:38:31 DEBUG "74817.47923781244" 06:38:31 DEBUG ] 06:38:31 DEBUG } 06:38:31 DEBUG ] 06:38:31 DEBUG } 06:38:31 DEBUG } 06:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712727511" 06:38:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712727511" 06:38:31 INFO [http_cmd]: http status code OK 06:38:31 DEBUG --- status code --- 06:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:31 DEBUG --- http response --- 06:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:38:31 DEBUG ----- output ----- 06:38:31 DEBUG { 06:38:31 DEBUG "status": "success", 06:38:31 DEBUG "data": { 06:38:31 DEBUG "resultType": "vector", 06:38:31 DEBUG "result": [ 06:38:31 DEBUG { 06:38:31 DEBUG "metric": { 06:38:31 DEBUG "pod": "ds-idrepo-0" 06:38:31 DEBUG }, 06:38:31 DEBUG "value": [ 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG [...] 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG "pod": "ds-cts-2" 06:38:31 DEBUG }, 06:38:31 DEBUG "value": [ 06:38:31 DEBUG 1712727511, 06:38:31 DEBUG "0" 06:38:31 DEBUG ] 06:38:31 DEBUG } 06:38:31 DEBUG ] 06:38:31 DEBUG } 06:38:31 DEBUG } 06:38:31 INFO [http_cmd]: http status code OK 06:38:31 DEBUG --- status code --- 06:38:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:31 DEBUG --- http response --- 06:38:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:38:31 DEBUG ----- output ----- 06:38:31 DEBUG { 06:38:31 DEBUG "status": "success", 06:38:31 DEBUG "data": { 06:38:31 DEBUG "resultType": "vector", 06:38:31 DEBUG "result": [ 06:38:31 DEBUG { 06:38:31 DEBUG "metric": { 06:38:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:38:31 DEBUG }, 06:38:31 DEBUG "value": [ 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG [...] 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG 06:38:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:38:31 DEBUG }, 06:38:31 DEBUG "value": [ 06:38:31 DEBUG 1712727511, 06:38:31 DEBUG "4.453111904360826" 06:38:31 DEBUG ] 06:38:31 DEBUG } 06:38:31 DEBUG ] 06:38:31 DEBUG } 06:38:31 DEBUG } 06:38:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712727512" 06:38:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727512" 06:38:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727512" 06:38:32 INFO [http_cmd]: http status code OK 06:38:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727512" 06:38:32 DEBUG --- status code --- 06:38:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:32 DEBUG --- http response --- 06:38:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:38:32 DEBUG ----- output ----- 06:38:32 DEBUG { 06:38:32 DEBUG "status": "success", 06:38:32 DEBUG "data": { 06:38:32 DEBUG "resultType": "vector", 06:38:32 DEBUG "result": [ 06:38:32 DEBUG { 06:38:32 DEBUG "metric": { 06:38:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:38:32 DEBUG }, 06:38:32 DEBUG "value": [ 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG [...] 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:38:32 DEBUG }, 06:38:32 DEBUG "value": [ 06:38:32 DEBUG 1712727512, 06:38:32 DEBUG "0.053" 06:38:32 DEBUG ] 06:38:32 DEBUG } 06:38:32 DEBUG ] 06:38:32 DEBUG } 06:38:32 DEBUG } 06:38:32 INFO [http_cmd]: http status code OK 06:38:32 DEBUG --- status code --- 06:38:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:32 DEBUG --- http response --- 06:38:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:38:32 DEBUG ----- output ----- 06:38:32 DEBUG { 06:38:32 DEBUG "status": "success", 06:38:32 DEBUG "data": { 06:38:32 DEBUG "resultType": "vector", 06:38:32 DEBUG "result": [ 06:38:32 DEBUG { 06:38:32 DEBUG "metric": { 06:38:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:38:32 DEBUG }, 06:38:32 DEBUG "value": [ 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG [...] 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:38:32 DEBUG }, 06:38:32 DEBUG "value": [ 06:38:32 DEBUG 1712727512, 06:38:32 DEBUG "1411.1666666666667" 06:38:32 DEBUG ] 06:38:32 DEBUG } 06:38:32 DEBUG ] 06:38:32 DEBUG } 06:38:32 DEBUG } 06:38:32 INFO [http_cmd]: http status code OK 06:38:32 DEBUG --- status code --- 06:38:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:32 DEBUG --- http response --- 06:38:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:38:32 DEBUG ----- output ----- 06:38:32 DEBUG { 06:38:32 DEBUG "status": "success", 06:38:32 DEBUG "data": { 06:38:32 DEBUG "resultType": "vector", 06:38:32 DEBUG "result": [ 06:38:32 DEBUG { 06:38:32 INFO [http_cmd]: http status code OK 06:38:32 DEBUG "metric": { 06:38:32 DEBUG "pod": "ds-idrepo-2" 06:38:32 DEBUG --- status code --- 06:38:32 DEBUG }, 06:38:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:32 DEBUG "value": [ 06:38:32 DEBUG --- http response --- 06:38:32 DEBUG 06:38:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:38:32 DEBUG 06:38:32 DEBUG ----- output ----- 06:38:32 DEBUG 06:38:32 DEBUG { 06:38:32 DEBUG [...] 06:38:32 DEBUG "status": "success", 06:38:32 DEBUG "data": { 06:38:32 DEBUG 06:38:32 DEBUG "resultType": "vector", 06:38:32 DEBUG 06:38:32 DEBUG "result": [ 06:38:32 DEBUG 06:38:32 DEBUG { 06:38:32 DEBUG "pod": "ds-cts-2" 06:38:32 DEBUG "metric": { 06:38:32 DEBUG }, 06:38:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:38:32 DEBUG "value": [ 06:38:32 DEBUG }, 06:38:32 DEBUG 1712727512, 06:38:32 DEBUG "value": [ 06:38:32 DEBUG "0.00023994267239942676" 06:38:32 DEBUG 06:38:32 DEBUG ] 06:38:32 DEBUG 06:38:32 DEBUG } 06:38:32 DEBUG 06:38:32 DEBUG ] 06:38:32 DEBUG [...] 06:38:32 DEBUG } 06:38:32 DEBUG 06:38:32 DEBUG } 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:38:32 DEBUG }, 06:38:32 DEBUG "value": [ 06:38:32 DEBUG 1712727512, 06:38:32 DEBUG "4.1384374682925635" 06:38:32 DEBUG ] 06:38:32 DEBUG } 06:38:32 DEBUG ] 06:38:32 DEBUG } 06:38:32 DEBUG } 06:38:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727512" 06:38:32 INFO [http_cmd]: http status code OK 06:38:32 DEBUG --- status code --- 06:38:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:32 DEBUG --- http response --- 06:38:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:38:32 DEBUG ----- output ----- 06:38:32 DEBUG { 06:38:32 DEBUG "status": "success", 06:38:32 DEBUG "data": { 06:38:32 DEBUG "resultType": "vector", 06:38:32 DEBUG "result": [ 06:38:32 DEBUG { 06:38:32 DEBUG "metric": { 06:38:32 DEBUG "pod": "ds-idrepo-0" 06:38:32 DEBUG }, 06:38:32 DEBUG "value": [ 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG [...] 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG "pod": "ds-idrepo-1" 06:38:32 DEBUG }, 06:38:32 DEBUG "value": [ 06:38:32 DEBUG 1712727512, 06:38:32 DEBUG "0" 06:38:32 DEBUG ] 06:38:32 DEBUG } 06:38:32 DEBUG ] 06:38:32 DEBUG } 06:38:32 DEBUG } 06:38:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727512" 06:38:32 INFO [http_cmd]: http status code OK 06:38:32 DEBUG --- status code --- 06:38:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:32 DEBUG --- http response --- 06:38:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:38:32 DEBUG ----- output ----- 06:38:32 DEBUG { 06:38:32 DEBUG "status": "success", 06:38:32 DEBUG "data": { 06:38:32 DEBUG "resultType": "vector", 06:38:32 DEBUG "result": [ 06:38:32 DEBUG { 06:38:32 DEBUG "metric": { 06:38:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:38:32 DEBUG }, 06:38:32 DEBUG "value": [ 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG [...] 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:38:32 DEBUG }, 06:38:32 DEBUG "value": [ 06:38:32 DEBUG 1712727512, 06:38:32 DEBUG "1405.0333333333333" 06:38:32 DEBUG ] 06:38:32 DEBUG } 06:38:32 DEBUG ] 06:38:32 DEBUG } 06:38:32 DEBUG } 06:38:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727512" 06:38:32 INFO [http_cmd]: http status code OK 06:38:32 DEBUG --- status code --- 06:38:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:32 DEBUG --- http response --- 06:38:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:38:32 DEBUG ----- output ----- 06:38:32 DEBUG { 06:38:32 DEBUG "status": "success", 06:38:32 DEBUG "data": { 06:38:32 DEBUG "resultType": "vector", 06:38:32 DEBUG "result": [ 06:38:32 DEBUG { 06:38:32 DEBUG "metric": { 06:38:32 DEBUG "pod": "ds-idrepo-0" 06:38:32 DEBUG }, 06:38:32 DEBUG "value": [ 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG [...] 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG "pod": "ds-idrepo-1" 06:38:32 DEBUG }, 06:38:32 DEBUG "value": [ 06:38:32 DEBUG 1712727512, 06:38:32 DEBUG "0" 06:38:32 DEBUG ] 06:38:32 DEBUG } 06:38:32 DEBUG ] 06:38:32 DEBUG } 06:38:32 DEBUG } 06:38:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727512" 06:38:32 INFO [http_cmd]: http status code OK 06:38:32 DEBUG --- status code --- 06:38:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:32 DEBUG --- http response --- 06:38:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:38:32 DEBUG ----- output ----- 06:38:32 DEBUG { 06:38:32 DEBUG "status": "success", 06:38:32 DEBUG "data": { 06:38:32 DEBUG "resultType": "vector", 06:38:32 DEBUG "result": [ 06:38:32 DEBUG { 06:38:32 DEBUG "metric": { 06:38:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:38:32 DEBUG }, 06:38:32 DEBUG "value": [ 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG [...] 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:38:32 DEBUG }, 06:38:32 DEBUG "value": [ 06:38:32 DEBUG 1712727512, 06:38:32 DEBUG "0" 06:38:32 DEBUG ] 06:38:32 DEBUG } 06:38:32 DEBUG ] 06:38:32 DEBUG } 06:38:32 DEBUG } 06:38:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712727512" 06:38:32 INFO [http_cmd]: http status code OK 06:38:32 DEBUG --- status code --- 06:38:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:32 DEBUG --- http response --- 06:38:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:38:32 DEBUG ----- output ----- 06:38:32 DEBUG { 06:38:32 DEBUG "status": "success", 06:38:32 DEBUG "data": { 06:38:32 DEBUG "resultType": "vector", 06:38:32 DEBUG "result": [ 06:38:32 DEBUG { 06:38:32 DEBUG "metric": { 06:38:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:38:32 DEBUG }, 06:38:32 DEBUG "value": [ 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG [...] 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:38:32 DEBUG }, 06:38:32 DEBUG "value": [ 06:38:32 DEBUG 1712727512, 06:38:32 DEBUG "2487037952" 06:38:32 DEBUG ] 06:38:32 DEBUG } 06:38:32 DEBUG ] 06:38:32 DEBUG } 06:38:32 DEBUG } 06:38:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727512" 06:38:32 INFO [http_cmd]: http status code OK 06:38:32 DEBUG --- status code --- 06:38:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:32 DEBUG --- http response --- 06:38:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:38:32 DEBUG ----- output ----- 06:38:32 DEBUG { 06:38:32 DEBUG "status": "success", 06:38:32 DEBUG "data": { 06:38:32 DEBUG "resultType": "vector", 06:38:32 DEBUG "result": [ 06:38:32 DEBUG { 06:38:32 DEBUG "metric": { 06:38:32 DEBUG "instance": "10.142.0.25:9100" 06:38:32 DEBUG }, 06:38:32 DEBUG "value": [ 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG [...] 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG "instance": "10.142.0.77:9100" 06:38:32 DEBUG }, 06:38:32 DEBUG "value": [ 06:38:32 DEBUG 1712727512, 06:38:32 DEBUG "42756328.166666664" 06:38:32 DEBUG ] 06:38:32 DEBUG } 06:38:32 DEBUG ] 06:38:32 DEBUG } 06:38:32 DEBUG } 06:38:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727512" 06:38:32 INFO [http_cmd]: http status code OK 06:38:32 DEBUG --- status code --- 06:38:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:32 DEBUG --- http response --- 06:38:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:38:32 DEBUG ----- output ----- 06:38:32 DEBUG { 06:38:32 DEBUG "status": "success", 06:38:32 DEBUG "data": { 06:38:32 DEBUG "resultType": "vector", 06:38:32 DEBUG "result": [ 06:38:32 DEBUG { 06:38:32 DEBUG "metric": { 06:38:32 DEBUG "instance": "10.142.0.25:9100" 06:38:32 DEBUG }, 06:38:32 DEBUG "value": [ 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG [...] 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG 06:38:32 DEBUG "instance": "10.142.0.77:9100" 06:38:32 DEBUG }, 06:38:32 DEBUG "value": [ 06:38:32 DEBUG 1712727512, 06:38:32 DEBUG "0.05399999999999979" 06:38:32 DEBUG ] 06:38:32 DEBUG } 06:38:32 DEBUG ] 06:38:32 DEBUG } 06:38:32 DEBUG } 06:38:52 INFO 06:38:52 INFO [loop_until]: kubectl --namespace=xlou top pods 06:38:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:38:52 INFO [loop_until]: OK (rc = 0) 06:38:52 DEBUG --- stdout --- 06:38:52 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10992m 9023Mi am-654c4fcbcf-9scrj 9874m 8461Mi am-654c4fcbcf-v5qb5 8791m 8489Mi ds-cts-0 8364m 13394Mi ds-cts-1 8640m 12688Mi ds-cts-2 7071m 12732Mi ds-idrepo-0 11013m 5634Mi ds-idrepo-1 373m 5672Mi ds-idrepo-2 378m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1191Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 5m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4389m 2631Mi 06:38:52 DEBUG --- stderr --- 06:38:52 DEBUG 06:39:01 INFO 06:39:01 INFO [loop_until]: kubectl --namespace=xlou top node 06:39:01 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:39:01 INFO [loop_until]: OK (rc = 0) 06:39:01 DEBUG --- stdout --- 06:39:01 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 91m 0% 2310Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9442m 59% 9362Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 78m 0% 3259Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13875m 87% 6915Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 11035m 69% 9338Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11998m 75% 9847Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 78m 0% 859Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10973m 69% 6321Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8454m 53% 14084Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 7597m 47% 13534Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 433m 2% 6395Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8690m 54% 13391Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 436m 2% 6360Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4392m 27% 6211Mi 10% 06:39:01 DEBUG --- stderr --- 06:39:01 DEBUG 06:39:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727569" 06:39:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727569" 06:39:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712727569" 06:39:29 INFO [http_cmd]: http status code OK 06:39:29 DEBUG --- status code --- 06:39:29 ERROR [http_cmd]: ERROR 06:39:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:29 ERROR --- status code --- 06:39:29 DEBUG --- http response --- 06:39:29 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:39:29 ERROR --- http response --- 06:39:29 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:39:29 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:39:29 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:39:29 INFO [http_cmd]: http status code OK 06:39:29 DEBUG --- status code --- 06:39:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:29 DEBUG --- http response --- 06:39:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:39:29 DEBUG ----- output ----- 06:39:29 DEBUG { 06:39:29 DEBUG "status": "success", 06:39:29 DEBUG "data": { 06:39:29 DEBUG "resultType": "vector", 06:39:29 DEBUG "result": [ 06:39:29 DEBUG { 06:39:29 DEBUG "metric": { 06:39:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:39:29 DEBUG }, 06:39:29 DEBUG "value": [ 06:39:29 DEBUG 06:39:29 DEBUG 06:39:29 DEBUG 06:39:29 DEBUG [...] 06:39:29 DEBUG 06:39:29 DEBUG 06:39:29 DEBUG 06:39:29 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:39:29 DEBUG }, 06:39:29 DEBUG "value": [ 06:39:29 DEBUG 1712727569, 06:39:29 DEBUG "0" 06:39:29 DEBUG ] 06:39:29 DEBUG } 06:39:29 DEBUG ] 06:39:29 DEBUG } 06:39:29 DEBUG } 06:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727571" 06:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727571" 06:39:31 INFO [http_cmd]: http status code OK 06:39:31 DEBUG --- status code --- 06:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:31 DEBUG --- http response --- 06:39:31 INFO [http_cmd]: http status code OK 06:39:31 DEBUG --- status code --- 06:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:31 DEBUG --- http response --- 06:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:39:31 DEBUG ----- output ----- 06:39:31 DEBUG { 06:39:31 DEBUG "status": "success", 06:39:31 DEBUG "data": { 06:39:31 DEBUG "resultType": "vector", 06:39:31 DEBUG "result": [ 06:39:31 DEBUG { 06:39:31 DEBUG "metric": { 06:39:31 DEBUG "instance": "10.142.0.25:9100" 06:39:31 DEBUG }, 06:39:31 DEBUG "value": [ 06:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:39:31 DEBUG ----- output ----- 06:39:31 DEBUG { 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG [...] 06:39:31 DEBUG 06:39:31 DEBUG "status": "success", 06:39:31 DEBUG "data": { 06:39:31 DEBUG "resultType": "vector", 06:39:31 DEBUG "result": [ 06:39:31 DEBUG 06:39:31 DEBUG { 06:39:31 DEBUG 06:39:31 DEBUG "metric": { 06:39:31 DEBUG "instance": "10.142.0.77:9100" 06:39:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:39:31 DEBUG }, 06:39:31 DEBUG "value": [ 06:39:31 DEBUG }, 06:39:31 DEBUG "value": [ 06:39:31 DEBUG 1712727571, 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG [...] 06:39:31 DEBUG "42664752.56666666" 06:39:31 DEBUG ] 06:39:31 DEBUG } 06:39:31 DEBUG ] 06:39:31 DEBUG } 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:39:31 DEBUG }, 06:39:31 DEBUG } 06:39:31 DEBUG "value": [ 06:39:31 DEBUG 1712727571, 06:39:31 DEBUG "1383.0666666666666" 06:39:31 DEBUG ] 06:39:31 DEBUG } 06:39:31 DEBUG ] 06:39:31 DEBUG } 06:39:31 DEBUG } 06:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727571" 06:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727571" 06:39:31 INFO [http_cmd]: http status code OK 06:39:31 INFO [http_cmd]: http status code OK 06:39:31 DEBUG --- status code --- 06:39:31 DEBUG --- status code --- 06:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:31 DEBUG --- http response --- 06:39:31 DEBUG --- http response --- 06:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:39:31 DEBUG ----- output ----- 06:39:31 DEBUG ----- output ----- 06:39:31 DEBUG { 06:39:31 DEBUG { 06:39:31 DEBUG "status": "success", 06:39:31 DEBUG "status": "success", 06:39:31 DEBUG "data": { 06:39:31 DEBUG "data": { 06:39:31 DEBUG "resultType": "vector", 06:39:31 DEBUG "resultType": "vector", 06:39:31 DEBUG "result": [ 06:39:31 DEBUG "result": [ 06:39:31 DEBUG { 06:39:31 DEBUG { 06:39:31 DEBUG "metric": { 06:39:31 DEBUG "metric": { 06:39:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:39:31 DEBUG "pod": "ds-idrepo-0" 06:39:31 DEBUG }, 06:39:31 DEBUG }, 06:39:31 DEBUG "value": [ 06:39:31 DEBUG "value": [ 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG [...] 06:39:31 DEBUG [...] 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG "pod": "ds-idrepo-1" 06:39:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:39:31 DEBUG }, 06:39:31 DEBUG }, 06:39:31 DEBUG "value": [ 06:39:31 DEBUG "value": [ 06:39:31 DEBUG 1712727571, 06:39:31 DEBUG 1712727571, 06:39:31 DEBUG "0" 06:39:31 DEBUG "1383.3333333333333" 06:39:31 DEBUG ] 06:39:31 DEBUG ] 06:39:31 DEBUG } 06:39:31 DEBUG } 06:39:31 DEBUG ] 06:39:31 DEBUG ] 06:39:31 DEBUG } 06:39:31 DEBUG } 06:39:31 DEBUG } 06:39:31 DEBUG } 06:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727571" 06:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727571" 06:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727571" 06:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727571" 06:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727571" 06:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727571" 06:39:31 INFO [http_cmd]: http status code OK 06:39:31 INFO [http_cmd]: http status code OK 06:39:31 DEBUG --- status code --- 06:39:31 DEBUG --- status code --- 06:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:31 DEBUG --- http response --- 06:39:31 DEBUG --- http response --- 06:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:39:31 DEBUG ----- output ----- 06:39:31 DEBUG ----- output ----- 06:39:31 DEBUG { 06:39:31 DEBUG { 06:39:31 INFO [http_cmd]: http status code OK 06:39:31 DEBUG "status": "success", 06:39:31 DEBUG "status": "success", 06:39:31 DEBUG --- status code --- 06:39:31 DEBUG "data": { 06:39:31 INFO [http_cmd]: http status code OK 06:39:31 DEBUG "data": { 06:39:31 INFO [http_cmd]: http status code OK 06:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:31 DEBUG "resultType": "vector", 06:39:31 DEBUG --- status code --- 06:39:31 DEBUG "resultType": "vector", 06:39:31 DEBUG --- status code --- 06:39:31 DEBUG --- http response --- 06:39:31 DEBUG "result": [ 06:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:31 DEBUG "result": [ 06:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:39:31 DEBUG { 06:39:31 DEBUG --- http response --- 06:39:31 DEBUG { 06:39:31 DEBUG --- http response --- 06:39:31 DEBUG ----- output ----- 06:39:31 DEBUG "metric": { 06:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:39:31 DEBUG "metric": { 06:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:39:31 DEBUG { 06:39:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:39:31 DEBUG ----- output ----- 06:39:31 DEBUG "pod": "ds-idrepo-0" 06:39:31 INFO [http_cmd]: http status code OK 06:39:31 DEBUG ----- output ----- 06:39:31 DEBUG { 06:39:31 DEBUG }, 06:39:31 DEBUG "value": [ 06:39:31 DEBUG }, 06:39:31 DEBUG --- status code --- 06:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:31 DEBUG "status": "success", 06:39:31 DEBUG { 06:39:31 DEBUG "status": "success", 06:39:31 DEBUG "value": [ 06:39:31 DEBUG --- http response --- 06:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:39:31 DEBUG "data": { 06:39:31 DEBUG "data": { 06:39:31 DEBUG "resultType": "vector", 06:39:31 DEBUG 06:39:31 DEBUG "status": "success", 06:39:31 DEBUG "data": { 06:39:31 DEBUG "resultType": "vector", 06:39:31 DEBUG "result": [ 06:39:31 DEBUG { 06:39:31 DEBUG 06:39:31 DEBUG ----- output ----- 06:39:31 DEBUG { 06:39:31 DEBUG "result": [ 06:39:31 DEBUG "metric": { 06:39:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:39:31 DEBUG 06:39:31 DEBUG "resultType": "vector", 06:39:31 DEBUG "result": [ 06:39:31 DEBUG { 06:39:31 DEBUG }, 06:39:31 DEBUG "value": [ 06:39:31 DEBUG [...] 06:39:31 DEBUG "status": "success", 06:39:31 DEBUG "data": { 06:39:31 DEBUG "metric": { 06:39:31 DEBUG "instance": "10.142.0.25:9100" 06:39:31 DEBUG }, 06:39:31 DEBUG 06:39:31 DEBUG "resultType": "vector", 06:39:31 DEBUG { 06:39:31 DEBUG "value": [ 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG "result": [ 06:39:31 DEBUG "metric": { 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG [...] 06:39:31 DEBUG 06:39:31 DEBUG { 06:39:31 DEBUG "metric": { 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG "pod": "ds-cts-2" 06:39:31 DEBUG "pod": "ds-idrepo-0" 06:39:31 DEBUG }, 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:39:31 DEBUG }, 06:39:31 DEBUG "value": [ 06:39:31 DEBUG 06:39:31 DEBUG [...] 06:39:31 DEBUG 06:39:31 DEBUG }, 06:39:31 DEBUG "value": [ 06:39:31 DEBUG "instance": "10.142.0.25:9100" 06:39:31 DEBUG }, 06:39:31 DEBUG "value": [ 06:39:31 DEBUG 06:39:31 DEBUG "value": [ 06:39:31 DEBUG 1712727571, 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG 1712727571, 06:39:31 DEBUG "29.7" 06:39:31 DEBUG ] 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG "instance": "10.142.0.77:9100" 06:39:31 DEBUG "0" 06:39:31 DEBUG ] 06:39:31 DEBUG } 06:39:31 DEBUG ] 06:39:31 DEBUG } 06:39:31 DEBUG }, 06:39:31 DEBUG } 06:39:31 DEBUG ] 06:39:31 DEBUG } 06:39:31 DEBUG [...] 06:39:31 DEBUG 06:39:31 DEBUG "value": [ 06:39:31 DEBUG } 06:39:31 DEBUG } 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG "pod": "ds-cts-2" 06:39:31 DEBUG }, 06:39:31 DEBUG 1712727571, 06:39:31 DEBUG "421205451776" 06:39:31 DEBUG ] 06:39:31 DEBUG "value": [ 06:39:31 DEBUG } 06:39:31 DEBUG ] 06:39:31 DEBUG } 06:39:31 DEBUG 1712727571, 06:39:31 DEBUG } 06:39:31 DEBUG [...] 06:39:31 DEBUG 06:39:31 DEBUG "0" 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG ] 06:39:31 DEBUG } 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG ] 06:39:31 DEBUG [...] 06:39:31 DEBUG 06:39:31 DEBUG } 06:39:31 DEBUG } 06:39:31 DEBUG "instance": "10.142.0.77:9100" 06:39:31 DEBUG }, 06:39:31 DEBUG "value": [ 06:39:31 DEBUG 1712727571, 06:39:31 DEBUG "0.16580000000000084" 06:39:31 DEBUG ] 06:39:31 DEBUG } 06:39:31 DEBUG ] 06:39:31 DEBUG } 06:39:31 DEBUG } 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:39:31 DEBUG }, 06:39:31 DEBUG "value": [ 06:39:31 DEBUG 1712727571, 06:39:31 DEBUG "252585" 06:39:31 DEBUG ] 06:39:31 DEBUG } 06:39:31 DEBUG ] 06:39:31 DEBUG } 06:39:31 DEBUG } 06:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727571" 06:39:31 INFO [http_cmd]: http status code OK 06:39:31 DEBUG --- status code --- 06:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:31 DEBUG --- http response --- 06:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:39:31 DEBUG ----- output ----- 06:39:31 DEBUG { 06:39:31 DEBUG "status": "success", 06:39:31 DEBUG "data": { 06:39:31 DEBUG "resultType": "vector", 06:39:31 DEBUG "result": [ 06:39:31 DEBUG { 06:39:31 DEBUG "metric": { 06:39:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:39:31 DEBUG }, 06:39:31 DEBUG "value": [ 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG [...] 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:39:31 DEBUG }, 06:39:31 DEBUG "value": [ 06:39:31 DEBUG 1712727571, 06:39:31 DEBUG "3848.796494245812" 06:39:31 DEBUG ] 06:39:31 DEBUG } 06:39:31 DEBUG ] 06:39:31 DEBUG } 06:39:31 DEBUG } 06:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712727571" 06:39:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712727571" 06:39:31 INFO [http_cmd]: http status code OK 06:39:31 DEBUG --- status code --- 06:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:31 DEBUG --- http response --- 06:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:39:31 DEBUG ----- output ----- 06:39:31 DEBUG { 06:39:31 DEBUG "status": "success", 06:39:31 DEBUG "data": { 06:39:31 DEBUG "resultType": "vector", 06:39:31 DEBUG "result": [ 06:39:31 DEBUG { 06:39:31 DEBUG "metric": { 06:39:31 DEBUG "pod": "ds-idrepo-0" 06:39:31 DEBUG }, 06:39:31 DEBUG "value": [ 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG [...] 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG "pod": "ds-cts-2" 06:39:31 DEBUG }, 06:39:31 DEBUG "value": [ 06:39:31 DEBUG 1712727571, 06:39:31 DEBUG "0" 06:39:31 DEBUG ] 06:39:31 DEBUG } 06:39:31 DEBUG ] 06:39:31 DEBUG } 06:39:31 DEBUG } 06:39:31 INFO [http_cmd]: http status code OK 06:39:31 DEBUG --- status code --- 06:39:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:31 DEBUG --- http response --- 06:39:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:39:31 DEBUG ----- output ----- 06:39:31 DEBUG { 06:39:31 DEBUG "status": "success", 06:39:31 DEBUG "data": { 06:39:31 DEBUG "resultType": "vector", 06:39:31 DEBUG "result": [ 06:39:31 DEBUG { 06:39:31 DEBUG "metric": { 06:39:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:39:31 DEBUG }, 06:39:31 DEBUG "value": [ 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG [...] 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG 06:39:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:39:31 DEBUG }, 06:39:31 DEBUG "value": [ 06:39:31 DEBUG 1712727571, 06:39:31 DEBUG "4.43455382231946" 06:39:31 DEBUG ] 06:39:31 DEBUG } 06:39:31 DEBUG ] 06:39:31 DEBUG } 06:39:31 DEBUG } 06:39:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712727572" 06:39:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727572" 06:39:32 INFO [http_cmd]: http status code OK 06:39:32 DEBUG --- status code --- 06:39:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727572" 06:39:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727572" 06:39:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:32 INFO [http_cmd]: http status code OK 06:39:32 DEBUG --- http response --- 06:39:32 DEBUG --- status code --- 06:39:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:39:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:32 DEBUG ----- output ----- 06:39:32 DEBUG --- http response --- 06:39:32 DEBUG { 06:39:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:39:32 DEBUG "status": "success", 06:39:32 DEBUG ----- output ----- 06:39:32 DEBUG "data": { 06:39:32 DEBUG { 06:39:32 DEBUG "resultType": "vector", 06:39:32 DEBUG "status": "success", 06:39:32 DEBUG "result": [ 06:39:32 DEBUG "data": { 06:39:32 DEBUG { 06:39:32 DEBUG "resultType": "vector", 06:39:32 DEBUG "metric": { 06:39:32 DEBUG "result": [ 06:39:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:39:32 DEBUG { 06:39:32 DEBUG }, 06:39:32 DEBUG "metric": { 06:39:32 DEBUG "value": [ 06:39:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:39:32 DEBUG }, 06:39:32 DEBUG 06:39:32 DEBUG "value": [ 06:39:32 DEBUG 06:39:32 DEBUG 06:39:32 DEBUG 06:39:32 DEBUG 06:39:32 INFO [http_cmd]: http status code OK 06:39:32 DEBUG 06:39:32 DEBUG [...] 06:39:32 DEBUG --- status code --- 06:39:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:32 DEBUG 06:39:32 DEBUG --- http response --- 06:39:32 DEBUG [...] 06:39:32 DEBUG 06:39:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:39:32 DEBUG 06:39:32 DEBUG 06:39:32 DEBUG 06:39:32 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:39:32 DEBUG }, 06:39:32 DEBUG ----- output ----- 06:39:32 DEBUG "value": [ 06:39:32 DEBUG 06:39:32 DEBUG { 06:39:32 DEBUG "status": "success", 06:39:32 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:39:32 DEBUG "data": { 06:39:32 DEBUG 1712727572, 06:39:32 DEBUG "1383.6333333333332" 06:39:32 DEBUG "resultType": "vector", 06:39:32 DEBUG "result": [ 06:39:32 DEBUG { 06:39:32 DEBUG "metric": { 06:39:32 DEBUG "pod": "ds-idrepo-2" 06:39:32 DEBUG }, 06:39:32 DEBUG "value": [ 06:39:32 DEBUG 06:39:32 DEBUG 06:39:32 DEBUG 06:39:32 DEBUG [...] 06:39:32 DEBUG 06:39:32 DEBUG 06:39:32 DEBUG 06:39:32 DEBUG "pod": "ds-cts-2" 06:39:32 DEBUG }, 06:39:32 DEBUG "value": [ 06:39:32 DEBUG 1712727572, 06:39:32 DEBUG "0.0003106466666666667" 06:39:32 DEBUG ] 06:39:32 DEBUG } 06:39:32 DEBUG ] 06:39:32 DEBUG } 06:39:32 DEBUG } 06:39:32 DEBUG }, 06:39:32 DEBUG ] 06:39:32 INFO [http_cmd]: http status code OK 06:39:32 DEBUG --- status code --- 06:39:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:32 DEBUG "value": [ 06:39:32 DEBUG } 06:39:32 DEBUG --- http response --- 06:39:32 DEBUG 1712727572, 06:39:32 DEBUG ] 06:39:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:39:32 DEBUG ----- output ----- 06:39:32 DEBUG { 06:39:32 DEBUG } 06:39:32 DEBUG "status": "success", 06:39:32 DEBUG "data": { 06:39:32 DEBUG } 06:39:32 DEBUG "resultType": "vector", 06:39:32 DEBUG "result": [ 06:39:32 DEBUG { 06:39:32 DEBUG "metric": { 06:39:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:39:32 DEBUG }, 06:39:32 DEBUG "value": [ 06:39:32 DEBUG 06:39:32 DEBUG 06:39:32 DEBUG 06:39:32 DEBUG [...] 06:39:32 DEBUG 06:39:32 DEBUG 06:39:32 DEBUG 06:39:32 DEBUG "pod": "ds-idrepo-0" 06:39:32 DEBUG }, 06:39:32 DEBUG "value": [ 06:39:32 DEBUG 1712727572, 06:39:32 DEBUG "10.829223485640474" 06:39:32 DEBUG ] 06:39:32 DEBUG } 06:39:32 DEBUG ] 06:39:32 DEBUG } 06:39:32 DEBUG } 06:39:32 DEBUG "0.053" 06:39:32 DEBUG ] 06:39:32 DEBUG } 06:39:32 DEBUG ] 06:39:32 DEBUG } 06:39:32 DEBUG } 06:39:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727572" 06:39:32 INFO [http_cmd]: http status code OK 06:39:32 DEBUG --- status code --- 06:39:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:32 DEBUG --- http response --- 06:39:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:39:32 DEBUG ----- output ----- 06:39:32 DEBUG { 06:39:32 DEBUG "status": "success", 06:39:32 DEBUG "data": { 06:39:32 DEBUG "resultType": "vector", 06:39:32 DEBUG "result": [ 06:39:32 DEBUG { 06:39:32 DEBUG "metric": { 06:39:32 DEBUG "pod": "ds-idrepo-0" 06:39:32 DEBUG }, 06:39:32 DEBUG "value": [ 06:39:32 DEBUG 06:39:32 DEBUG 06:39:32 DEBUG 06:39:32 DEBUG [...] 06:39:32 DEBUG 06:39:32 DEBUG 06:39:32 DEBUG 06:39:32 DEBUG "pod": "ds-idrepo-1" 06:39:32 DEBUG }, 06:39:32 DEBUG "value": [ 06:39:32 DEBUG 1712727572, 06:39:32 DEBUG "0" 06:39:32 DEBUG ] 06:39:32 DEBUG } 06:39:32 DEBUG ] 06:39:32 DEBUG } 06:39:32 DEBUG } 06:39:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727572" 06:39:32 INFO [http_cmd]: http status code OK 06:39:32 DEBUG --- status code --- 06:39:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:32 DEBUG --- http response --- 06:39:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:39:32 DEBUG ----- output ----- 06:39:32 DEBUG { 06:39:32 DEBUG "status": "success", 06:39:32 DEBUG "data": { 06:39:32 DEBUG "resultType": "vector", 06:39:32 DEBUG "result": [ 06:39:32 DEBUG { 06:39:32 DEBUG "metric": { 06:39:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:39:32 DEBUG }, 06:39:32 DEBUG "value": [ 06:39:32 DEBUG 06:39:32 DEBUG 06:39:32 DEBUG 06:39:32 DEBUG [...] 06:39:32 DEBUG 06:39:32 DEBUG 06:39:32 DEBUG 06:39:32 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:39:32 DEBUG }, 06:39:32 DEBUG "value": [ 06:39:32 DEBUG 1712727572, 06:39:32 DEBUG "1380.8" 06:39:32 DEBUG ] 06:39:32 DEBUG } 06:39:32 DEBUG ] 06:39:32 DEBUG } 06:39:32 DEBUG } 06:39:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727572" 06:39:32 INFO [http_cmd]: http status code OK 06:39:32 DEBUG --- status code --- 06:39:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:32 DEBUG --- http response --- 06:39:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:39:32 DEBUG ----- output ----- 06:39:32 DEBUG { 06:39:32 DEBUG "status": "success", 06:39:32 DEBUG "data": { 06:39:32 DEBUG "resultType": "vector", 06:39:32 DEBUG "result": [ 06:39:32 DEBUG { 06:39:32 DEBUG "metric": { 06:39:32 DEBUG "pod": "ds-idrepo-0" 06:39:32 DEBUG }, 06:39:32 DEBUG "value": [ 06:39:32 DEBUG 06:39:32 DEBUG 06:39:32 DEBUG 06:39:32 DEBUG [...] 06:39:32 DEBUG 06:39:32 DEBUG 06:39:32 DEBUG 06:39:32 DEBUG "pod": "ds-idrepo-1" 06:39:32 DEBUG }, 06:39:32 DEBUG "value": [ 06:39:32 DEBUG 1712727572, 06:39:32 DEBUG "0" 06:39:32 DEBUG ] 06:39:32 DEBUG } 06:39:32 DEBUG ] 06:39:32 DEBUG } 06:39:32 DEBUG } 06:39:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727572" 06:39:32 INFO [http_cmd]: http status code OK 06:39:32 DEBUG --- status code --- 06:39:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:32 DEBUG --- http response --- 06:39:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:39:32 DEBUG ----- output ----- 06:39:32 DEBUG { 06:39:32 DEBUG "status": "success", 06:39:32 DEBUG "data": { 06:39:32 DEBUG "resultType": "vector", 06:39:32 DEBUG "result": [ 06:39:32 DEBUG { 06:39:32 DEBUG "metric": { 06:39:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:39:32 DEBUG }, 06:39:32 DEBUG "value": [ 06:39:32 DEBUG 06:39:32 DEBUG 06:39:32 DEBUG 06:39:32 DEBUG [...] 06:39:32 DEBUG 06:39:32 DEBUG 06:39:32 DEBUG 06:39:32 DEBUG "pod": "ds-idrepo-0" 06:39:32 DEBUG }, 06:39:32 DEBUG "value": [ 06:39:32 DEBUG 1712727572, 06:39:32 DEBUG "3701.448702380616" 06:39:32 DEBUG ] 06:39:32 DEBUG } 06:39:32 DEBUG ] 06:39:32 DEBUG } 06:39:32 DEBUG } 06:39:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712727572" 06:39:33 INFO [http_cmd]: http status code OK 06:39:33 DEBUG --- status code --- 06:39:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:33 DEBUG --- http response --- 06:39:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:39:33 DEBUG ----- output ----- 06:39:33 DEBUG { 06:39:33 DEBUG "status": "success", 06:39:33 DEBUG "data": { 06:39:33 DEBUG "resultType": "vector", 06:39:33 DEBUG "result": [ 06:39:33 DEBUG { 06:39:33 DEBUG "metric": { 06:39:33 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:39:33 DEBUG }, 06:39:33 DEBUG "value": [ 06:39:33 DEBUG 06:39:33 DEBUG 06:39:33 DEBUG 06:39:33 DEBUG [...] 06:39:33 DEBUG 06:39:33 DEBUG 06:39:33 DEBUG 06:39:33 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:39:33 DEBUG }, 06:39:33 DEBUG "value": [ 06:39:33 DEBUG 1712727572, 06:39:33 DEBUG "2486231040" 06:39:33 DEBUG ] 06:39:33 DEBUG } 06:39:33 DEBUG ] 06:39:33 DEBUG } 06:39:33 DEBUG } 06:39:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727573" 06:39:33 INFO [http_cmd]: http status code OK 06:39:33 DEBUG --- status code --- 06:39:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:33 DEBUG --- http response --- 06:39:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:39:33 DEBUG ----- output ----- 06:39:33 DEBUG { 06:39:33 DEBUG "status": "success", 06:39:33 DEBUG "data": { 06:39:33 DEBUG "resultType": "vector", 06:39:33 DEBUG "result": [ 06:39:33 DEBUG { 06:39:33 DEBUG "metric": { 06:39:33 DEBUG "instance": "10.142.0.25:9100" 06:39:33 DEBUG }, 06:39:33 DEBUG "value": [ 06:39:33 DEBUG 06:39:33 DEBUG 06:39:33 DEBUG 06:39:33 DEBUG [...] 06:39:33 DEBUG 06:39:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727573" 06:39:33 DEBUG 06:39:33 DEBUG 06:39:33 DEBUG "instance": "10.142.0.77:9100" 06:39:33 DEBUG }, 06:39:33 DEBUG "value": [ 06:39:33 DEBUG 1712727573, 06:39:33 DEBUG "42665862.900000006" 06:39:33 DEBUG ] 06:39:33 DEBUG } 06:39:33 DEBUG ] 06:39:33 DEBUG } 06:39:33 DEBUG } 06:39:33 INFO [http_cmd]: http status code OK 06:39:33 DEBUG --- status code --- 06:39:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:33 DEBUG --- http response --- 06:39:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:39:33 DEBUG ----- output ----- 06:39:33 DEBUG { 06:39:33 DEBUG "status": "success", 06:39:33 DEBUG "data": { 06:39:33 DEBUG "resultType": "vector", 06:39:33 DEBUG "result": [ 06:39:33 DEBUG { 06:39:33 DEBUG "metric": { 06:39:33 DEBUG "instance": "10.142.0.25:9100" 06:39:33 DEBUG }, 06:39:33 DEBUG "value": [ 06:39:33 DEBUG 06:39:33 DEBUG 06:39:33 DEBUG 06:39:33 DEBUG [...] 06:39:33 DEBUG 06:39:33 DEBUG 06:39:33 DEBUG 06:39:33 DEBUG "instance": "10.142.0.77:9100" 06:39:33 DEBUG }, 06:39:33 DEBUG "value": [ 06:39:33 DEBUG 1712727573, 06:39:33 DEBUG "0.0510000000000004" 06:39:33 DEBUG ] 06:39:33 DEBUG } 06:39:33 DEBUG ] 06:39:33 DEBUG } 06:39:33 DEBUG } 06:39:52 INFO 06:39:52 INFO [loop_until]: kubectl --namespace=xlou top pods 06:39:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:39:53 INFO [loop_until]: OK (rc = 0) 06:39:53 DEBUG --- stdout --- 06:39:53 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10987m 9041Mi am-654c4fcbcf-9scrj 9742m 8454Mi am-654c4fcbcf-v5qb5 8351m 8492Mi ds-cts-0 8503m 12517Mi ds-cts-1 8648m 13395Mi ds-cts-2 7811m 13394Mi ds-idrepo-0 10787m 5633Mi ds-idrepo-1 403m 5672Mi ds-idrepo-2 414m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1191Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 9m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4368m 2715Mi 06:39:53 DEBUG --- stderr --- 06:39:53 DEBUG 06:40:01 INFO 06:40:01 INFO [loop_until]: kubectl --namespace=xlou top node 06:40:01 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:40:01 INFO [loop_until]: OK (rc = 0) 06:40:01 DEBUG --- stdout --- 06:40:01 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 95m 0% 2313Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 8958m 56% 9363Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 79m 0% 3258Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13983m 87% 6896Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10580m 66% 9339Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 12024m 75% 9837Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 80m 0% 856Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10953m 68% 6322Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8275m 52% 13294Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 7234m 45% 14091Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 417m 2% 6391Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8797m 55% 14116Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 466m 2% 6360Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4509m 28% 6304Mi 10% 06:40:01 DEBUG --- stderr --- 06:40:01 DEBUG 06:40:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712727629" 06:40:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727629" 06:40:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727629" 06:40:29 INFO [http_cmd]: http status code OK 06:40:29 DEBUG --- status code --- 06:40:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:29 DEBUG --- http response --- 06:40:29 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:40:29 ERROR [http_cmd]: ERROR 06:40:29 ERROR --- status code --- 06:40:29 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:40:29 ERROR --- http response --- 06:40:29 INFO [http_cmd]: http status code OK 06:40:29 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:40:29 DEBUG --- status code --- 06:40:29 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:40:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:29 DEBUG --- http response --- 06:40:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:40:29 DEBUG ----- output ----- 06:40:29 DEBUG { 06:40:29 DEBUG "status": "success", 06:40:29 DEBUG "data": { 06:40:29 DEBUG "resultType": "vector", 06:40:29 DEBUG "result": [ 06:40:29 DEBUG { 06:40:29 DEBUG "metric": { 06:40:29 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:40:29 DEBUG }, 06:40:29 DEBUG "value": [ 06:40:29 DEBUG 06:40:29 DEBUG 06:40:29 DEBUG 06:40:29 DEBUG [...] 06:40:29 DEBUG 06:40:29 DEBUG 06:40:29 DEBUG 06:40:29 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:40:29 DEBUG }, 06:40:29 DEBUG "value": [ 06:40:29 DEBUG 1712727629, 06:40:29 DEBUG "0" 06:40:29 DEBUG ] 06:40:29 DEBUG } 06:40:29 DEBUG ] 06:40:29 DEBUG } 06:40:29 DEBUG } 06:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727631" 06:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727631" 06:40:31 INFO [http_cmd]: http status code OK 06:40:31 DEBUG --- status code --- 06:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:31 DEBUG --- http response --- 06:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:40:31 DEBUG ----- output ----- 06:40:31 DEBUG { 06:40:31 DEBUG "status": "success", 06:40:31 DEBUG "data": { 06:40:31 DEBUG "resultType": "vector", 06:40:31 DEBUG "result": [ 06:40:31 DEBUG { 06:40:31 DEBUG "metric": { 06:40:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:40:31 DEBUG }, 06:40:31 DEBUG "value": [ 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG [...] 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:40:31 DEBUG }, 06:40:31 DEBUG "value": [ 06:40:31 DEBUG 1712727631, 06:40:31 DEBUG "1399.880015997867" 06:40:31 DEBUG ] 06:40:31 DEBUG } 06:40:31 DEBUG ] 06:40:31 DEBUG } 06:40:31 DEBUG } 06:40:31 INFO [http_cmd]: http status code OK 06:40:31 DEBUG --- status code --- 06:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:31 DEBUG --- http response --- 06:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:40:31 DEBUG ----- output ----- 06:40:31 DEBUG { 06:40:31 DEBUG "status": "success", 06:40:31 DEBUG "data": { 06:40:31 DEBUG "resultType": "vector", 06:40:31 DEBUG "result": [ 06:40:31 DEBUG { 06:40:31 DEBUG "metric": { 06:40:31 DEBUG "instance": "10.142.0.25:9100" 06:40:31 DEBUG }, 06:40:31 DEBUG "value": [ 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG [...] 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG "instance": "10.142.0.77:9100" 06:40:31 DEBUG }, 06:40:31 DEBUG "value": [ 06:40:31 DEBUG 1712727631, 06:40:31 DEBUG "42616448.4" 06:40:31 DEBUG ] 06:40:31 DEBUG } 06:40:31 DEBUG ] 06:40:31 DEBUG } 06:40:31 DEBUG } 06:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727631" 06:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727631" 06:40:31 INFO [http_cmd]: http status code OK 06:40:31 INFO [http_cmd]: http status code OK 06:40:31 DEBUG --- status code --- 06:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:31 DEBUG --- status code --- 06:40:31 DEBUG --- http response --- 06:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:40:31 DEBUG --- http response --- 06:40:31 DEBUG ----- output ----- 06:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:40:31 DEBUG ----- output ----- 06:40:31 DEBUG { 06:40:31 DEBUG "status": "success", 06:40:31 DEBUG "data": { 06:40:31 DEBUG "resultType": "vector", 06:40:31 DEBUG "result": [ 06:40:31 DEBUG { 06:40:31 DEBUG "metric": { 06:40:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:40:31 DEBUG }, 06:40:31 DEBUG "value": [ 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG [...] 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:40:31 DEBUG }, 06:40:31 DEBUG "value": [ 06:40:31 DEBUG 1712727631, 06:40:31 DEBUG "1400.5799226769764" 06:40:31 DEBUG ] 06:40:31 DEBUG } 06:40:31 DEBUG ] 06:40:31 DEBUG } 06:40:31 DEBUG } 06:40:31 DEBUG { 06:40:31 DEBUG "status": "success", 06:40:31 DEBUG "data": { 06:40:31 DEBUG "resultType": "vector", 06:40:31 DEBUG "result": [ 06:40:31 DEBUG { 06:40:31 DEBUG "metric": { 06:40:31 DEBUG "pod": "ds-idrepo-0" 06:40:31 DEBUG }, 06:40:31 DEBUG "value": [ 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG [...] 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG "pod": "ds-idrepo-1" 06:40:31 DEBUG }, 06:40:31 DEBUG "value": [ 06:40:31 DEBUG 1712727631, 06:40:31 DEBUG "0" 06:40:31 DEBUG ] 06:40:31 DEBUG } 06:40:31 DEBUG ] 06:40:31 DEBUG } 06:40:31 DEBUG } 06:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727631" 06:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727631" 06:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727631" 06:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727631" 06:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727631" 06:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727631" 06:40:31 INFO [http_cmd]: http status code OK 06:40:31 INFO [http_cmd]: http status code OK 06:40:31 DEBUG --- status code --- 06:40:31 DEBUG --- status code --- 06:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:31 DEBUG --- http response --- 06:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:40:31 DEBUG ----- output ----- 06:40:31 DEBUG { 06:40:31 DEBUG "status": "success", 06:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:31 DEBUG --- http response --- 06:40:31 DEBUG "data": { 06:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:40:31 INFO [http_cmd]: http status code OK 06:40:31 DEBUG "resultType": "vector", 06:40:31 DEBUG "result": [ 06:40:31 DEBUG { 06:40:31 DEBUG ----- output ----- 06:40:31 DEBUG { 06:40:31 DEBUG "status": "success", 06:40:31 DEBUG "metric": { 06:40:31 INFO [http_cmd]: http status code OK 06:40:31 DEBUG --- status code --- 06:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:31 DEBUG --- http response --- 06:40:31 DEBUG "data": { 06:40:31 DEBUG "resultType": "vector", 06:40:31 DEBUG "result": [ 06:40:31 DEBUG "instance": "10.142.0.25:9100" 06:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:40:31 DEBUG ----- output ----- 06:40:31 DEBUG --- status code --- 06:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:31 DEBUG }, 06:40:31 DEBUG { 06:40:31 DEBUG --- http response --- 06:40:31 INFO [http_cmd]: http status code OK 06:40:31 DEBUG --- status code --- 06:40:31 DEBUG "value": [ 06:40:31 DEBUG "status": "success", 06:40:31 INFO [http_cmd]: http status code OK 06:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:40:31 DEBUG ----- output ----- 06:40:31 DEBUG { 06:40:31 DEBUG 06:40:31 DEBUG "data": { 06:40:31 DEBUG --- status code --- 06:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:31 DEBUG --- http response --- 06:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:40:31 DEBUG 06:40:31 DEBUG "resultType": "vector", 06:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:31 DEBUG --- http response --- 06:40:31 DEBUG ----- output ----- 06:40:31 DEBUG { 06:40:31 DEBUG "status": "success", 06:40:31 DEBUG 06:40:31 DEBUG "status": "success", 06:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:40:31 DEBUG "data": { 06:40:31 DEBUG "resultType": "vector", 06:40:31 DEBUG "result": [ 06:40:31 DEBUG { 06:40:31 DEBUG "metric": { 06:40:31 DEBUG ----- output ----- 06:40:31 DEBUG [...] 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG "result": [ 06:40:31 DEBUG { 06:40:31 DEBUG "metric": { 06:40:31 DEBUG 06:40:31 DEBUG "data": { 06:40:31 DEBUG "instance": "10.142.0.25:9100" 06:40:31 DEBUG { 06:40:31 DEBUG "status": "success", 06:40:31 DEBUG "data": { 06:40:31 DEBUG "resultType": "vector", 06:40:31 DEBUG "resultType": "vector", 06:40:31 DEBUG "result": [ 06:40:31 DEBUG { 06:40:31 DEBUG "result": [ 06:40:31 DEBUG { 06:40:31 DEBUG "metric": { 06:40:31 DEBUG "pod": "ds-idrepo-0" 06:40:31 DEBUG }, 06:40:31 DEBUG "value": [ 06:40:31 DEBUG "metric": { 06:40:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:40:31 DEBUG }, 06:40:31 DEBUG "value": [ 06:40:31 DEBUG "pod": "ds-idrepo-0" 06:40:31 DEBUG }, 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG "value": [ 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG [...] 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG [...] 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:40:31 DEBUG }, 06:40:31 DEBUG "value": [ 06:40:31 DEBUG }, 06:40:31 DEBUG { 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG [...] 06:40:31 DEBUG 06:40:31 DEBUG "instance": "10.142.0.77:9100" 06:40:31 DEBUG "metric": { 06:40:31 DEBUG 1712727631, 06:40:31 DEBUG "value": [ 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG }, 06:40:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:40:31 DEBUG }, 06:40:31 DEBUG "value": [ 06:40:31 DEBUG "pod": "ds-cts-2" 06:40:31 DEBUG 06:40:31 DEBUG "value": [ 06:40:31 DEBUG "252098" 06:40:31 DEBUG ] 06:40:31 DEBUG 06:40:31 DEBUG }, 06:40:31 DEBUG "pod": "ds-cts-2" 06:40:31 DEBUG 1712727631, 06:40:31 DEBUG } 06:40:31 DEBUG ] 06:40:31 DEBUG 06:40:31 DEBUG "value": [ 06:40:31 DEBUG }, 06:40:31 DEBUG "423225730048" 06:40:31 DEBUG } 06:40:31 DEBUG } 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG "value": [ 06:40:31 DEBUG 1712727631, 06:40:31 DEBUG "0" 06:40:31 DEBUG ] 06:40:31 DEBUG } 06:40:31 DEBUG ] 06:40:31 DEBUG [...] 06:40:31 DEBUG ] 06:40:31 DEBUG } 06:40:31 DEBUG 1712727631, 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG } 06:40:31 DEBUG } 06:40:31 DEBUG "29.869653632029866" 06:40:31 DEBUG 06:40:31 DEBUG [...] 06:40:31 DEBUG ] 06:40:31 DEBUG ] 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG } 06:40:31 DEBUG } 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG } 06:40:31 DEBUG ] 06:40:31 DEBUG } 06:40:31 DEBUG "instance": "10.142.0.77:9100" 06:40:31 DEBUG 06:40:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:40:31 DEBUG }, 06:40:31 DEBUG } 06:40:31 DEBUG }, 06:40:31 DEBUG "value": [ 06:40:31 DEBUG "value": [ 06:40:31 DEBUG 1712727631, 06:40:31 DEBUG "0.04353333333333419" 06:40:31 DEBUG ] 06:40:31 DEBUG } 06:40:31 DEBUG ] 06:40:31 DEBUG } 06:40:31 DEBUG } 06:40:31 DEBUG 1712727631, 06:40:31 DEBUG "0" 06:40:31 DEBUG ] 06:40:31 DEBUG } 06:40:31 DEBUG ] 06:40:31 DEBUG } 06:40:31 DEBUG } 06:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727631" 06:40:31 INFO [http_cmd]: http status code OK 06:40:31 DEBUG --- status code --- 06:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:31 DEBUG --- http response --- 06:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:40:31 DEBUG ----- output ----- 06:40:31 DEBUG { 06:40:31 DEBUG "status": "success", 06:40:31 DEBUG "data": { 06:40:31 DEBUG "resultType": "vector", 06:40:31 DEBUG "result": [ 06:40:31 DEBUG { 06:40:31 DEBUG "metric": { 06:40:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:40:31 DEBUG }, 06:40:31 DEBUG "value": [ 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG [...] 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:40:31 DEBUG }, 06:40:31 DEBUG "value": [ 06:40:31 DEBUG 1712727631, 06:40:31 DEBUG "254145800.06281942" 06:40:31 DEBUG ] 06:40:31 DEBUG } 06:40:31 DEBUG ] 06:40:31 DEBUG } 06:40:31 DEBUG } 06:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712727631" 06:40:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712727631" 06:40:31 INFO [http_cmd]: http status code OK 06:40:31 DEBUG --- status code --- 06:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:31 DEBUG --- http response --- 06:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:40:31 DEBUG ----- output ----- 06:40:31 DEBUG { 06:40:31 DEBUG "status": "success", 06:40:31 DEBUG "data": { 06:40:31 DEBUG "resultType": "vector", 06:40:31 DEBUG "result": [ 06:40:31 DEBUG { 06:40:31 DEBUG "metric": { 06:40:31 DEBUG "pod": "ds-idrepo-0" 06:40:31 DEBUG }, 06:40:31 DEBUG "value": [ 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG [...] 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG "pod": "ds-cts-2" 06:40:31 DEBUG }, 06:40:31 DEBUG "value": [ 06:40:31 DEBUG 1712727631, 06:40:31 DEBUG "0" 06:40:31 DEBUG ] 06:40:31 DEBUG } 06:40:31 DEBUG ] 06:40:31 DEBUG } 06:40:31 DEBUG } 06:40:31 INFO [http_cmd]: http status code OK 06:40:31 DEBUG --- status code --- 06:40:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:31 DEBUG --- http response --- 06:40:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:40:31 DEBUG ----- output ----- 06:40:31 DEBUG { 06:40:31 DEBUG "status": "success", 06:40:31 DEBUG "data": { 06:40:31 DEBUG "resultType": "vector", 06:40:31 DEBUG "result": [ 06:40:31 DEBUG { 06:40:31 DEBUG "metric": { 06:40:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:40:31 DEBUG }, 06:40:31 DEBUG "value": [ 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG [...] 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG 06:40:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:40:31 DEBUG }, 06:40:31 DEBUG "value": [ 06:40:31 DEBUG 1712727631, 06:40:31 DEBUG "4.427597868805942" 06:40:31 DEBUG ] 06:40:31 DEBUG } 06:40:31 DEBUG ] 06:40:31 DEBUG } 06:40:31 DEBUG } 06:40:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727632" 06:40:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727632" 06:40:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727632" 06:40:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712727632" 06:40:32 INFO [http_cmd]: http status code OK 06:40:32 INFO [http_cmd]: http status code OK 06:40:32 DEBUG --- status code --- 06:40:32 DEBUG --- status code --- 06:40:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:32 DEBUG --- http response --- 06:40:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:40:32 DEBUG --- http response --- 06:40:32 DEBUG ----- output ----- 06:40:32 DEBUG { 06:40:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:40:32 INFO [http_cmd]: http status code OK 06:40:32 DEBUG --- status code --- 06:40:32 DEBUG ----- output ----- 06:40:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:32 DEBUG --- http response --- 06:40:32 DEBUG { 06:40:32 INFO [http_cmd]: http status code OK 06:40:32 DEBUG "status": "success", 06:40:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:40:32 DEBUG "status": "success", 06:40:32 DEBUG --- status code --- 06:40:32 DEBUG "data": { 06:40:32 DEBUG ----- output ----- 06:40:32 DEBUG "data": { 06:40:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:32 DEBUG "resultType": "vector", 06:40:32 DEBUG { 06:40:32 DEBUG "status": "success", 06:40:32 DEBUG --- http response --- 06:40:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:40:32 DEBUG ----- output ----- 06:40:32 DEBUG { 06:40:32 DEBUG "status": "success", 06:40:32 DEBUG "data": { 06:40:32 DEBUG "resultType": "vector", 06:40:32 DEBUG "result": [ 06:40:32 DEBUG { 06:40:32 DEBUG "metric": { 06:40:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:40:32 DEBUG }, 06:40:32 DEBUG "value": [ 06:40:32 DEBUG 06:40:32 DEBUG 06:40:32 DEBUG 06:40:32 DEBUG [...] 06:40:32 DEBUG 06:40:32 DEBUG 06:40:32 DEBUG 06:40:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:40:32 DEBUG }, 06:40:32 DEBUG "value": [ 06:40:32 DEBUG "data": { 06:40:32 DEBUG 1712727632, 06:40:32 DEBUG "result": [ 06:40:32 DEBUG "resultType": "vector", 06:40:32 DEBUG "resultType": "vector", 06:40:32 DEBUG "8.394357762713684" 06:40:32 DEBUG { 06:40:32 DEBUG "metric": { 06:40:32 DEBUG "result": [ 06:40:32 DEBUG { 06:40:32 DEBUG "metric": { 06:40:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:40:32 DEBUG "result": [ 06:40:32 DEBUG { 06:40:32 DEBUG "metric": { 06:40:32 DEBUG }, 06:40:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:40:32 DEBUG }, 06:40:32 DEBUG "value": [ 06:40:32 DEBUG "value": [ 06:40:32 DEBUG 06:40:32 DEBUG 06:40:32 DEBUG 06:40:32 DEBUG [...] 06:40:32 DEBUG 06:40:32 DEBUG 06:40:32 DEBUG 06:40:32 DEBUG [...] 06:40:32 DEBUG ] 06:40:32 DEBUG } 06:40:32 DEBUG ] 06:40:32 DEBUG 06:40:32 DEBUG 06:40:32 DEBUG 06:40:32 DEBUG 06:40:32 DEBUG 06:40:32 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:40:32 DEBUG }, 06:40:32 DEBUG "value": [ 06:40:32 DEBUG 1712727632, 06:40:32 DEBUG "0.053" 06:40:32 DEBUG 06:40:32 DEBUG } 06:40:32 DEBUG } 06:40:32 DEBUG "pod": "ds-idrepo-2" 06:40:32 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:40:32 DEBUG ] 06:40:32 DEBUG }, 06:40:32 DEBUG }, 06:40:32 DEBUG } 06:40:32 DEBUG ] 06:40:32 DEBUG } 06:40:32 DEBUG } 06:40:32 DEBUG "value": [ 06:40:32 DEBUG 06:40:32 DEBUG 06:40:32 DEBUG 06:40:32 DEBUG "value": [ 06:40:32 DEBUG 1712727632, 06:40:32 DEBUG [...] 06:40:32 DEBUG 06:40:32 DEBUG 06:40:32 DEBUG 06:40:32 DEBUG "pod": "ds-cts-2" 06:40:32 DEBUG }, 06:40:32 DEBUG "value": [ 06:40:32 DEBUG 1712727632, 06:40:32 DEBUG "0.023565446544654468" 06:40:32 DEBUG ] 06:40:32 DEBUG } 06:40:32 DEBUG ] 06:40:32 DEBUG } 06:40:32 DEBUG } 06:40:32 DEBUG "1399.9466737768298" 06:40:32 DEBUG ] 06:40:32 DEBUG } 06:40:32 DEBUG ] 06:40:32 DEBUG } 06:40:32 DEBUG } 06:40:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727632" 06:40:32 INFO [http_cmd]: http status code OK 06:40:32 DEBUG --- status code --- 06:40:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:32 DEBUG --- http response --- 06:40:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:40:32 DEBUG ----- output ----- 06:40:32 DEBUG { 06:40:32 DEBUG "status": "success", 06:40:32 DEBUG "data": { 06:40:32 DEBUG "resultType": "vector", 06:40:32 DEBUG "result": [ 06:40:32 DEBUG { 06:40:32 DEBUG "metric": { 06:40:32 DEBUG "pod": "ds-idrepo-0" 06:40:32 DEBUG }, 06:40:32 DEBUG "value": [ 06:40:32 DEBUG 06:40:32 DEBUG 06:40:32 DEBUG 06:40:32 DEBUG [...] 06:40:32 DEBUG 06:40:32 DEBUG 06:40:32 DEBUG 06:40:32 DEBUG "pod": "ds-idrepo-1" 06:40:32 DEBUG }, 06:40:32 DEBUG "value": [ 06:40:32 DEBUG 1712727632, 06:40:32 DEBUG "0" 06:40:32 DEBUG ] 06:40:32 DEBUG } 06:40:32 DEBUG ] 06:40:32 DEBUG } 06:40:32 DEBUG } 06:40:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727632" 06:40:32 INFO [http_cmd]: http status code OK 06:40:32 DEBUG --- status code --- 06:40:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:32 DEBUG --- http response --- 06:40:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:40:32 DEBUG ----- output ----- 06:40:32 DEBUG { 06:40:32 DEBUG "status": "success", 06:40:32 DEBUG "data": { 06:40:32 DEBUG "resultType": "vector", 06:40:32 DEBUG "result": [ 06:40:32 DEBUG { 06:40:32 DEBUG "metric": { 06:40:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:40:32 DEBUG }, 06:40:32 DEBUG "value": [ 06:40:32 DEBUG 06:40:32 DEBUG 06:40:32 DEBUG 06:40:32 DEBUG [...] 06:40:32 DEBUG 06:40:32 DEBUG 06:40:32 DEBUG 06:40:32 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:40:32 DEBUG }, 06:40:32 DEBUG "value": [ 06:40:32 DEBUG 1712727632, 06:40:32 DEBUG "1398.4468737501668" 06:40:32 DEBUG ] 06:40:32 DEBUG } 06:40:32 DEBUG ] 06:40:32 DEBUG } 06:40:32 DEBUG } 06:40:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727632" 06:40:32 INFO [http_cmd]: http status code OK 06:40:32 DEBUG --- status code --- 06:40:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:32 DEBUG --- http response --- 06:40:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:40:32 DEBUG ----- output ----- 06:40:32 DEBUG { 06:40:32 DEBUG "status": "success", 06:40:32 DEBUG "data": { 06:40:32 DEBUG "resultType": "vector", 06:40:32 DEBUG "result": [ 06:40:32 DEBUG { 06:40:32 DEBUG "metric": { 06:40:32 DEBUG "pod": "ds-idrepo-0" 06:40:32 DEBUG }, 06:40:32 DEBUG "value": [ 06:40:32 DEBUG 06:40:32 DEBUG 06:40:32 DEBUG 06:40:32 DEBUG [...] 06:40:32 DEBUG 06:40:32 DEBUG 06:40:32 DEBUG 06:40:32 DEBUG "pod": "ds-idrepo-1" 06:40:32 DEBUG }, 06:40:32 DEBUG "value": [ 06:40:32 DEBUG 1712727632, 06:40:32 DEBUG "0" 06:40:32 DEBUG ] 06:40:32 DEBUG } 06:40:32 DEBUG ] 06:40:32 DEBUG } 06:40:32 DEBUG } 06:40:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727633" 06:40:33 INFO [http_cmd]: http status code OK 06:40:33 DEBUG --- status code --- 06:40:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:33 DEBUG --- http response --- 06:40:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:40:33 DEBUG ----- output ----- 06:40:33 DEBUG { 06:40:33 DEBUG "status": "success", 06:40:33 DEBUG "data": { 06:40:33 DEBUG "resultType": "vector", 06:40:33 DEBUG "result": [ 06:40:33 DEBUG { 06:40:33 DEBUG "metric": { 06:40:33 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:40:33 DEBUG }, 06:40:33 DEBUG "value": [ 06:40:33 DEBUG 06:40:33 DEBUG 06:40:33 DEBUG 06:40:33 DEBUG [...] 06:40:33 DEBUG 06:40:33 DEBUG 06:40:33 DEBUG 06:40:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:40:33 DEBUG }, 06:40:33 DEBUG "value": [ 06:40:33 DEBUG 1712727633, 06:40:33 DEBUG "0" 06:40:33 DEBUG ] 06:40:33 DEBUG } 06:40:33 DEBUG ] 06:40:33 DEBUG } 06:40:33 DEBUG } 06:40:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712727633" 06:40:33 INFO [http_cmd]: http status code OK 06:40:33 DEBUG --- status code --- 06:40:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:33 DEBUG --- http response --- 06:40:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:40:33 DEBUG ----- output ----- 06:40:33 DEBUG { 06:40:33 DEBUG "status": "success", 06:40:33 DEBUG "data": { 06:40:33 DEBUG "resultType": "vector", 06:40:33 DEBUG "result": [ 06:40:33 DEBUG { 06:40:33 DEBUG "metric": { 06:40:33 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:40:33 DEBUG }, 06:40:33 DEBUG "value": [ 06:40:33 DEBUG 06:40:33 DEBUG 06:40:33 DEBUG 06:40:33 DEBUG [...] 06:40:33 DEBUG 06:40:33 DEBUG 06:40:33 DEBUG 06:40:33 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:40:33 DEBUG }, 06:40:33 DEBUG "value": [ 06:40:33 DEBUG 1712727633, 06:40:33 DEBUG "2488893440" 06:40:33 DEBUG ] 06:40:33 DEBUG } 06:40:33 DEBUG ] 06:40:33 DEBUG } 06:40:33 DEBUG } 06:40:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727633" 06:40:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727633" 06:40:33 INFO [http_cmd]: http status code OK 06:40:33 DEBUG --- status code --- 06:40:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:33 DEBUG --- http response --- 06:40:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:40:33 DEBUG ----- output ----- 06:40:33 DEBUG { 06:40:33 DEBUG "status": "success", 06:40:33 DEBUG "data": { 06:40:33 DEBUG "resultType": "vector", 06:40:33 DEBUG "result": [ 06:40:33 DEBUG { 06:40:33 DEBUG "metric": { 06:40:33 DEBUG "instance": "10.142.0.25:9100" 06:40:33 DEBUG }, 06:40:33 DEBUG "value": [ 06:40:33 DEBUG 06:40:33 DEBUG 06:40:33 DEBUG 06:40:33 DEBUG [...] 06:40:33 DEBUG 06:40:33 DEBUG 06:40:33 DEBUG 06:40:33 DEBUG "instance": "10.142.0.77:9100" 06:40:33 DEBUG }, 06:40:33 DEBUG "value": [ 06:40:33 DEBUG 1712727633, 06:40:33 DEBUG "42619204.733333334" 06:40:33 DEBUG ] 06:40:33 DEBUG } 06:40:33 DEBUG ] 06:40:33 DEBUG } 06:40:33 DEBUG } 06:40:33 INFO [http_cmd]: http status code OK 06:40:33 DEBUG --- status code --- 06:40:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:33 DEBUG --- http response --- 06:40:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:40:33 DEBUG ----- output ----- 06:40:33 DEBUG { 06:40:33 DEBUG "status": "success", 06:40:33 DEBUG "data": { 06:40:33 DEBUG "resultType": "vector", 06:40:33 DEBUG "result": [ 06:40:33 DEBUG { 06:40:33 DEBUG "metric": { 06:40:33 DEBUG "instance": "10.142.0.25:9100" 06:40:33 DEBUG }, 06:40:33 DEBUG "value": [ 06:40:33 DEBUG 06:40:33 DEBUG 06:40:33 DEBUG 06:40:33 DEBUG [...] 06:40:33 DEBUG 06:40:33 DEBUG 06:40:33 DEBUG 06:40:33 DEBUG "instance": "10.142.0.77:9100" 06:40:33 DEBUG }, 06:40:33 DEBUG "value": [ 06:40:33 DEBUG 1712727633, 06:40:33 DEBUG "0.017666666666666466" 06:40:33 DEBUG ] 06:40:33 DEBUG } 06:40:33 DEBUG ] 06:40:33 DEBUG } 06:40:33 DEBUG } 06:40:53 INFO 06:40:53 INFO [loop_until]: kubectl --namespace=xlou top pods 06:40:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:40:53 INFO [loop_until]: OK (rc = 0) 06:40:53 DEBUG --- stdout --- 06:40:53 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 11130m 9035Mi am-654c4fcbcf-9scrj 10044m 8456Mi am-654c4fcbcf-v5qb5 8459m 8490Mi ds-cts-0 8781m 13150Mi ds-cts-1 8270m 12889Mi ds-cts-2 7410m 12911Mi ds-idrepo-0 10659m 5634Mi ds-idrepo-1 368m 5672Mi ds-idrepo-2 385m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1191Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 9m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4458m 2716Mi 06:40:53 DEBUG --- stderr --- 06:40:53 DEBUG 06:41:01 INFO 06:41:01 INFO [loop_until]: kubectl --namespace=xlou top node 06:41:01 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:41:01 INFO [loop_until]: OK (rc = 0) 06:41:01 DEBUG --- stdout --- 06:41:01 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 95m 0% 2309Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9511m 59% 9364Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 77m 0% 3257Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 14205m 89% 6897Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 11004m 69% 9341Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11598m 72% 9844Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 79m 0% 857Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10993m 69% 6321Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8702m 54% 13907Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 7606m 47% 13718Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 447m 2% 6392Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8516m 53% 13581Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 409m 2% 6363Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4512m 28% 6304Mi 10% 06:41:01 DEBUG --- stderr --- 06:41:01 DEBUG 06:41:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727689" 06:41:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712727689" 06:41:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727689" 06:41:29 ERROR [http_cmd]: ERROR 06:41:29 INFO [http_cmd]: http status code OK 06:41:29 ERROR --- status code --- 06:41:29 DEBUG --- status code --- 06:41:29 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:41:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:29 ERROR --- http response --- 06:41:30 DEBUG --- http response --- 06:41:30 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:41:30 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:41:30 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:41:30 INFO [http_cmd]: http status code OK 06:41:30 DEBUG --- status code --- 06:41:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:30 DEBUG --- http response --- 06:41:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:41:30 DEBUG ----- output ----- 06:41:30 DEBUG { 06:41:30 DEBUG "status": "success", 06:41:30 DEBUG "data": { 06:41:30 DEBUG "resultType": "vector", 06:41:30 DEBUG "result": [ 06:41:30 DEBUG { 06:41:30 DEBUG "metric": { 06:41:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:41:30 DEBUG }, 06:41:30 DEBUG "value": [ 06:41:30 DEBUG 06:41:30 DEBUG 06:41:30 DEBUG 06:41:30 DEBUG [...] 06:41:30 DEBUG 06:41:30 DEBUG 06:41:30 DEBUG 06:41:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:41:30 DEBUG }, 06:41:30 DEBUG "value": [ 06:41:30 DEBUG 1712727689, 06:41:30 DEBUG "0" 06:41:30 DEBUG ] 06:41:30 DEBUG } 06:41:30 DEBUG ] 06:41:30 DEBUG } 06:41:30 DEBUG } 06:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727691" 06:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727691" 06:41:31 INFO [http_cmd]: http status code OK 06:41:31 DEBUG --- status code --- 06:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:31 DEBUG --- http response --- 06:41:31 INFO [http_cmd]: http status code OK 06:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:41:31 DEBUG --- status code --- 06:41:31 DEBUG ----- output ----- 06:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:31 DEBUG { 06:41:31 DEBUG --- http response --- 06:41:31 DEBUG "status": "success", 06:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:41:31 DEBUG "data": { 06:41:31 DEBUG "resultType": "vector", 06:41:31 DEBUG ----- output ----- 06:41:31 DEBUG "result": [ 06:41:31 DEBUG { 06:41:31 DEBUG { 06:41:31 DEBUG "status": "success", 06:41:31 DEBUG "metric": { 06:41:31 DEBUG "data": { 06:41:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:41:31 DEBUG "resultType": "vector", 06:41:31 DEBUG }, 06:41:31 DEBUG "result": [ 06:41:31 DEBUG "value": [ 06:41:31 DEBUG { 06:41:31 DEBUG 06:41:31 DEBUG "metric": { 06:41:31 DEBUG 06:41:31 DEBUG "instance": "10.142.0.25:9100" 06:41:31 DEBUG 06:41:31 DEBUG }, 06:41:31 DEBUG [...] 06:41:31 DEBUG "value": [ 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:41:31 DEBUG [...] 06:41:31 DEBUG }, 06:41:31 DEBUG 06:41:31 DEBUG "value": [ 06:41:31 DEBUG 06:41:31 DEBUG 1712727691, 06:41:31 DEBUG 06:41:31 DEBUG "1187.6333333333332" 06:41:31 DEBUG "instance": "10.142.0.77:9100" 06:41:31 DEBUG ] 06:41:31 DEBUG } 06:41:31 DEBUG }, 06:41:31 DEBUG ] 06:41:31 DEBUG "value": [ 06:41:31 DEBUG } 06:41:31 DEBUG 1712727691, 06:41:31 DEBUG "42882380.86666667" 06:41:31 DEBUG ] 06:41:31 DEBUG } 06:41:31 DEBUG } 06:41:31 DEBUG ] 06:41:31 DEBUG } 06:41:31 DEBUG } 06:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727691" 06:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727691" 06:41:31 INFO [http_cmd]: http status code OK 06:41:31 DEBUG --- status code --- 06:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:31 DEBUG --- http response --- 06:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:41:31 DEBUG ----- output ----- 06:41:31 DEBUG { 06:41:31 DEBUG "status": "success", 06:41:31 DEBUG "data": { 06:41:31 DEBUG "resultType": "vector", 06:41:31 DEBUG "result": [ 06:41:31 DEBUG { 06:41:31 DEBUG "metric": { 06:41:31 DEBUG "pod": "ds-idrepo-0" 06:41:31 DEBUG }, 06:41:31 DEBUG "value": [ 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG [...] 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG "pod": "ds-idrepo-1" 06:41:31 DEBUG }, 06:41:31 DEBUG "value": [ 06:41:31 DEBUG 1712727691, 06:41:31 DEBUG "0" 06:41:31 DEBUG ] 06:41:31 DEBUG } 06:41:31 DEBUG ] 06:41:31 DEBUG } 06:41:31 DEBUG } 06:41:31 INFO [http_cmd]: http status code OK 06:41:31 DEBUG --- status code --- 06:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:31 DEBUG --- http response --- 06:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:41:31 DEBUG ----- output ----- 06:41:31 DEBUG { 06:41:31 DEBUG "status": "success", 06:41:31 DEBUG "data": { 06:41:31 DEBUG "resultType": "vector", 06:41:31 DEBUG "result": [ 06:41:31 DEBUG { 06:41:31 DEBUG "metric": { 06:41:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:41:31 DEBUG }, 06:41:31 DEBUG "value": [ 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG [...] 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:41:31 DEBUG }, 06:41:31 DEBUG "value": [ 06:41:31 DEBUG 1712727691, 06:41:31 DEBUG "1186.9" 06:41:31 DEBUG ] 06:41:31 DEBUG } 06:41:31 DEBUG ] 06:41:31 DEBUG } 06:41:31 DEBUG } 06:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727691" 06:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727691" 06:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727691" 06:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727691" 06:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727691" 06:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727691" 06:41:31 INFO [http_cmd]: http status code OK 06:41:31 DEBUG --- status code --- 06:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:31 DEBUG --- http response --- 06:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:41:31 INFO [http_cmd]: http status code OK 06:41:31 INFO [http_cmd]: http status code OK 06:41:31 DEBUG --- status code --- 06:41:31 DEBUG --- status code --- 06:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:31 DEBUG --- http response --- 06:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:31 DEBUG --- http response --- 06:41:31 INFO [http_cmd]: http status code OK 06:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:41:31 DEBUG ----- output ----- 06:41:31 DEBUG --- status code --- 06:41:31 DEBUG ----- output ----- 06:41:31 DEBUG { 06:41:31 INFO [http_cmd]: http status code OK 06:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:41:31 INFO [http_cmd]: http status code OK 06:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:31 DEBUG { 06:41:31 DEBUG "status": "success", 06:41:31 DEBUG --- status code --- 06:41:31 DEBUG ----- output ----- 06:41:31 DEBUG --- status code --- 06:41:31 DEBUG --- http response --- 06:41:31 DEBUG "status": "success", 06:41:31 DEBUG "data": { 06:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:31 DEBUG { 06:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:41:31 DEBUG "data": { 06:41:31 DEBUG "resultType": "vector", 06:41:31 DEBUG --- http response --- 06:41:31 DEBUG "status": "success", 06:41:31 DEBUG "data": { 06:41:31 DEBUG "resultType": "vector", 06:41:31 DEBUG "resultType": "vector", 06:41:31 DEBUG "result": [ 06:41:31 DEBUG --- http response --- 06:41:31 DEBUG ----- output ----- 06:41:31 DEBUG { 06:41:31 DEBUG "status": "success", 06:41:31 DEBUG "result": [ 06:41:31 DEBUG { 06:41:31 DEBUG "metric": { 06:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:41:31 DEBUG "data": { 06:41:31 DEBUG "resultType": "vector", 06:41:31 DEBUG { 06:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:41:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:41:31 DEBUG ----- output ----- 06:41:31 DEBUG { 06:41:31 DEBUG "status": "success", 06:41:31 DEBUG "metric": { 06:41:31 DEBUG "instance": "10.142.0.25:9100" 06:41:31 DEBUG }, 06:41:31 DEBUG "data": { 06:41:31 DEBUG "resultType": "vector", 06:41:31 DEBUG "result": [ 06:41:31 DEBUG { 06:41:31 DEBUG "metric": { 06:41:31 DEBUG "value": [ 06:41:31 DEBUG }, 06:41:31 DEBUG "value": [ 06:41:31 DEBUG ----- output ----- 06:41:31 DEBUG { 06:41:31 DEBUG "status": "success", 06:41:31 DEBUG 06:41:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:41:31 DEBUG }, 06:41:31 DEBUG 06:41:31 DEBUG "result": [ 06:41:31 DEBUG "result": [ 06:41:31 DEBUG 06:41:31 DEBUG "data": { 06:41:31 DEBUG "value": [ 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG { 06:41:31 DEBUG "resultType": "vector", 06:41:31 DEBUG 06:41:31 DEBUG [...] 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG "result": [ 06:41:31 DEBUG { 06:41:31 DEBUG [...] 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG "metric": { 06:41:31 DEBUG "instance": "10.142.0.25:9100" 06:41:31 DEBUG }, 06:41:31 DEBUG 06:41:31 DEBUG "instance": "10.142.0.77:9100" 06:41:31 DEBUG }, 06:41:31 DEBUG "value": [ 06:41:31 DEBUG 06:41:31 DEBUG { 06:41:31 DEBUG "metric": { 06:41:31 DEBUG [...] 06:41:31 DEBUG "value": [ 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG [...] 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG 1712727691, 06:41:31 DEBUG "pod": "ds-idrepo-0" 06:41:31 DEBUG }, 06:41:31 DEBUG "metric": { 06:41:31 DEBUG "pod": "ds-idrepo-0" 06:41:31 DEBUG }, 06:41:31 DEBUG "425394883584" 06:41:31 DEBUG "value": [ 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG "instance": "10.142.0.77:9100" 06:41:31 DEBUG }, 06:41:31 DEBUG ] 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG "value": [ 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG } 06:41:31 DEBUG [...] 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG "value": [ 06:41:31 DEBUG 1712727691, 06:41:31 DEBUG ] 06:41:31 DEBUG 06:41:31 DEBUG "pod": "ds-cts-2" 06:41:31 DEBUG }, 06:41:31 DEBUG "value": [ 06:41:31 DEBUG 1712727691, 06:41:31 DEBUG } 06:41:31 DEBUG "0.1573999999999984" 06:41:31 DEBUG ] 06:41:31 DEBUG "29.633333333333333" 06:41:31 DEBUG ] 06:41:31 DEBUG } 06:41:31 DEBUG } 06:41:31 DEBUG } 06:41:31 DEBUG ] 06:41:31 DEBUG ] 06:41:31 DEBUG } 06:41:31 DEBUG } 06:41:31 DEBUG } 06:41:31 DEBUG } 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG [...] 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:41:31 DEBUG 06:41:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:41:31 DEBUG }, 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG "value": [ 06:41:31 DEBUG "pod": "ds-cts-2" 06:41:31 DEBUG }, 06:41:31 DEBUG 1712727691, 06:41:31 DEBUG "0" 06:41:31 DEBUG ] 06:41:31 DEBUG "value": [ 06:41:31 DEBUG 1712727691, 06:41:31 DEBUG "0" 06:41:31 DEBUG } 06:41:31 DEBUG ] 06:41:31 DEBUG } 06:41:31 DEBUG } 06:41:31 DEBUG ] 06:41:31 DEBUG } 06:41:31 DEBUG ] 06:41:31 DEBUG } 06:41:31 DEBUG } 06:41:31 DEBUG }, 06:41:31 DEBUG "value": [ 06:41:31 DEBUG 1712727691, 06:41:31 DEBUG "256852" 06:41:31 DEBUG ] 06:41:31 DEBUG } 06:41:31 DEBUG ] 06:41:31 DEBUG } 06:41:31 DEBUG } 06:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727691" 06:41:31 INFO [http_cmd]: http status code OK 06:41:31 DEBUG --- status code --- 06:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:31 DEBUG --- http response --- 06:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:41:31 DEBUG ----- output ----- 06:41:31 DEBUG { 06:41:31 DEBUG "status": "success", 06:41:31 DEBUG "data": { 06:41:31 DEBUG "resultType": "vector", 06:41:31 DEBUG "result": [ 06:41:31 DEBUG { 06:41:31 DEBUG "metric": { 06:41:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:41:31 DEBUG }, 06:41:31 DEBUG "value": [ 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG [...] 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:41:31 DEBUG }, 06:41:31 DEBUG "value": [ 06:41:31 DEBUG 1712727691, 06:41:31 DEBUG "270314235.7722247" 06:41:31 DEBUG ] 06:41:31 DEBUG } 06:41:31 DEBUG ] 06:41:31 DEBUG } 06:41:31 DEBUG } 06:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712727691" 06:41:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712727691" 06:41:31 INFO [http_cmd]: http status code OK 06:41:31 DEBUG --- status code --- 06:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:31 DEBUG --- http response --- 06:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:41:31 DEBUG ----- output ----- 06:41:31 DEBUG { 06:41:31 DEBUG "status": "success", 06:41:31 DEBUG "data": { 06:41:31 DEBUG "resultType": "vector", 06:41:31 DEBUG "result": [ 06:41:31 DEBUG { 06:41:31 DEBUG "metric": { 06:41:31 DEBUG "pod": "ds-idrepo-0" 06:41:31 DEBUG }, 06:41:31 DEBUG "value": [ 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG [...] 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG "pod": "ds-cts-2" 06:41:31 DEBUG }, 06:41:31 DEBUG "value": [ 06:41:31 DEBUG 1712727691, 06:41:31 DEBUG "0" 06:41:31 DEBUG ] 06:41:31 DEBUG } 06:41:31 DEBUG ] 06:41:31 DEBUG } 06:41:31 DEBUG } 06:41:31 INFO [http_cmd]: http status code OK 06:41:31 DEBUG --- status code --- 06:41:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:31 DEBUG --- http response --- 06:41:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:41:31 DEBUG ----- output ----- 06:41:31 DEBUG { 06:41:31 DEBUG "status": "success", 06:41:31 DEBUG "data": { 06:41:31 DEBUG "resultType": "vector", 06:41:31 DEBUG "result": [ 06:41:31 DEBUG { 06:41:31 DEBUG "metric": { 06:41:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:41:31 DEBUG }, 06:41:31 DEBUG "value": [ 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG [...] 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG 06:41:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:41:31 DEBUG }, 06:41:31 DEBUG "value": [ 06:41:31 DEBUG 1712727691, 06:41:31 DEBUG "4.430480375393042" 06:41:31 DEBUG ] 06:41:31 DEBUG } 06:41:31 DEBUG ] 06:41:31 DEBUG } 06:41:31 DEBUG } 06:41:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727692" 06:41:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712727692" 06:41:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727692" 06:41:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727692" 06:41:32 INFO [http_cmd]: http status code OK 06:41:32 DEBUG --- status code --- 06:41:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:32 DEBUG --- http response --- 06:41:32 INFO [http_cmd]: http status code OK 06:41:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:41:32 DEBUG --- status code --- 06:41:32 INFO [http_cmd]: http status code OK 06:41:32 DEBUG --- status code --- 06:41:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:32 DEBUG --- http response --- 06:41:32 DEBUG --- http response --- 06:41:32 DEBUG ----- output ----- 06:41:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:41:32 DEBUG ----- output ----- 06:41:32 DEBUG { 06:41:32 DEBUG { 06:41:32 DEBUG "status": "success", 06:41:32 DEBUG "status": "success", 06:41:32 DEBUG "data": { 06:41:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:41:32 DEBUG "data": { 06:41:32 DEBUG "resultType": "vector", 06:41:32 INFO [http_cmd]: http status code OK 06:41:32 DEBUG --- status code --- 06:41:32 DEBUG "resultType": "vector", 06:41:32 DEBUG "result": [ 06:41:32 DEBUG { 06:41:32 DEBUG ----- output ----- 06:41:32 DEBUG "result": [ 06:41:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:32 DEBUG "metric": { 06:41:32 DEBUG { 06:41:32 DEBUG { 06:41:32 DEBUG --- http response --- 06:41:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:41:32 DEBUG "status": "success", 06:41:32 DEBUG "metric": { 06:41:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:41:32 DEBUG }, 06:41:32 DEBUG "value": [ 06:41:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:41:32 DEBUG ----- output ----- 06:41:32 DEBUG "data": { 06:41:32 DEBUG "resultType": "vector", 06:41:32 DEBUG }, 06:41:32 DEBUG { 06:41:32 DEBUG "result": [ 06:41:32 DEBUG 06:41:32 DEBUG "value": [ 06:41:32 DEBUG "status": "success", 06:41:32 DEBUG { 06:41:32 DEBUG 06:41:32 DEBUG 06:41:32 DEBUG [...] 06:41:32 DEBUG "data": { 06:41:32 DEBUG "metric": { 06:41:32 DEBUG 06:41:32 DEBUG 06:41:32 DEBUG "resultType": "vector", 06:41:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:41:32 DEBUG 06:41:32 DEBUG 06:41:32 DEBUG "result": [ 06:41:32 DEBUG }, 06:41:32 DEBUG 06:41:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:41:32 DEBUG }, 06:41:32 DEBUG "value": [ 06:41:32 DEBUG 06:41:32 DEBUG { 06:41:32 DEBUG "metric": { 06:41:32 DEBUG 06:41:32 DEBUG [...] 06:41:32 DEBUG 06:41:32 DEBUG 06:41:32 DEBUG 06:41:32 DEBUG "pod": "ds-idrepo-2" 06:41:32 DEBUG "value": [ 06:41:32 DEBUG 1712727692, 06:41:32 DEBUG 06:41:32 DEBUG [...] 06:41:32 DEBUG "0.051" 06:41:32 DEBUG ] 06:41:32 DEBUG 06:41:32 DEBUG 06:41:32 DEBUG } 06:41:32 DEBUG 06:41:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:41:32 DEBUG 06:41:32 DEBUG ] 06:41:32 DEBUG }, 06:41:32 DEBUG }, 06:41:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:41:32 DEBUG } 06:41:32 DEBUG "value": [ 06:41:32 DEBUG "value": [ 06:41:32 DEBUG }, 06:41:32 DEBUG "value": [ 06:41:32 DEBUG 1712727692, 06:41:32 DEBUG 06:41:32 DEBUG 06:41:32 DEBUG 1712727692, 06:41:32 DEBUG "4.880812468052369" 06:41:32 DEBUG ] 06:41:32 DEBUG } 06:41:32 DEBUG "1187.0666666666666" 06:41:32 DEBUG ] 06:41:32 DEBUG } 06:41:32 DEBUG } 06:41:32 DEBUG ] 06:41:32 DEBUG 06:41:32 DEBUG } 06:41:32 DEBUG } 06:41:32 DEBUG [...] 06:41:32 DEBUG ] 06:41:32 DEBUG 06:41:32 DEBUG } 06:41:32 DEBUG 06:41:32 DEBUG } 06:41:32 DEBUG 06:41:32 DEBUG "pod": "ds-cts-2" 06:41:32 DEBUG }, 06:41:32 DEBUG "value": [ 06:41:32 DEBUG 1712727692, 06:41:32 DEBUG "0.00006666666666666667" 06:41:32 DEBUG ] 06:41:32 DEBUG } 06:41:32 DEBUG ] 06:41:32 DEBUG } 06:41:32 DEBUG } 06:41:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727692" 06:41:32 INFO [http_cmd]: http status code OK 06:41:32 DEBUG --- status code --- 06:41:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:32 DEBUG --- http response --- 06:41:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:41:32 DEBUG ----- output ----- 06:41:32 DEBUG { 06:41:32 DEBUG "status": "success", 06:41:32 DEBUG "data": { 06:41:32 DEBUG "resultType": "vector", 06:41:32 DEBUG "result": [ 06:41:32 DEBUG { 06:41:32 DEBUG "metric": { 06:41:32 DEBUG "pod": "ds-idrepo-0" 06:41:32 DEBUG }, 06:41:32 DEBUG "value": [ 06:41:32 DEBUG 06:41:32 DEBUG 06:41:32 DEBUG 06:41:32 DEBUG [...] 06:41:32 DEBUG 06:41:32 DEBUG 06:41:32 DEBUG 06:41:32 DEBUG "pod": "ds-idrepo-1" 06:41:32 DEBUG }, 06:41:32 DEBUG "value": [ 06:41:32 DEBUG 1712727692, 06:41:32 DEBUG "0" 06:41:32 DEBUG ] 06:41:32 DEBUG } 06:41:32 DEBUG ] 06:41:32 DEBUG } 06:41:32 DEBUG } 06:41:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727692" 06:41:32 INFO [http_cmd]: http status code OK 06:41:32 DEBUG --- status code --- 06:41:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:32 DEBUG --- http response --- 06:41:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:41:32 DEBUG ----- output ----- 06:41:32 DEBUG { 06:41:32 DEBUG "status": "success", 06:41:32 DEBUG "data": { 06:41:32 DEBUG "resultType": "vector", 06:41:32 DEBUG "result": [ 06:41:32 DEBUG { 06:41:32 DEBUG "metric": { 06:41:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:41:32 DEBUG }, 06:41:32 DEBUG "value": [ 06:41:32 DEBUG 06:41:32 DEBUG 06:41:32 DEBUG 06:41:32 DEBUG [...] 06:41:32 DEBUG 06:41:32 DEBUG 06:41:32 DEBUG 06:41:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:41:32 DEBUG }, 06:41:32 DEBUG "value": [ 06:41:32 DEBUG 1712727692, 06:41:32 DEBUG "1185.3666666666666" 06:41:32 DEBUG ] 06:41:32 DEBUG } 06:41:32 DEBUG ] 06:41:32 DEBUG } 06:41:32 DEBUG } 06:41:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727692" 06:41:32 INFO [http_cmd]: http status code OK 06:41:32 DEBUG --- status code --- 06:41:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:32 DEBUG --- http response --- 06:41:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:41:32 DEBUG ----- output ----- 06:41:32 DEBUG { 06:41:32 DEBUG "status": "success", 06:41:32 DEBUG "data": { 06:41:32 DEBUG "resultType": "vector", 06:41:32 DEBUG "result": [ 06:41:32 DEBUG { 06:41:32 DEBUG "metric": { 06:41:32 DEBUG "pod": "ds-idrepo-0" 06:41:32 DEBUG }, 06:41:32 DEBUG "value": [ 06:41:32 DEBUG 06:41:32 DEBUG 06:41:32 DEBUG 06:41:32 DEBUG [...] 06:41:32 DEBUG 06:41:32 DEBUG 06:41:32 DEBUG 06:41:32 DEBUG "pod": "ds-idrepo-1" 06:41:32 DEBUG }, 06:41:32 DEBUG "value": [ 06:41:32 DEBUG 1712727692, 06:41:32 DEBUG "0" 06:41:32 DEBUG ] 06:41:32 DEBUG } 06:41:32 DEBUG ] 06:41:32 DEBUG } 06:41:32 DEBUG } 06:41:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727693" 06:41:33 INFO [http_cmd]: http status code OK 06:41:33 DEBUG --- status code --- 06:41:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:33 DEBUG --- http response --- 06:41:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:41:33 DEBUG ----- output ----- 06:41:33 DEBUG { 06:41:33 DEBUG "status": "success", 06:41:33 DEBUG "data": { 06:41:33 DEBUG "resultType": "vector", 06:41:33 DEBUG "result": [ 06:41:33 DEBUG { 06:41:33 DEBUG "metric": { 06:41:33 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:41:33 DEBUG }, 06:41:33 DEBUG "value": [ 06:41:33 DEBUG 06:41:33 DEBUG 06:41:33 DEBUG 06:41:33 DEBUG [...] 06:41:33 DEBUG 06:41:33 DEBUG 06:41:33 DEBUG 06:41:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:41:33 DEBUG }, 06:41:33 DEBUG "value": [ 06:41:33 DEBUG 1712727693, 06:41:33 DEBUG "0" 06:41:33 DEBUG ] 06:41:33 DEBUG } 06:41:33 DEBUG ] 06:41:33 DEBUG } 06:41:33 DEBUG } 06:41:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712727693" 06:41:33 INFO [http_cmd]: http status code OK 06:41:33 DEBUG --- status code --- 06:41:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:33 DEBUG --- http response --- 06:41:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:41:33 DEBUG ----- output ----- 06:41:33 DEBUG { 06:41:33 DEBUG "status": "success", 06:41:33 DEBUG "data": { 06:41:33 DEBUG "resultType": "vector", 06:41:33 DEBUG "result": [ 06:41:33 DEBUG { 06:41:33 DEBUG "metric": { 06:41:33 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:41:33 DEBUG }, 06:41:33 DEBUG "value": [ 06:41:33 DEBUG 06:41:33 DEBUG 06:41:33 DEBUG 06:41:33 DEBUG [...] 06:41:33 DEBUG 06:41:33 DEBUG 06:41:33 DEBUG 06:41:33 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:41:33 DEBUG }, 06:41:33 DEBUG "value": [ 06:41:33 DEBUG 1712727693, 06:41:33 DEBUG "2486300672" 06:41:33 DEBUG ] 06:41:33 DEBUG } 06:41:33 DEBUG ] 06:41:33 DEBUG } 06:41:33 DEBUG } 06:41:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727693" 06:41:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727693" 06:41:33 INFO [http_cmd]: http status code OK 06:41:33 DEBUG --- status code --- 06:41:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:33 DEBUG --- http response --- 06:41:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:41:33 DEBUG ----- output ----- 06:41:33 DEBUG { 06:41:33 DEBUG "status": "success", 06:41:33 DEBUG "data": { 06:41:33 DEBUG "resultType": "vector", 06:41:33 DEBUG "result": [ 06:41:33 DEBUG { 06:41:33 DEBUG "metric": { 06:41:33 DEBUG "instance": "10.142.0.25:9100" 06:41:33 DEBUG }, 06:41:33 DEBUG "value": [ 06:41:33 DEBUG 06:41:33 DEBUG 06:41:33 DEBUG 06:41:33 DEBUG [...] 06:41:33 DEBUG 06:41:33 DEBUG 06:41:33 DEBUG 06:41:33 DEBUG "instance": "10.142.0.77:9100" 06:41:33 DEBUG }, 06:41:33 DEBUG "value": [ 06:41:33 DEBUG 1712727693, 06:41:33 DEBUG "42884674.06666666" 06:41:33 DEBUG ] 06:41:33 DEBUG } 06:41:33 DEBUG ] 06:41:33 DEBUG } 06:41:33 DEBUG } 06:41:33 INFO [http_cmd]: http status code OK 06:41:33 DEBUG --- status code --- 06:41:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:33 DEBUG --- http response --- 06:41:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:41:33 DEBUG ----- output ----- 06:41:33 DEBUG { 06:41:33 DEBUG "status": "success", 06:41:33 DEBUG "data": { 06:41:33 DEBUG "resultType": "vector", 06:41:33 DEBUG "result": [ 06:41:33 DEBUG { 06:41:33 DEBUG "metric": { 06:41:33 DEBUG "instance": "10.142.0.25:9100" 06:41:33 DEBUG }, 06:41:33 DEBUG "value": [ 06:41:33 DEBUG 06:41:33 DEBUG 06:41:33 DEBUG 06:41:33 DEBUG [...] 06:41:33 DEBUG 06:41:33 DEBUG 06:41:33 DEBUG 06:41:33 DEBUG "instance": "10.142.0.77:9100" 06:41:33 DEBUG }, 06:41:33 DEBUG "value": [ 06:41:33 DEBUG 1712727693, 06:41:33 DEBUG "0.04799999999999981" 06:41:33 DEBUG ] 06:41:33 DEBUG } 06:41:33 DEBUG ] 06:41:33 DEBUG } 06:41:33 DEBUG } 06:41:53 INFO 06:41:53 INFO [loop_until]: kubectl --namespace=xlou top pods 06:41:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:41:53 INFO [loop_until]: OK (rc = 0) 06:41:53 DEBUG --- stdout --- 06:41:53 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 11155m 9023Mi am-654c4fcbcf-9scrj 9912m 8460Mi am-654c4fcbcf-v5qb5 8499m 8489Mi ds-cts-0 8191m 13394Mi ds-cts-1 8331m 12484Mi ds-cts-2 7680m 12483Mi ds-idrepo-0 10914m 5634Mi ds-idrepo-1 350m 5672Mi ds-idrepo-2 433m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 5m 1191Mi idm-6bfb7c96cc-vsb58 5m 2213Mi lodemon-b776b7d88-thrqf 8m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4432m 2718Mi 06:41:53 DEBUG --- stderr --- 06:41:53 DEBUG 06:42:01 INFO 06:42:01 INFO [loop_until]: kubectl --namespace=xlou top node 06:42:01 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:42:02 INFO [loop_until]: OK (rc = 0) 06:42:02 DEBUG --- stdout --- 06:42:02 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 94m 0% 2307Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9351m 58% 9365Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 81m 0% 3259Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13891m 87% 6908Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 11076m 69% 9340Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11883m 74% 9831Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 77m 0% 856Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10968m 69% 6321Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8633m 54% 14106Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 7771m 48% 13290Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 485m 3% 6406Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8225m 51% 14113Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 415m 2% 6361Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4620m 29% 6306Mi 10% 06:42:02 DEBUG --- stderr --- 06:42:02 DEBUG 06:42:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712727750" 06:42:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727750" 06:42:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727750" 06:42:30 ERROR [http_cmd]: ERROR 06:42:30 ERROR --- status code --- 06:42:30 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:42:30 ERROR --- http response --- 06:42:30 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:42:30 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:42:30 INFO [http_cmd]: http status code OK 06:42:30 INFO [http_cmd]: http status code OK 06:42:30 DEBUG --- status code --- 06:42:30 DEBUG --- status code --- 06:42:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:30 DEBUG --- http response --- 06:42:30 DEBUG --- http response --- 06:42:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:42:30 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:42:30 DEBUG ----- output ----- 06:42:30 DEBUG { 06:42:30 DEBUG "status": "success", 06:42:30 DEBUG "data": { 06:42:30 DEBUG "resultType": "vector", 06:42:30 DEBUG "result": [ 06:42:30 DEBUG { 06:42:30 DEBUG "metric": { 06:42:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:42:30 DEBUG }, 06:42:30 DEBUG "value": [ 06:42:30 DEBUG 06:42:30 DEBUG 06:42:30 DEBUG 06:42:30 DEBUG [...] 06:42:30 DEBUG 06:42:30 DEBUG 06:42:30 DEBUG 06:42:30 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:42:30 DEBUG }, 06:42:30 DEBUG "value": [ 06:42:30 DEBUG 1712727750, 06:42:30 DEBUG "0" 06:42:30 DEBUG ] 06:42:30 DEBUG } 06:42:30 DEBUG ] 06:42:30 DEBUG } 06:42:30 DEBUG } 06:42:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727751" 06:42:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727751" 06:42:31 INFO [http_cmd]: http status code OK 06:42:31 DEBUG --- status code --- 06:42:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:31 DEBUG --- http response --- 06:42:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:42:31 DEBUG ----- output ----- 06:42:31 DEBUG { 06:42:31 DEBUG "status": "success", 06:42:31 INFO [http_cmd]: http status code OK 06:42:31 DEBUG "data": { 06:42:31 DEBUG --- status code --- 06:42:31 DEBUG "resultType": "vector", 06:42:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:31 DEBUG "result": [ 06:42:31 DEBUG --- http response --- 06:42:31 DEBUG { 06:42:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:42:31 DEBUG "metric": { 06:42:31 DEBUG ----- output ----- 06:42:31 DEBUG "instance": "10.142.0.25:9100" 06:42:31 DEBUG { 06:42:31 DEBUG }, 06:42:31 DEBUG "status": "success", 06:42:31 DEBUG "value": [ 06:42:31 DEBUG "data": { 06:42:31 DEBUG 06:42:31 DEBUG "resultType": "vector", 06:42:31 DEBUG 06:42:31 DEBUG "result": [ 06:42:31 DEBUG 06:42:31 DEBUG { 06:42:31 DEBUG [...] 06:42:31 DEBUG "metric": { 06:42:31 DEBUG 06:42:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:42:31 DEBUG 06:42:31 DEBUG }, 06:42:31 DEBUG 06:42:31 DEBUG "value": [ 06:42:31 DEBUG "instance": "10.142.0.77:9100" 06:42:31 DEBUG 06:42:31 DEBUG }, 06:42:31 DEBUG 06:42:31 DEBUG "value": [ 06:42:31 DEBUG 06:42:31 DEBUG 1712727751, 06:42:31 DEBUG [...] 06:42:31 DEBUG "42618576.2" 06:42:31 DEBUG 06:42:31 DEBUG ] 06:42:31 DEBUG 06:42:31 DEBUG } 06:42:31 DEBUG 06:42:31 DEBUG ] 06:42:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:42:31 DEBUG } 06:42:31 DEBUG }, 06:42:31 DEBUG } 06:42:31 DEBUG "value": [ 06:42:31 DEBUG 1712727751, 06:42:31 DEBUG "1403.8461538461538" 06:42:31 DEBUG ] 06:42:31 DEBUG } 06:42:31 DEBUG ] 06:42:31 DEBUG } 06:42:31 DEBUG } 06:42:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727751" 06:42:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727751" 06:42:31 INFO [http_cmd]: http status code OK 06:42:31 DEBUG --- status code --- 06:42:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:31 DEBUG --- http response --- 06:42:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:42:31 DEBUG ----- output ----- 06:42:31 DEBUG { 06:42:31 DEBUG "status": "success", 06:42:31 DEBUG "data": { 06:42:31 DEBUG "resultType": "vector", 06:42:31 DEBUG "result": [ 06:42:31 DEBUG { 06:42:31 DEBUG "metric": { 06:42:31 DEBUG "pod": "ds-idrepo-0" 06:42:31 DEBUG }, 06:42:31 DEBUG "value": [ 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG [...] 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG "pod": "ds-idrepo-1" 06:42:31 DEBUG }, 06:42:31 DEBUG "value": [ 06:42:31 DEBUG 1712727751, 06:42:31 DEBUG "0" 06:42:31 DEBUG ] 06:42:31 DEBUG } 06:42:31 DEBUG ] 06:42:31 DEBUG } 06:42:31 DEBUG } 06:42:31 INFO [http_cmd]: http status code OK 06:42:31 DEBUG --- status code --- 06:42:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:31 DEBUG --- http response --- 06:42:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:42:31 DEBUG ----- output ----- 06:42:31 DEBUG { 06:42:31 DEBUG "status": "success", 06:42:31 DEBUG "data": { 06:42:31 DEBUG "resultType": "vector", 06:42:31 DEBUG "result": [ 06:42:31 DEBUG { 06:42:31 DEBUG "metric": { 06:42:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:42:31 DEBUG }, 06:42:31 DEBUG "value": [ 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG [...] 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:42:31 DEBUG }, 06:42:31 DEBUG "value": [ 06:42:31 DEBUG 1712727751, 06:42:31 DEBUG "1401.8464204772697" 06:42:31 DEBUG ] 06:42:31 DEBUG } 06:42:31 DEBUG ] 06:42:31 DEBUG } 06:42:31 DEBUG } 06:42:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727751" 06:42:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727751" 06:42:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727751" 06:42:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727751" 06:42:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727751" 06:42:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727751" 06:42:31 INFO [http_cmd]: http status code OK 06:42:31 DEBUG --- status code --- 06:42:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:31 DEBUG --- http response --- 06:42:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:42:31 INFO [http_cmd]: http status code OK 06:42:31 DEBUG ----- output ----- 06:42:31 DEBUG { 06:42:31 DEBUG "status": "success", 06:42:31 DEBUG --- status code --- 06:42:31 DEBUG "data": { 06:42:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727751" 06:42:31 DEBUG "resultType": "vector", 06:42:31 DEBUG --- http response --- 06:42:31 INFO [http_cmd]: http status code OK 06:42:31 INFO [http_cmd]: http status code OK 06:42:31 INFO [http_cmd]: http status code OK 06:42:31 INFO [http_cmd]: http status code OK 06:42:31 DEBUG "result": [ 06:42:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:42:31 DEBUG --- status code --- 06:42:31 DEBUG --- status code --- 06:42:31 DEBUG --- status code --- 06:42:31 DEBUG --- status code --- 06:42:31 DEBUG { 06:42:31 DEBUG ----- output ----- 06:42:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:31 DEBUG --- http response --- 06:42:31 DEBUG "metric": { 06:42:31 DEBUG { 06:42:31 DEBUG --- http response --- 06:42:31 DEBUG --- http response --- 06:42:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:31 DEBUG --- http response --- 06:42:31 DEBUG "pod": "ds-idrepo-0" 06:42:31 DEBUG "status": "success", 06:42:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:42:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:42:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:42:31 DEBUG }, 06:42:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:42:31 DEBUG "data": { 06:42:31 DEBUG ----- output ----- 06:42:31 DEBUG ----- output ----- 06:42:31 DEBUG ----- output ----- 06:42:31 DEBUG "value": [ 06:42:31 DEBUG ----- output ----- 06:42:31 DEBUG "resultType": "vector", 06:42:31 DEBUG { 06:42:31 DEBUG { 06:42:31 DEBUG { 06:42:31 DEBUG 06:42:31 DEBUG { 06:42:31 DEBUG "status": "success", 06:42:31 DEBUG "data": { 06:42:31 DEBUG "resultType": "vector", 06:42:31 DEBUG "status": "success", 06:42:31 DEBUG 06:42:31 DEBUG "status": "success", 06:42:31 DEBUG "data": { 06:42:31 DEBUG "resultType": "vector", 06:42:31 INFO [http_cmd]: http status code OK 06:42:31 DEBUG "result": [ 06:42:31 DEBUG "data": { 06:42:31 DEBUG "resultType": "vector", 06:42:31 DEBUG "status": "success", 06:42:31 DEBUG "data": { 06:42:31 DEBUG "resultType": "vector", 06:42:31 DEBUG --- status code --- 06:42:31 DEBUG { 06:42:31 DEBUG "metric": { 06:42:31 DEBUG "instance": "10.142.0.25:9100" 06:42:31 DEBUG "result": [ 06:42:31 DEBUG { 06:42:31 DEBUG "metric": { 06:42:31 DEBUG "result": [ 06:42:31 DEBUG }, 06:42:31 DEBUG "result": [ 06:42:31 DEBUG { 06:42:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:31 DEBUG --- http response --- 06:42:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:42:31 DEBUG ----- output ----- 06:42:31 DEBUG "value": [ 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG { 06:42:31 DEBUG "status": "success", 06:42:31 DEBUG "data": { 06:42:31 DEBUG { 06:42:31 DEBUG "metric": { 06:42:31 DEBUG "instance": "10.142.0.25:9100" 06:42:31 DEBUG }, 06:42:31 DEBUG "resultType": "vector", 06:42:31 DEBUG "result": [ 06:42:31 DEBUG { 06:42:31 DEBUG "metric": { 06:42:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:42:31 DEBUG }, 06:42:31 DEBUG "value": [ 06:42:31 DEBUG "metric": { 06:42:31 DEBUG "result": [ 06:42:31 DEBUG "pod": "ds-idrepo-0" 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:42:31 DEBUG { 06:42:31 DEBUG }, 06:42:31 DEBUG [...] 06:42:31 DEBUG 06:42:31 DEBUG [...] 06:42:31 DEBUG [...] 06:42:31 DEBUG }, 06:42:31 DEBUG "metric": { 06:42:31 DEBUG "value": [ 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG "value": [ 06:42:31 DEBUG "value": [ 06:42:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:42:31 DEBUG 06:42:31 DEBUG [...] 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG }, 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG "value": [ 06:42:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:42:31 DEBUG "pod": "ds-cts-2" 06:42:31 DEBUG }, 06:42:31 DEBUG "value": [ 06:42:31 DEBUG 1712727751, 06:42:31 DEBUG 06:42:31 DEBUG [...] 06:42:31 DEBUG }, 06:42:31 DEBUG "0" 06:42:31 DEBUG "instance": "10.142.0.77:9100" 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG "value": [ 06:42:31 DEBUG 1712727751, 06:42:31 DEBUG }, 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:42:31 DEBUG "0" 06:42:31 DEBUG ] 06:42:31 DEBUG "value": [ 06:42:31 DEBUG [...] 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG } 06:42:31 DEBUG ] 06:42:31 DEBUG 1712727751, 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG [...] 06:42:31 DEBUG } 06:42:31 DEBUG } 06:42:31 DEBUG "0.043533333333333604" 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG "instance": "10.142.0.77:9100" 06:42:31 DEBUG }, 06:42:31 DEBUG ] 06:42:31 DEBUG } 06:42:31 DEBUG "pod": "ds-cts-2" 06:42:31 DEBUG }, 06:42:31 DEBUG }, 06:42:31 DEBUG "value": [ 06:42:31 DEBUG 1712727751, 06:42:31 DEBUG "287585948.2491542" 06:42:31 DEBUG ] 06:42:31 DEBUG } 06:42:31 DEBUG "value": [ 06:42:31 DEBUG 1712727751, 06:42:31 DEBUG ] 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:42:31 DEBUG }, 06:42:31 DEBUG "value": [ 06:42:31 DEBUG 1712727751, 06:42:31 DEBUG "256916" 06:42:31 DEBUG ] 06:42:31 DEBUG } 06:42:31 DEBUG "427587611648" 06:42:31 DEBUG ] 06:42:31 DEBUG ] 06:42:31 DEBUG ] 06:42:31 DEBUG } 06:42:31 DEBUG "value": [ 06:42:31 DEBUG } 06:42:31 DEBUG ] 06:42:31 DEBUG } 06:42:31 DEBUG } 06:42:31 DEBUG 1712727751, 06:42:31 DEBUG } 06:42:31 DEBUG } 06:42:31 DEBUG ] 06:42:31 DEBUG ] 06:42:31 DEBUG "29.097090290970904" 06:42:31 DEBUG } 06:42:31 DEBUG } 06:42:31 DEBUG } 06:42:31 DEBUG ] 06:42:31 DEBUG } 06:42:31 DEBUG } 06:42:31 DEBUG } 06:42:31 DEBUG } 06:42:31 DEBUG ] 06:42:31 DEBUG } 06:42:31 DEBUG } 06:42:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712727751" 06:42:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712727751" 06:42:31 INFO [http_cmd]: http status code OK 06:42:31 DEBUG --- status code --- 06:42:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:31 DEBUG --- http response --- 06:42:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:42:31 DEBUG ----- output ----- 06:42:31 DEBUG { 06:42:31 DEBUG "status": "success", 06:42:31 DEBUG "data": { 06:42:31 DEBUG "resultType": "vector", 06:42:31 DEBUG "result": [ 06:42:31 DEBUG { 06:42:31 DEBUG "metric": { 06:42:31 DEBUG "pod": "ds-idrepo-0" 06:42:31 DEBUG }, 06:42:31 DEBUG "value": [ 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG [...] 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG "pod": "ds-cts-2" 06:42:31 DEBUG }, 06:42:31 DEBUG "value": [ 06:42:31 DEBUG 1712727751, 06:42:31 DEBUG "0" 06:42:31 DEBUG ] 06:42:31 DEBUG } 06:42:31 DEBUG ] 06:42:31 DEBUG } 06:42:31 DEBUG } 06:42:31 INFO [http_cmd]: http status code OK 06:42:31 DEBUG --- status code --- 06:42:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:31 DEBUG --- http response --- 06:42:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:42:31 DEBUG ----- output ----- 06:42:31 DEBUG { 06:42:31 DEBUG "status": "success", 06:42:31 DEBUG "data": { 06:42:31 DEBUG "resultType": "vector", 06:42:31 DEBUG "result": [ 06:42:31 DEBUG { 06:42:31 DEBUG "metric": { 06:42:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:42:31 DEBUG }, 06:42:31 DEBUG "value": [ 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG [...] 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG 06:42:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:42:31 DEBUG }, 06:42:31 DEBUG "value": [ 06:42:31 DEBUG 1712727751, 06:42:31 DEBUG "4.404614420027346" 06:42:31 DEBUG ] 06:42:31 DEBUG } 06:42:31 DEBUG ] 06:42:31 DEBUG } 06:42:31 DEBUG } 06:42:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727752" 06:42:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712727752" 06:42:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727752" 06:42:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727752" 06:42:32 INFO [http_cmd]: http status code OK 06:42:32 DEBUG --- status code --- 06:42:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:32 INFO [http_cmd]: http status code OK 06:42:32 DEBUG --- status code --- 06:42:32 DEBUG --- http response --- 06:42:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:42:32 DEBUG --- http response --- 06:42:32 DEBUG ----- output ----- 06:42:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:42:32 DEBUG { 06:42:32 DEBUG ----- output ----- 06:42:32 DEBUG { 06:42:32 DEBUG "status": "success", 06:42:32 DEBUG "status": "success", 06:42:32 DEBUG "data": { 06:42:32 DEBUG "data": { 06:42:32 DEBUG "resultType": "vector", 06:42:32 DEBUG "result": [ 06:42:32 INFO [http_cmd]: http status code OK 06:42:32 INFO [http_cmd]: http status code OK 06:42:32 DEBUG --- status code --- 06:42:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:32 DEBUG --- status code --- 06:42:32 DEBUG --- http response --- 06:42:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:42:32 DEBUG ----- output ----- 06:42:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:32 DEBUG { 06:42:32 DEBUG { 06:42:32 DEBUG "metric": { 06:42:32 DEBUG --- http response --- 06:42:32 DEBUG "status": "success", 06:42:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:42:32 DEBUG }, 06:42:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:42:32 DEBUG "data": { 06:42:32 DEBUG "value": [ 06:42:32 DEBUG 06:42:32 DEBUG ----- output ----- 06:42:32 DEBUG "resultType": "vector", 06:42:32 DEBUG 06:42:32 DEBUG 06:42:32 DEBUG { 06:42:32 DEBUG "result": [ 06:42:32 DEBUG { 06:42:32 DEBUG "metric": { 06:42:32 DEBUG "status": "success", 06:42:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:42:32 DEBUG }, 06:42:32 DEBUG "value": [ 06:42:32 DEBUG "data": { 06:42:32 DEBUG "resultType": "vector", 06:42:32 DEBUG "result": [ 06:42:32 DEBUG "resultType": "vector", 06:42:32 DEBUG [...] 06:42:32 DEBUG 06:42:32 DEBUG { 06:42:32 DEBUG "result": [ 06:42:32 DEBUG 06:42:32 DEBUG 06:42:32 DEBUG "metric": { 06:42:32 DEBUG { 06:42:32 DEBUG "metric": { 06:42:32 DEBUG 06:42:32 DEBUG "pod": "ds-idrepo-2" 06:42:32 DEBUG 06:42:32 DEBUG 06:42:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:42:32 DEBUG }, 06:42:32 DEBUG [...] 06:42:32 DEBUG 06:42:32 DEBUG 06:42:32 DEBUG "value": [ 06:42:32 DEBUG }, 06:42:32 DEBUG "value": [ 06:42:32 DEBUG 1712727752, 06:42:32 DEBUG 06:42:32 DEBUG "0.051" 06:42:32 DEBUG ] 06:42:32 DEBUG } 06:42:32 DEBUG 06:42:32 DEBUG ] 06:42:32 DEBUG 06:42:32 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:42:32 DEBUG 06:42:32 DEBUG } 06:42:32 DEBUG } 06:42:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:42:32 DEBUG [...] 06:42:32 DEBUG }, 06:42:32 DEBUG "value": [ 06:42:32 DEBUG 06:42:32 DEBUG 06:42:32 DEBUG }, 06:42:32 DEBUG 06:42:32 DEBUG 06:42:32 DEBUG "value": [ 06:42:32 DEBUG 06:42:32 DEBUG "pod": "ds-cts-2" 06:42:32 DEBUG 1712727752, 06:42:32 DEBUG }, 06:42:32 DEBUG 06:42:32 DEBUG [...] 06:42:32 DEBUG "value": [ 06:42:32 DEBUG "1403.779496067191" 06:42:32 DEBUG 06:42:32 DEBUG 1712727752, 06:42:32 DEBUG ] 06:42:32 DEBUG 06:42:32 DEBUG "0.00006666000066660001" 06:42:32 DEBUG } 06:42:32 DEBUG ] 06:42:32 DEBUG ] 06:42:32 DEBUG 06:42:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:42:32 DEBUG } 06:42:32 DEBUG }, 06:42:32 DEBUG "value": [ 06:42:32 DEBUG ] 06:42:32 DEBUG } 06:42:32 DEBUG 1712727752, 06:42:32 DEBUG } 06:42:32 DEBUG } 06:42:32 DEBUG "6.074610999171136" 06:42:32 DEBUG } 06:42:32 DEBUG ] 06:42:32 DEBUG } 06:42:32 DEBUG ] 06:42:32 DEBUG } 06:42:32 DEBUG } 06:42:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727752" 06:42:32 INFO [http_cmd]: http status code OK 06:42:32 DEBUG --- status code --- 06:42:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:32 DEBUG --- http response --- 06:42:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:42:32 DEBUG ----- output ----- 06:42:32 DEBUG { 06:42:32 DEBUG "status": "success", 06:42:32 DEBUG "data": { 06:42:32 DEBUG "resultType": "vector", 06:42:32 DEBUG "result": [ 06:42:32 DEBUG { 06:42:32 DEBUG "metric": { 06:42:32 DEBUG "pod": "ds-idrepo-0" 06:42:32 DEBUG }, 06:42:32 DEBUG "value": [ 06:42:32 DEBUG 06:42:32 DEBUG 06:42:32 DEBUG 06:42:32 DEBUG [...] 06:42:32 DEBUG 06:42:32 DEBUG 06:42:32 DEBUG 06:42:32 DEBUG "pod": "ds-idrepo-1" 06:42:32 DEBUG }, 06:42:32 DEBUG "value": [ 06:42:32 DEBUG 1712727752, 06:42:32 DEBUG "0" 06:42:32 DEBUG ] 06:42:32 DEBUG } 06:42:32 DEBUG ] 06:42:32 DEBUG } 06:42:32 DEBUG } 06:42:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727752" 06:42:32 INFO [http_cmd]: http status code OK 06:42:32 DEBUG --- status code --- 06:42:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:32 DEBUG --- http response --- 06:42:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:42:32 DEBUG ----- output ----- 06:42:32 DEBUG { 06:42:32 DEBUG "status": "success", 06:42:32 DEBUG "data": { 06:42:32 DEBUG "resultType": "vector", 06:42:32 DEBUG "result": [ 06:42:32 DEBUG { 06:42:32 DEBUG "metric": { 06:42:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:42:32 DEBUG }, 06:42:32 DEBUG "value": [ 06:42:32 DEBUG 06:42:32 DEBUG 06:42:32 DEBUG 06:42:32 DEBUG [...] 06:42:32 DEBUG 06:42:32 DEBUG 06:42:32 DEBUG 06:42:32 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:42:32 DEBUG }, 06:42:32 DEBUG "value": [ 06:42:32 DEBUG 1712727752, 06:42:32 DEBUG "1384.882015731236" 06:42:32 DEBUG ] 06:42:32 DEBUG } 06:42:32 DEBUG ] 06:42:32 DEBUG } 06:42:32 DEBUG } 06:42:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727752" 06:42:32 INFO [http_cmd]: http status code OK 06:42:32 DEBUG --- status code --- 06:42:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:32 DEBUG --- http response --- 06:42:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:42:32 DEBUG ----- output ----- 06:42:32 DEBUG { 06:42:32 DEBUG "status": "success", 06:42:32 DEBUG "data": { 06:42:32 DEBUG "resultType": "vector", 06:42:32 DEBUG "result": [ 06:42:32 DEBUG { 06:42:32 DEBUG "metric": { 06:42:32 DEBUG "pod": "ds-idrepo-0" 06:42:32 DEBUG }, 06:42:32 DEBUG "value": [ 06:42:32 DEBUG 06:42:32 DEBUG 06:42:32 DEBUG 06:42:32 DEBUG [...] 06:42:32 DEBUG 06:42:32 DEBUG 06:42:32 DEBUG 06:42:32 DEBUG "pod": "ds-idrepo-1" 06:42:32 DEBUG }, 06:42:32 DEBUG "value": [ 06:42:32 DEBUG 1712727752, 06:42:32 DEBUG "0" 06:42:32 DEBUG ] 06:42:32 DEBUG } 06:42:32 DEBUG ] 06:42:32 DEBUG } 06:42:32 DEBUG } 06:42:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727753" 06:42:33 INFO [http_cmd]: http status code OK 06:42:33 DEBUG --- status code --- 06:42:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:33 DEBUG --- http response --- 06:42:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:42:33 DEBUG ----- output ----- 06:42:33 DEBUG { 06:42:33 DEBUG "status": "success", 06:42:33 DEBUG "data": { 06:42:33 DEBUG "resultType": "vector", 06:42:33 DEBUG "result": [ 06:42:33 DEBUG { 06:42:33 DEBUG "metric": { 06:42:33 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:42:33 DEBUG }, 06:42:33 DEBUG "value": [ 06:42:33 DEBUG 06:42:33 DEBUG 06:42:33 DEBUG 06:42:33 DEBUG [...] 06:42:33 DEBUG 06:42:33 DEBUG 06:42:33 DEBUG 06:42:33 DEBUG "pod": "ds-idrepo-0" 06:42:33 DEBUG }, 06:42:33 DEBUG "value": [ 06:42:33 DEBUG 1712727753, 06:42:33 DEBUG "3874.2335290280494" 06:42:33 DEBUG ] 06:42:33 DEBUG } 06:42:33 DEBUG ] 06:42:33 DEBUG } 06:42:33 DEBUG } 06:42:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712727753" 06:42:33 INFO [http_cmd]: http status code OK 06:42:33 DEBUG --- status code --- 06:42:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:33 DEBUG --- http response --- 06:42:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:42:33 DEBUG ----- output ----- 06:42:33 DEBUG { 06:42:33 DEBUG "status": "success", 06:42:33 DEBUG "data": { 06:42:33 DEBUG "resultType": "vector", 06:42:33 DEBUG "result": [ 06:42:33 DEBUG { 06:42:33 DEBUG "metric": { 06:42:33 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:42:33 DEBUG }, 06:42:33 DEBUG "value": [ 06:42:33 DEBUG 06:42:33 DEBUG 06:42:33 DEBUG 06:42:33 DEBUG [...] 06:42:33 DEBUG 06:42:33 DEBUG 06:42:33 DEBUG 06:42:33 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:42:33 DEBUG }, 06:42:33 DEBUG "value": [ 06:42:33 DEBUG 1712727753, 06:42:33 DEBUG "2486591488" 06:42:33 DEBUG ] 06:42:33 DEBUG } 06:42:33 DEBUG ] 06:42:33 DEBUG } 06:42:33 DEBUG } 06:42:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727753" 06:42:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727753" 06:42:33 INFO [http_cmd]: http status code OK 06:42:33 DEBUG --- status code --- 06:42:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:33 DEBUG --- http response --- 06:42:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:42:33 DEBUG ----- output ----- 06:42:33 DEBUG { 06:42:33 DEBUG "status": "success", 06:42:33 DEBUG "data": { 06:42:33 DEBUG "resultType": "vector", 06:42:33 DEBUG "result": [ 06:42:33 DEBUG { 06:42:33 DEBUG "metric": { 06:42:33 DEBUG "instance": "10.142.0.25:9100" 06:42:33 DEBUG }, 06:42:33 DEBUG "value": [ 06:42:33 DEBUG 06:42:33 DEBUG 06:42:33 DEBUG 06:42:33 DEBUG [...] 06:42:33 DEBUG 06:42:33 DEBUG 06:42:33 DEBUG 06:42:33 DEBUG "instance": "10.142.0.77:9100" 06:42:33 DEBUG }, 06:42:33 DEBUG "value": [ 06:42:33 DEBUG 1712727753, 06:42:33 DEBUG "42622285.83333333" 06:42:33 DEBUG ] 06:42:33 DEBUG } 06:42:33 DEBUG ] 06:42:33 DEBUG } 06:42:33 DEBUG } 06:42:33 INFO [http_cmd]: http status code OK 06:42:33 DEBUG --- status code --- 06:42:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:33 DEBUG --- http response --- 06:42:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:42:33 DEBUG ----- output ----- 06:42:33 DEBUG { 06:42:33 DEBUG "status": "success", 06:42:33 DEBUG "data": { 06:42:33 DEBUG "resultType": "vector", 06:42:33 DEBUG "result": [ 06:42:33 DEBUG { 06:42:33 DEBUG "metric": { 06:42:33 DEBUG "instance": "10.142.0.25:9100" 06:42:33 DEBUG }, 06:42:33 DEBUG "value": [ 06:42:33 DEBUG 06:42:33 DEBUG 06:42:33 DEBUG 06:42:33 DEBUG [...] 06:42:33 DEBUG 06:42:33 DEBUG 06:42:33 DEBUG 06:42:33 DEBUG "instance": "10.142.0.77:9100" 06:42:33 DEBUG }, 06:42:33 DEBUG "value": [ 06:42:33 DEBUG 1712727753, 06:42:33 DEBUG "0.015666666666667217" 06:42:33 DEBUG ] 06:42:33 DEBUG } 06:42:33 DEBUG ] 06:42:33 DEBUG } 06:42:33 DEBUG } 06:42:53 INFO 06:42:53 INFO [loop_until]: kubectl --namespace=xlou top pods 06:42:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:42:53 INFO [loop_until]: OK (rc = 0) 06:42:53 DEBUG --- stdout --- 06:42:53 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 11104m 9037Mi am-654c4fcbcf-9scrj 9676m 8458Mi am-654c4fcbcf-v5qb5 8441m 8489Mi ds-cts-0 8347m 12495Mi ds-cts-1 8430m 13241Mi ds-cts-2 7699m 13346Mi ds-idrepo-0 10810m 5634Mi ds-idrepo-1 356m 5672Mi ds-idrepo-2 387m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1191Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 9m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4399m 2721Mi 06:42:53 DEBUG --- stderr --- 06:42:53 DEBUG 06:43:02 INFO 06:43:02 INFO [loop_until]: kubectl --namespace=xlou top node 06:43:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:43:02 INFO [loop_until]: OK (rc = 0) 06:43:02 DEBUG --- stdout --- 06:43:02 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 96m 0% 2305Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9483m 59% 9362Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 78m 0% 3260Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13901m 87% 6908Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10710m 67% 9338Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11931m 75% 9842Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 76m 0% 854Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10949m 68% 6317Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8567m 53% 13403Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 7881m 49% 14092Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 448m 2% 6395Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8468m 53% 13780Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 410m 2% 6362Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4529m 28% 6303Mi 10% 06:43:02 DEBUG --- stderr --- 06:43:02 DEBUG 06:43:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712727810" 06:43:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727810" 06:43:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727810" 06:43:30 ERROR [http_cmd]: ERROR 06:43:30 ERROR --- status code --- 06:43:30 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:43:30 ERROR --- http response --- 06:43:30 INFO [http_cmd]: http status code OK 06:43:30 DEBUG --- status code --- 06:43:30 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:43:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:30 INFO [http_cmd]: http status code OK 06:43:30 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:43:30 DEBUG --- http response --- 06:43:30 DEBUG --- status code --- 06:43:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:43:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:30 DEBUG ----- output ----- 06:43:30 DEBUG --- http response --- 06:43:30 DEBUG { 06:43:30 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:43:30 DEBUG "status": "success", 06:43:30 DEBUG "data": { 06:43:30 DEBUG "resultType": "vector", 06:43:30 DEBUG "result": [ 06:43:30 DEBUG { 06:43:30 DEBUG "metric": { 06:43:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:43:30 DEBUG }, 06:43:30 DEBUG "value": [ 06:43:30 DEBUG 06:43:30 DEBUG 06:43:30 DEBUG 06:43:30 DEBUG [...] 06:43:30 DEBUG 06:43:30 DEBUG 06:43:30 DEBUG 06:43:30 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:43:30 DEBUG }, 06:43:30 DEBUG "value": [ 06:43:30 DEBUG 1712727810, 06:43:30 DEBUG "0" 06:43:30 DEBUG ] 06:43:30 DEBUG } 06:43:30 DEBUG ] 06:43:30 DEBUG } 06:43:30 DEBUG } 06:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727811" 06:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727811" 06:43:31 INFO [http_cmd]: http status code OK 06:43:31 DEBUG --- status code --- 06:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:31 DEBUG --- http response --- 06:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:43:31 DEBUG ----- output ----- 06:43:31 DEBUG { 06:43:31 DEBUG "status": "success", 06:43:31 DEBUG "data": { 06:43:31 DEBUG "resultType": "vector", 06:43:31 DEBUG "result": [ 06:43:31 DEBUG { 06:43:31 DEBUG "metric": { 06:43:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:43:31 DEBUG }, 06:43:31 DEBUG "value": [ 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG [...] 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:43:31 DEBUG }, 06:43:31 DEBUG "value": [ 06:43:31 DEBUG 1712727811, 06:43:31 DEBUG "1404.5872783037737" 06:43:31 DEBUG ] 06:43:31 DEBUG } 06:43:31 DEBUG ] 06:43:31 DEBUG } 06:43:31 DEBUG } 06:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727811" 06:43:31 INFO [http_cmd]: http status code OK 06:43:31 DEBUG --- status code --- 06:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:31 DEBUG --- http response --- 06:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727811" 06:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:43:31 DEBUG ----- output ----- 06:43:31 DEBUG { 06:43:31 DEBUG "status": "success", 06:43:31 DEBUG "data": { 06:43:31 DEBUG "resultType": "vector", 06:43:31 DEBUG "result": [ 06:43:31 DEBUG { 06:43:31 DEBUG "metric": { 06:43:31 DEBUG "instance": "10.142.0.25:9100" 06:43:31 DEBUG }, 06:43:31 DEBUG "value": [ 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG [...] 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG "instance": "10.142.0.77:9100" 06:43:31 DEBUG }, 06:43:31 DEBUG "value": [ 06:43:31 DEBUG 1712727811, 06:43:31 DEBUG "42544182.63333333" 06:43:31 DEBUG ] 06:43:31 DEBUG } 06:43:31 DEBUG ] 06:43:31 DEBUG } 06:43:31 DEBUG } 06:43:31 INFO [http_cmd]: http status code OK 06:43:31 DEBUG --- status code --- 06:43:31 INFO [http_cmd]: http status code OK 06:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:31 DEBUG --- status code --- 06:43:31 DEBUG --- http response --- 06:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:43:31 DEBUG --- http response --- 06:43:31 DEBUG ----- output ----- 06:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:43:31 DEBUG { 06:43:31 DEBUG ----- output ----- 06:43:31 DEBUG "status": "success", 06:43:31 DEBUG { 06:43:31 DEBUG "data": { 06:43:31 DEBUG "status": "success", 06:43:31 DEBUG "resultType": "vector", 06:43:31 DEBUG "data": { 06:43:31 DEBUG "result": [ 06:43:31 DEBUG "resultType": "vector", 06:43:31 DEBUG { 06:43:31 DEBUG "result": [ 06:43:31 DEBUG "metric": { 06:43:31 DEBUG { 06:43:31 DEBUG "pod": "ds-idrepo-0" 06:43:31 DEBUG "metric": { 06:43:31 DEBUG }, 06:43:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:43:31 DEBUG "value": [ 06:43:31 DEBUG }, 06:43:31 DEBUG 06:43:31 DEBUG "value": [ 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG [...] 06:43:31 DEBUG [...] 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:43:31 DEBUG "pod": "ds-idrepo-1" 06:43:31 DEBUG }, 06:43:31 DEBUG }, 06:43:31 DEBUG "value": [ 06:43:31 DEBUG "value": [ 06:43:31 DEBUG 1712727811, 06:43:31 DEBUG "1402.220296039472" 06:43:31 DEBUG 1712727811, 06:43:31 DEBUG ] 06:43:31 DEBUG "0" 06:43:31 DEBUG ] 06:43:31 DEBUG } 06:43:31 DEBUG } 06:43:31 DEBUG ] 06:43:31 DEBUG ] 06:43:31 DEBUG } 06:43:31 DEBUG } 06:43:31 DEBUG } 06:43:31 DEBUG } 06:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727811" 06:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727811" 06:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727811" 06:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727811" 06:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727811" 06:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727811" 06:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727811" 06:43:31 INFO [http_cmd]: http status code OK 06:43:31 DEBUG --- status code --- 06:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:31 DEBUG --- http response --- 06:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:43:31 DEBUG ----- output ----- 06:43:31 DEBUG { 06:43:31 DEBUG "status": "success", 06:43:31 DEBUG "data": { 06:43:31 DEBUG "resultType": "vector", 06:43:31 DEBUG "result": [ 06:43:31 DEBUG { 06:43:31 DEBUG "metric": { 06:43:31 DEBUG "instance": "10.142.0.25:9100" 06:43:31 DEBUG }, 06:43:31 DEBUG "value": [ 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG [...] 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG "instance": "10.142.0.77:9100" 06:43:31 DEBUG }, 06:43:31 DEBUG "value": [ 06:43:31 DEBUG 1712727811, 06:43:31 DEBUG "429591211008" 06:43:31 DEBUG ] 06:43:31 DEBUG } 06:43:31 DEBUG ] 06:43:31 DEBUG } 06:43:31 DEBUG } 06:43:31 INFO [http_cmd]: http status code OK 06:43:31 DEBUG --- status code --- 06:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:31 DEBUG --- http response --- 06:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:43:31 DEBUG ----- output ----- 06:43:31 DEBUG { 06:43:31 DEBUG "status": "success", 06:43:31 DEBUG "data": { 06:43:31 INFO [http_cmd]: http status code OK 06:43:31 DEBUG "resultType": "vector", 06:43:31 DEBUG --- status code --- 06:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:31 DEBUG --- http response --- 06:43:31 DEBUG "result": [ 06:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:43:31 INFO [http_cmd]: http status code OK 06:43:31 DEBUG --- status code --- 06:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:31 DEBUG ----- output ----- 06:43:31 INFO [http_cmd]: http status code OK 06:43:31 DEBUG --- status code --- 06:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:31 DEBUG --- http response --- 06:43:31 INFO [http_cmd]: http status code OK 06:43:31 DEBUG { 06:43:31 DEBUG --- http response --- 06:43:31 INFO [http_cmd]: http status code OK 06:43:31 DEBUG --- status code --- 06:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:31 DEBUG --- http response --- 06:43:31 DEBUG "status": "success", 06:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:43:31 DEBUG ----- output ----- 06:43:31 DEBUG { 06:43:31 DEBUG "status": "success", 06:43:31 DEBUG "data": { 06:43:31 DEBUG "data": { 06:43:31 DEBUG --- status code --- 06:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:43:31 DEBUG ----- output ----- 06:43:31 DEBUG "resultType": "vector", 06:43:31 DEBUG "resultType": "vector", 06:43:31 DEBUG "result": [ 06:43:31 DEBUG { 06:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:43:31 DEBUG { 06:43:31 DEBUG "result": [ 06:43:31 DEBUG --- http response --- 06:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:43:31 DEBUG ----- output ----- 06:43:31 DEBUG { 06:43:31 DEBUG "status": "success", 06:43:31 DEBUG { 06:43:31 DEBUG "data": { 06:43:31 DEBUG ----- output ----- 06:43:31 DEBUG { 06:43:31 DEBUG "status": "success", 06:43:31 DEBUG "data": { 06:43:31 DEBUG "metric": { 06:43:31 DEBUG "resultType": "vector", 06:43:31 DEBUG "metric": { 06:43:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:43:31 DEBUG }, 06:43:31 DEBUG "value": [ 06:43:31 DEBUG "pod": "ds-idrepo-0" 06:43:31 DEBUG "result": [ 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG { 06:43:31 DEBUG }, 06:43:31 DEBUG "value": [ 06:43:31 DEBUG 06:43:31 DEBUG [...] 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG { 06:43:31 DEBUG "metric": { 06:43:31 DEBUG "pod": "ds-idrepo-0" 06:43:31 DEBUG }, 06:43:31 DEBUG "status": "success", 06:43:31 DEBUG "data": { 06:43:31 DEBUG "resultType": "vector", 06:43:31 DEBUG "result": [ 06:43:31 DEBUG { 06:43:31 DEBUG "metric": { 06:43:31 DEBUG "instance": "10.142.0.25:9100" 06:43:31 DEBUG }, 06:43:31 DEBUG "value": [ 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG [...] 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG "resultType": "vector", 06:43:31 DEBUG "result": [ 06:43:31 DEBUG { 06:43:31 DEBUG "metric": { 06:43:31 DEBUG "instance": "10.142.0.77:9100" 06:43:31 DEBUG }, 06:43:31 DEBUG "value": [ 06:43:31 DEBUG 1712727811, 06:43:31 DEBUG "metric": { 06:43:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:43:31 DEBUG }, 06:43:31 DEBUG "value": [ 06:43:31 DEBUG "0.15596666666666023" 06:43:31 DEBUG ] 06:43:31 DEBUG } 06:43:31 DEBUG ] 06:43:31 DEBUG } 06:43:31 DEBUG } 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG [...] 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:43:31 DEBUG }, 06:43:31 DEBUG "value": [ 06:43:31 DEBUG 1712727811, 06:43:31 DEBUG "0" 06:43:31 DEBUG ] 06:43:31 DEBUG } 06:43:31 DEBUG ] 06:43:31 DEBUG } 06:43:31 DEBUG } 06:43:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:43:31 DEBUG }, 06:43:31 DEBUG "value": [ 06:43:31 DEBUG 06:43:31 DEBUG "value": [ 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG [...] 06:43:31 DEBUG 06:43:31 DEBUG }, 06:43:31 DEBUG 06:43:31 DEBUG [...] 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG "pod": "ds-cts-2" 06:43:31 DEBUG }, 06:43:31 DEBUG "value": [ 06:43:31 DEBUG 1712727811, 06:43:31 DEBUG "29.9" 06:43:31 DEBUG ] 06:43:31 DEBUG "value": [ 06:43:31 DEBUG } 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG 1712727811, 06:43:31 DEBUG ] 06:43:31 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:43:31 DEBUG }, 06:43:31 DEBUG "value": [ 06:43:31 DEBUG 1712727811, 06:43:31 DEBUG "256870" 06:43:31 DEBUG "0" 06:43:31 DEBUG ] 06:43:31 DEBUG } 06:43:31 DEBUG ] 06:43:31 DEBUG } 06:43:31 DEBUG } 06:43:31 DEBUG ] 06:43:31 DEBUG } 06:43:31 DEBUG 06:43:31 DEBUG } 06:43:31 DEBUG } 06:43:31 DEBUG [...] 06:43:31 DEBUG ] 06:43:31 DEBUG 06:43:31 DEBUG } 06:43:31 DEBUG 06:43:31 DEBUG } 06:43:31 DEBUG 06:43:31 DEBUG "pod": "ds-cts-2" 06:43:31 DEBUG }, 06:43:31 DEBUG "value": [ 06:43:31 DEBUG 1712727811, 06:43:31 DEBUG "0" 06:43:31 DEBUG ] 06:43:31 DEBUG } 06:43:31 DEBUG ] 06:43:31 DEBUG } 06:43:31 DEBUG } 06:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712727811" 06:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712727811" 06:43:31 INFO [http_cmd]: http status code OK 06:43:31 DEBUG --- status code --- 06:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:31 DEBUG --- http response --- 06:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:43:31 DEBUG ----- output ----- 06:43:31 DEBUG { 06:43:31 INFO [http_cmd]: http status code OK 06:43:31 DEBUG "status": "success", 06:43:31 DEBUG --- status code --- 06:43:31 DEBUG "data": { 06:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:31 DEBUG "resultType": "vector", 06:43:31 DEBUG --- http response --- 06:43:31 DEBUG "result": [ 06:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:43:31 DEBUG ----- output ----- 06:43:31 DEBUG { 06:43:31 DEBUG { 06:43:31 DEBUG "status": "success", 06:43:31 DEBUG "data": { 06:43:31 DEBUG "resultType": "vector", 06:43:31 DEBUG "result": [ 06:43:31 DEBUG { 06:43:31 DEBUG "metric": { 06:43:31 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:43:31 DEBUG }, 06:43:31 DEBUG "value": [ 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG [...] 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:43:31 DEBUG }, 06:43:31 DEBUG "value": [ 06:43:31 DEBUG 1712727811, 06:43:31 DEBUG "4.4083688012295354" 06:43:31 DEBUG ] 06:43:31 DEBUG "metric": { 06:43:31 DEBUG } 06:43:31 DEBUG "pod": "ds-idrepo-0" 06:43:31 DEBUG ] 06:43:31 DEBUG }, 06:43:31 DEBUG } 06:43:31 DEBUG "value": [ 06:43:31 DEBUG } 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG [...] 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG 06:43:31 DEBUG "pod": "ds-cts-2" 06:43:31 DEBUG }, 06:43:31 DEBUG "value": [ 06:43:31 DEBUG 1712727811, 06:43:31 DEBUG "0" 06:43:31 DEBUG ] 06:43:31 DEBUG } 06:43:31 DEBUG ] 06:43:31 DEBUG } 06:43:31 DEBUG } 06:43:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712727812" 06:43:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727812" 06:43:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727812" 06:43:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727812" 06:43:32 INFO [http_cmd]: http status code OK 06:43:32 DEBUG --- status code --- 06:43:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:32 DEBUG --- http response --- 06:43:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:43:32 DEBUG ----- output ----- 06:43:32 DEBUG { 06:43:32 DEBUG "status": "success", 06:43:32 DEBUG "data": { 06:43:32 DEBUG "resultType": "vector", 06:43:32 DEBUG "result": [ 06:43:32 DEBUG { 06:43:32 DEBUG "metric": { 06:43:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:43:32 DEBUG }, 06:43:32 DEBUG "value": [ 06:43:32 DEBUG 06:43:32 DEBUG 06:43:32 DEBUG 06:43:32 DEBUG [...] 06:43:32 DEBUG 06:43:32 DEBUG 06:43:32 DEBUG 06:43:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:43:32 DEBUG }, 06:43:32 DEBUG "value": [ 06:43:32 DEBUG 1712727812, 06:43:32 DEBUG "0.051" 06:43:32 DEBUG ] 06:43:32 DEBUG } 06:43:32 DEBUG ] 06:43:32 DEBUG } 06:43:32 DEBUG } 06:43:32 INFO [http_cmd]: http status code OK 06:43:32 DEBUG --- status code --- 06:43:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:32 DEBUG --- http response --- 06:43:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:43:32 DEBUG ----- output ----- 06:43:32 DEBUG { 06:43:32 DEBUG "status": "success", 06:43:32 DEBUG "data": { 06:43:32 DEBUG "resultType": "vector", 06:43:32 DEBUG "result": [ 06:43:32 INFO [http_cmd]: http status code OK 06:43:32 DEBUG { 06:43:32 DEBUG "metric": { 06:43:32 DEBUG --- status code --- 06:43:32 DEBUG "pod": "ds-idrepo-2" 06:43:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:32 DEBUG }, 06:43:32 DEBUG --- http response --- 06:43:32 DEBUG "value": [ 06:43:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:43:32 DEBUG 06:43:32 DEBUG ----- output ----- 06:43:32 DEBUG 06:43:32 DEBUG { 06:43:32 DEBUG 06:43:32 DEBUG "status": "success", 06:43:32 DEBUG [...] 06:43:32 DEBUG 06:43:32 DEBUG 06:43:32 DEBUG 06:43:32 DEBUG "pod": "ds-cts-2" 06:43:32 DEBUG }, 06:43:32 DEBUG "value": [ 06:43:32 DEBUG 1712727812, 06:43:32 DEBUG "0.000033333333333333335" 06:43:32 DEBUG ] 06:43:32 DEBUG "data": { 06:43:32 DEBUG } 06:43:32 DEBUG ] 06:43:32 DEBUG } 06:43:32 DEBUG "resultType": "vector", 06:43:32 DEBUG } 06:43:32 DEBUG "result": [ 06:43:32 DEBUG { 06:43:32 DEBUG "metric": { 06:43:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:43:32 DEBUG }, 06:43:32 DEBUG "value": [ 06:43:32 DEBUG 06:43:32 DEBUG 06:43:32 DEBUG 06:43:32 DEBUG [...] 06:43:32 DEBUG 06:43:32 DEBUG 06:43:32 DEBUG 06:43:32 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:43:32 INFO [http_cmd]: http status code OK 06:43:32 DEBUG }, 06:43:32 DEBUG --- status code --- 06:43:32 DEBUG "value": [ 06:43:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:32 DEBUG 1712727812, 06:43:32 DEBUG --- http response --- 06:43:32 DEBUG "1404.2872382984397" 06:43:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:43:32 DEBUG ----- output ----- 06:43:32 DEBUG ] 06:43:32 DEBUG { 06:43:32 DEBUG } 06:43:32 DEBUG "status": "success", 06:43:32 DEBUG ] 06:43:32 DEBUG "data": { 06:43:32 DEBUG } 06:43:32 DEBUG "resultType": "vector", 06:43:32 DEBUG } 06:43:32 DEBUG "result": [ 06:43:32 DEBUG { 06:43:32 DEBUG "metric": { 06:43:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:43:32 DEBUG }, 06:43:32 DEBUG "value": [ 06:43:32 DEBUG 06:43:32 DEBUG 06:43:32 DEBUG 06:43:32 DEBUG [...] 06:43:32 DEBUG 06:43:32 DEBUG 06:43:32 DEBUG 06:43:32 DEBUG "pod": "lodemon-b776b7d88-thrqf" 06:43:32 DEBUG }, 06:43:32 DEBUG "value": [ 06:43:32 DEBUG 1712727812, 06:43:32 DEBUG "0.008243663007412593" 06:43:32 DEBUG ] 06:43:32 DEBUG } 06:43:32 DEBUG ] 06:43:32 DEBUG } 06:43:32 DEBUG } 06:43:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727812" 06:43:32 INFO [http_cmd]: http status code OK 06:43:32 DEBUG --- status code --- 06:43:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:32 DEBUG --- http response --- 06:43:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:43:32 DEBUG ----- output ----- 06:43:32 DEBUG { 06:43:32 DEBUG "status": "success", 06:43:32 DEBUG "data": { 06:43:32 DEBUG "resultType": "vector", 06:43:32 DEBUG "result": [ 06:43:32 DEBUG { 06:43:32 DEBUG "metric": { 06:43:32 DEBUG "pod": "ds-idrepo-0" 06:43:32 DEBUG }, 06:43:32 DEBUG "value": [ 06:43:32 DEBUG 06:43:32 DEBUG 06:43:32 DEBUG 06:43:32 DEBUG [...] 06:43:32 DEBUG 06:43:32 DEBUG 06:43:32 DEBUG 06:43:32 DEBUG "pod": "ds-idrepo-1" 06:43:32 DEBUG }, 06:43:32 DEBUG "value": [ 06:43:32 DEBUG 1712727812, 06:43:32 DEBUG "0" 06:43:32 DEBUG ] 06:43:32 DEBUG } 06:43:32 DEBUG ] 06:43:32 DEBUG } 06:43:32 DEBUG } 06:43:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727812" 06:43:32 INFO [http_cmd]: http status code OK 06:43:32 DEBUG --- status code --- 06:43:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:32 DEBUG --- http response --- 06:43:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:43:32 DEBUG ----- output ----- 06:43:32 DEBUG { 06:43:32 DEBUG "status": "success", 06:43:32 DEBUG "data": { 06:43:32 DEBUG "resultType": "vector", 06:43:32 DEBUG "result": [ 06:43:32 DEBUG { 06:43:32 DEBUG "metric": { 06:43:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:43:32 DEBUG }, 06:43:32 DEBUG "value": [ 06:43:32 DEBUG 06:43:32 DEBUG 06:43:32 DEBUG 06:43:32 DEBUG [...] 06:43:32 DEBUG 06:43:32 DEBUG 06:43:32 DEBUG 06:43:32 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:43:32 DEBUG }, 06:43:32 DEBUG "value": [ 06:43:32 DEBUG 1712727812, 06:43:32 DEBUG "1400.086678223763" 06:43:32 DEBUG ] 06:43:32 DEBUG } 06:43:32 DEBUG ] 06:43:32 DEBUG } 06:43:32 DEBUG } 06:43:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727812" 06:43:32 INFO [http_cmd]: http status code OK 06:43:32 DEBUG --- status code --- 06:43:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:32 DEBUG --- http response --- 06:43:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:43:32 DEBUG ----- output ----- 06:43:32 DEBUG { 06:43:32 DEBUG "status": "success", 06:43:32 DEBUG "data": { 06:43:32 DEBUG "resultType": "vector", 06:43:32 DEBUG "result": [ 06:43:32 DEBUG { 06:43:32 DEBUG "metric": { 06:43:32 DEBUG "pod": "ds-idrepo-0" 06:43:32 DEBUG }, 06:43:32 DEBUG "value": [ 06:43:32 DEBUG 06:43:32 DEBUG 06:43:32 DEBUG 06:43:32 DEBUG [...] 06:43:32 DEBUG 06:43:32 DEBUG 06:43:32 DEBUG 06:43:32 DEBUG "pod": "ds-idrepo-1" 06:43:32 DEBUG }, 06:43:32 DEBUG "value": [ 06:43:32 DEBUG 1712727812, 06:43:32 DEBUG "0" 06:43:32 DEBUG ] 06:43:32 DEBUG } 06:43:32 DEBUG ] 06:43:32 DEBUG } 06:43:32 DEBUG } 06:43:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727813" 06:43:33 INFO [http_cmd]: http status code OK 06:43:33 DEBUG --- status code --- 06:43:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:33 DEBUG --- http response --- 06:43:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:43:33 DEBUG ----- output ----- 06:43:33 DEBUG { 06:43:33 DEBUG "status": "success", 06:43:33 DEBUG "data": { 06:43:33 DEBUG "resultType": "vector", 06:43:33 DEBUG "result": [ 06:43:33 DEBUG { 06:43:33 DEBUG "metric": { 06:43:33 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:43:33 DEBUG }, 06:43:33 DEBUG "value": [ 06:43:33 DEBUG 06:43:33 DEBUG 06:43:33 DEBUG 06:43:33 DEBUG [...] 06:43:33 DEBUG 06:43:33 DEBUG 06:43:33 DEBUG 06:43:33 DEBUG "pod": "login-ui-5f964fb894-5zpvq" 06:43:33 DEBUG }, 06:43:33 DEBUG "value": [ 06:43:33 DEBUG 1712727813, 06:43:33 DEBUG "0" 06:43:33 DEBUG ] 06:43:33 DEBUG } 06:43:33 DEBUG ] 06:43:33 DEBUG } 06:43:33 DEBUG } 06:43:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712727813" 06:43:33 INFO [http_cmd]: http status code OK 06:43:33 DEBUG --- status code --- 06:43:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:33 DEBUG --- http response --- 06:43:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:43:33 DEBUG ----- output ----- 06:43:33 DEBUG { 06:43:33 DEBUG "status": "success", 06:43:33 DEBUG "data": { 06:43:33 DEBUG "resultType": "vector", 06:43:33 DEBUG "result": [ 06:43:33 DEBUG { 06:43:33 DEBUG "metric": { 06:43:33 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:43:33 DEBUG }, 06:43:33 DEBUG "value": [ 06:43:33 DEBUG 06:43:33 DEBUG 06:43:33 DEBUG 06:43:33 DEBUG [...] 06:43:33 DEBUG 06:43:33 DEBUG 06:43:33 DEBUG 06:43:33 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:43:33 DEBUG }, 06:43:33 DEBUG "value": [ 06:43:33 DEBUG 1712727813, 06:43:33 DEBUG "2485592064" 06:43:33 DEBUG ] 06:43:33 DEBUG } 06:43:33 DEBUG ] 06:43:33 DEBUG } 06:43:33 DEBUG } 06:43:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727813" 06:43:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727813" 06:43:33 INFO [http_cmd]: http status code OK 06:43:33 DEBUG --- status code --- 06:43:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:33 DEBUG --- http response --- 06:43:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:43:33 DEBUG ----- output ----- 06:43:33 DEBUG { 06:43:33 DEBUG "status": "success", 06:43:33 DEBUG "data": { 06:43:33 DEBUG "resultType": "vector", 06:43:33 DEBUG "result": [ 06:43:33 DEBUG { 06:43:33 DEBUG "metric": { 06:43:33 DEBUG "instance": "10.142.0.25:9100" 06:43:33 DEBUG }, 06:43:33 DEBUG "value": [ 06:43:33 DEBUG 06:43:33 DEBUG 06:43:33 DEBUG 06:43:33 DEBUG [...] 06:43:33 DEBUG 06:43:33 DEBUG 06:43:33 DEBUG 06:43:33 DEBUG "instance": "10.142.0.77:9100" 06:43:33 DEBUG }, 06:43:33 DEBUG "value": [ 06:43:33 DEBUG 1712727813, 06:43:33 DEBUG "42546138.1" 06:43:33 DEBUG ] 06:43:33 DEBUG } 06:43:33 DEBUG ] 06:43:33 DEBUG } 06:43:33 DEBUG } 06:43:33 INFO [http_cmd]: http status code OK 06:43:33 DEBUG --- status code --- 06:43:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:33 DEBUG --- http response --- 06:43:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:43:33 DEBUG ----- output ----- 06:43:33 DEBUG { 06:43:33 DEBUG "status": "success", 06:43:33 DEBUG "data": { 06:43:33 DEBUG "resultType": "vector", 06:43:33 DEBUG "result": [ 06:43:33 DEBUG { 06:43:33 DEBUG "metric": { 06:43:33 DEBUG "instance": "10.142.0.25:9100" 06:43:33 DEBUG }, 06:43:33 DEBUG "value": [ 06:43:33 DEBUG 06:43:33 DEBUG 06:43:33 DEBUG 06:43:33 DEBUG [...] 06:43:33 DEBUG 06:43:33 DEBUG 06:43:33 DEBUG 06:43:33 DEBUG "instance": "10.142.0.77:9100" 06:43:33 DEBUG }, 06:43:33 DEBUG "value": [ 06:43:33 DEBUG 1712727813, 06:43:33 DEBUG "0.054333333333333886" 06:43:33 DEBUG ] 06:43:33 DEBUG } 06:43:33 DEBUG ] 06:43:33 DEBUG } 06:43:33 DEBUG } 06:43:53 INFO 06:43:53 INFO [loop_until]: kubectl --namespace=xlou top pods 06:43:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:43:53 INFO [loop_until]: OK (rc = 0) 06:43:53 DEBUG --- stdout --- 06:43:53 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 10886m 9027Mi am-654c4fcbcf-9scrj 10085m 8459Mi am-654c4fcbcf-v5qb5 8366m 8508Mi ds-cts-0 8613m 13217Mi ds-cts-1 8773m 12646Mi ds-cts-2 7569m 12761Mi ds-idrepo-0 10778m 5634Mi ds-idrepo-1 351m 5672Mi ds-idrepo-2 376m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1191Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 9m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4351m 2619Mi 06:43:53 DEBUG --- stderr --- 06:43:53 DEBUG 06:44:02 INFO 06:44:02 INFO [loop_until]: kubectl --namespace=xlou top node 06:44:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:44:02 INFO [loop_until]: OK (rc = 0) 06:44:02 DEBUG --- stdout --- 06:44:02 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 97m 0% 2308Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9044m 56% 9367Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 76m 0% 3258Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 14263m 89% 6900Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 10986m 69% 9335Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11558m 72% 9834Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 76m 0% 855Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10940m 68% 6321Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8407m 52% 14016Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 7383m 46% 13471Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 483m 3% 6392Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8628m 54% 13349Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 425m 2% 6363Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4522m 28% 6201Mi 10% 06:44:02 DEBUG --- stderr --- 06:44:02 DEBUG 06:44:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712727870" 06:44:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727870" 06:44:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727870" 06:44:30 ERROR [http_cmd]: ERROR 06:44:30 ERROR --- status code --- 06:44:30 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:44:30 ERROR --- http response --- 06:44:30 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:44:30 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:44:30 INFO [http_cmd]: http status code OK 06:44:30 DEBUG --- status code --- 06:44:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:30 DEBUG --- http response --- 06:44:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:44:30 DEBUG ----- output ----- 06:44:30 DEBUG { 06:44:30 DEBUG "status": "success", 06:44:30 DEBUG "data": { 06:44:30 DEBUG "resultType": "vector", 06:44:30 DEBUG "result": [ 06:44:30 DEBUG { 06:44:30 DEBUG "metric": { 06:44:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:44:30 DEBUG }, 06:44:30 DEBUG "value": [ 06:44:30 DEBUG 06:44:30 DEBUG 06:44:30 DEBUG 06:44:30 DEBUG [...] 06:44:30 DEBUG 06:44:30 DEBUG 06:44:30 DEBUG 06:44:30 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:44:30 DEBUG }, 06:44:30 DEBUG "value": [ 06:44:30 DEBUG 1712727870, 06:44:30 DEBUG "0" 06:44:30 DEBUG ] 06:44:30 DEBUG } 06:44:30 DEBUG ] 06:44:30 DEBUG } 06:44:30 DEBUG } 06:44:30 INFO [http_cmd]: http status code OK 06:44:30 DEBUG --- status code --- 06:44:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:30 DEBUG --- http response --- 06:44:30 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727871" 06:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727871" 06:44:31 INFO [http_cmd]: http status code OK 06:44:31 DEBUG --- status code --- 06:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:31 DEBUG --- http response --- 06:44:31 INFO [http_cmd]: http status code OK 06:44:31 DEBUG --- status code --- 06:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:31 DEBUG ----- output ----- 06:44:31 DEBUG --- http response --- 06:44:31 DEBUG { 06:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:44:31 DEBUG "status": "success", 06:44:31 DEBUG ----- output ----- 06:44:31 DEBUG "data": { 06:44:31 DEBUG { 06:44:31 DEBUG "resultType": "vector", 06:44:31 DEBUG "status": "success", 06:44:31 DEBUG "data": { 06:44:31 DEBUG "resultType": "vector", 06:44:31 DEBUG "result": [ 06:44:31 DEBUG "result": [ 06:44:31 DEBUG { 06:44:31 DEBUG { 06:44:31 DEBUG "metric": { 06:44:31 DEBUG "metric": { 06:44:31 DEBUG "instance": "10.142.0.25:9100" 06:44:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:44:31 DEBUG }, 06:44:31 DEBUG }, 06:44:31 DEBUG "value": [ 06:44:31 DEBUG "value": [ 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG [...] 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG "instance": "10.142.0.77:9100" 06:44:31 DEBUG }, 06:44:31 DEBUG "value": [ 06:44:31 DEBUG 1712727871, 06:44:31 DEBUG "42830379.06666666" 06:44:31 DEBUG ] 06:44:31 DEBUG } 06:44:31 DEBUG 06:44:31 DEBUG ] 06:44:31 DEBUG 06:44:31 DEBUG } 06:44:31 DEBUG } 06:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727871" 06:44:31 DEBUG [...] 06:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727871" 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:44:31 DEBUG }, 06:44:31 DEBUG "value": [ 06:44:31 DEBUG 1712727871, 06:44:31 DEBUG "1400.4666666666667" 06:44:31 DEBUG ] 06:44:31 DEBUG } 06:44:31 DEBUG ] 06:44:31 DEBUG } 06:44:31 DEBUG } 06:44:31 INFO [http_cmd]: http status code OK 06:44:31 DEBUG --- status code --- 06:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:31 DEBUG --- http response --- 06:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:44:31 DEBUG ----- output ----- 06:44:31 DEBUG { 06:44:31 DEBUG "status": "success", 06:44:31 DEBUG "data": { 06:44:31 DEBUG "resultType": "vector", 06:44:31 DEBUG "result": [ 06:44:31 DEBUG { 06:44:31 DEBUG "metric": { 06:44:31 DEBUG "pod": "ds-idrepo-0" 06:44:31 DEBUG }, 06:44:31 DEBUG "value": [ 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG [...] 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG "pod": "ds-idrepo-1" 06:44:31 DEBUG }, 06:44:31 DEBUG "value": [ 06:44:31 DEBUG 1712727871, 06:44:31 DEBUG "0" 06:44:31 DEBUG ] 06:44:31 DEBUG } 06:44:31 DEBUG ] 06:44:31 DEBUG } 06:44:31 DEBUG } 06:44:31 INFO [http_cmd]: http status code OK 06:44:31 DEBUG --- status code --- 06:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:31 DEBUG --- http response --- 06:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:44:31 DEBUG ----- output ----- 06:44:31 DEBUG { 06:44:31 DEBUG "status": "success", 06:44:31 DEBUG "data": { 06:44:31 DEBUG "resultType": "vector", 06:44:31 DEBUG "result": [ 06:44:31 DEBUG { 06:44:31 DEBUG "metric": { 06:44:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:44:31 DEBUG }, 06:44:31 DEBUG "value": [ 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG [...] 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:44:31 DEBUG }, 06:44:31 DEBUG "value": [ 06:44:31 DEBUG 1712727871, 06:44:31 DEBUG "1409.7" 06:44:31 DEBUG ] 06:44:31 DEBUG } 06:44:31 DEBUG ] 06:44:31 DEBUG } 06:44:31 DEBUG } 06:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727871" 06:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727871" 06:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727871" 06:44:31 INFO [http_cmd]: http status code OK 06:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727871" 06:44:31 DEBUG --- status code --- 06:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727871" 06:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727871" 06:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727871" 06:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:31 DEBUG --- http response --- 06:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:44:31 DEBUG ----- output ----- 06:44:31 DEBUG { 06:44:31 DEBUG "status": "success", 06:44:31 DEBUG "data": { 06:44:31 DEBUG "resultType": "vector", 06:44:31 DEBUG "result": [ 06:44:31 DEBUG { 06:44:31 DEBUG "metric": { 06:44:31 DEBUG "instance": "10.142.0.25:9100" 06:44:31 DEBUG }, 06:44:31 DEBUG "value": [ 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG [...] 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG "instance": "10.142.0.77:9100" 06:44:31 DEBUG }, 06:44:31 DEBUG "value": [ 06:44:31 DEBUG 1712727871, 06:44:31 DEBUG "431522604032" 06:44:31 DEBUG ] 06:44:31 DEBUG } 06:44:31 DEBUG ] 06:44:31 DEBUG } 06:44:31 DEBUG } 06:44:31 INFO [http_cmd]: http status code OK 06:44:31 DEBUG --- status code --- 06:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:31 DEBUG --- http response --- 06:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:44:31 INFO [http_cmd]: http status code OK 06:44:31 DEBUG --- status code --- 06:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:31 DEBUG ----- output ----- 06:44:31 INFO [http_cmd]: http status code OK 06:44:31 DEBUG --- status code --- 06:44:31 DEBUG { 06:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:31 DEBUG --- http response --- 06:44:31 DEBUG "status": "success", 06:44:31 DEBUG --- http response --- 06:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:44:31 DEBUG "data": { 06:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:44:31 DEBUG ----- output ----- 06:44:31 DEBUG "resultType": "vector", 06:44:31 DEBUG ----- output ----- 06:44:31 DEBUG { 06:44:31 DEBUG "result": [ 06:44:31 DEBUG { 06:44:31 DEBUG "status": "success", 06:44:31 DEBUG { 06:44:31 DEBUG "data": { 06:44:31 DEBUG "resultType": "vector", 06:44:31 DEBUG "metric": { 06:44:31 DEBUG "status": "success", 06:44:31 DEBUG "data": { 06:44:31 DEBUG "resultType": "vector", 06:44:31 DEBUG "result": [ 06:44:31 INFO [http_cmd]: http status code OK 06:44:31 INFO [http_cmd]: http status code OK 06:44:31 DEBUG --- status code --- 06:44:31 DEBUG "result": [ 06:44:31 DEBUG { 06:44:31 DEBUG --- status code --- 06:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:31 DEBUG --- http response --- 06:44:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:44:31 DEBUG { 06:44:31 DEBUG "metric": { 06:44:31 INFO [http_cmd]: http status code OK 06:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:44:31 DEBUG }, 06:44:31 DEBUG "metric": { 06:44:31 DEBUG "instance": "10.142.0.25:9100" 06:44:31 DEBUG --- status code --- 06:44:31 DEBUG --- http response --- 06:44:31 DEBUG ----- output ----- 06:44:31 DEBUG "value": [ 06:44:31 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:44:31 DEBUG }, 06:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:44:31 DEBUG { 06:44:31 DEBUG "status": "success", 06:44:31 DEBUG }, 06:44:31 DEBUG "value": [ 06:44:31 DEBUG --- http response --- 06:44:31 DEBUG ----- output ----- 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG "value": [ 06:44:31 DEBUG 06:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:44:31 DEBUG { 06:44:31 DEBUG "status": "success", 06:44:31 DEBUG "data": { 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG ----- output ----- 06:44:31 DEBUG "data": { 06:44:31 DEBUG "resultType": "vector", 06:44:31 DEBUG "resultType": "vector", 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG { 06:44:31 DEBUG "result": [ 06:44:31 DEBUG { 06:44:31 DEBUG "metric": { 06:44:31 DEBUG 06:44:31 DEBUG [...] 06:44:31 DEBUG "status": "success", 06:44:31 DEBUG "data": { 06:44:31 DEBUG "resultType": "vector", 06:44:31 DEBUG 06:44:31 DEBUG [...] 06:44:31 DEBUG 06:44:31 DEBUG "pod": "ds-idrepo-0" 06:44:31 DEBUG }, 06:44:31 DEBUG "value": [ 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG "instance": "10.142.0.77:9100" 06:44:31 DEBUG }, 06:44:31 DEBUG [...] 06:44:31 DEBUG 06:44:31 DEBUG "result": [ 06:44:31 DEBUG "result": [ 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG { 06:44:31 DEBUG "metric": { 06:44:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:44:31 DEBUG "value": [ 06:44:31 DEBUG 06:44:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:44:31 DEBUG }, 06:44:31 DEBUG }, 06:44:31 DEBUG "value": [ 06:44:31 DEBUG 1712727871, 06:44:31 DEBUG "0.16633333333332795" 06:44:31 DEBUG ] 06:44:31 DEBUG } 06:44:31 DEBUG 06:44:31 DEBUG [...] 06:44:31 DEBUG { 06:44:31 DEBUG 06:44:31 DEBUG ] 06:44:31 DEBUG } 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG } 06:44:31 DEBUG "value": [ 06:44:31 DEBUG 06:44:31 DEBUG [...] 06:44:31 DEBUG 06:44:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:44:31 DEBUG 1712727871, 06:44:31 DEBUG "metric": { 06:44:31 DEBUG "pod": "ds-idrepo-0" 06:44:31 DEBUG 06:44:31 DEBUG }, 06:44:31 DEBUG "280301667.7454831" 06:44:31 DEBUG }, 06:44:31 DEBUG "value": [ 06:44:31 DEBUG 06:44:31 DEBUG "value": [ 06:44:31 DEBUG ] 06:44:31 DEBUG 06:44:31 DEBUG 06:44:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:44:31 DEBUG 1712727871, 06:44:31 DEBUG } 06:44:31 DEBUG 06:44:31 DEBUG [...] 06:44:31 DEBUG }, 06:44:31 DEBUG "0" 06:44:31 DEBUG ] 06:44:31 DEBUG 06:44:31 DEBUG "pod": "ds-cts-2" 06:44:31 DEBUG "value": [ 06:44:31 DEBUG ] 06:44:31 DEBUG } 06:44:31 DEBUG 06:44:31 DEBUG }, 06:44:31 DEBUG 1712727871, 06:44:31 DEBUG "252676" 06:44:31 DEBUG } 06:44:31 DEBUG 06:44:31 DEBUG "value": [ 06:44:31 DEBUG 1712727871, 06:44:31 DEBUG "0" 06:44:31 DEBUG ] 06:44:31 DEBUG } 06:44:31 DEBUG "pod": "ds-cts-2" 06:44:31 DEBUG ] 06:44:31 DEBUG } 06:44:31 DEBUG ] 06:44:31 DEBUG } 06:44:31 DEBUG }, 06:44:31 DEBUG "value": [ 06:44:31 DEBUG 1712727871, 06:44:31 DEBUG "29.930340299303403" 06:44:31 DEBUG ] 06:44:31 DEBUG } 06:44:31 DEBUG ] 06:44:31 DEBUG } 06:44:31 DEBUG } 06:44:31 DEBUG ] 06:44:31 DEBUG } 06:44:31 DEBUG } 06:44:31 DEBUG } 06:44:31 DEBUG } 06:44:31 DEBUG ] 06:44:31 DEBUG } 06:44:31 DEBUG } 06:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712727872" 06:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712727872" 06:44:32 INFO [http_cmd]: http status code OK 06:44:32 DEBUG --- status code --- 06:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:32 DEBUG --- http response --- 06:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:44:32 DEBUG ----- output ----- 06:44:32 DEBUG { 06:44:32 DEBUG "status": "success", 06:44:32 DEBUG "data": { 06:44:32 DEBUG "resultType": "vector", 06:44:32 DEBUG "result": [ 06:44:32 DEBUG { 06:44:32 DEBUG "metric": { 06:44:32 DEBUG "pod": "ds-idrepo-0" 06:44:32 DEBUG }, 06:44:32 DEBUG "value": [ 06:44:32 DEBUG 06:44:32 DEBUG 06:44:32 DEBUG 06:44:32 DEBUG [...] 06:44:32 DEBUG 06:44:32 DEBUG 06:44:32 DEBUG 06:44:32 DEBUG "pod": "ds-cts-2" 06:44:32 DEBUG }, 06:44:32 DEBUG "value": [ 06:44:32 DEBUG 1712727872, 06:44:32 INFO [http_cmd]: http status code OK 06:44:32 DEBUG --- status code --- 06:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:32 DEBUG "0" 06:44:32 DEBUG --- http response --- 06:44:32 DEBUG ] 06:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:44:32 DEBUG } 06:44:32 DEBUG ----- output ----- 06:44:32 DEBUG ] 06:44:32 DEBUG { 06:44:32 DEBUG } 06:44:32 DEBUG "status": "success", 06:44:32 DEBUG "data": { 06:44:32 DEBUG } 06:44:32 DEBUG "resultType": "vector", 06:44:32 DEBUG "result": [ 06:44:32 DEBUG { 06:44:32 DEBUG "metric": { 06:44:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:44:32 DEBUG }, 06:44:32 DEBUG "value": [ 06:44:32 DEBUG 06:44:32 DEBUG 06:44:32 DEBUG 06:44:32 DEBUG [...] 06:44:32 DEBUG 06:44:32 DEBUG 06:44:32 DEBUG 06:44:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:44:32 DEBUG }, 06:44:32 DEBUG "value": [ 06:44:32 DEBUG 1712727872, 06:44:32 DEBUG "4.487481347416324" 06:44:32 DEBUG ] 06:44:32 DEBUG } 06:44:32 DEBUG ] 06:44:32 DEBUG } 06:44:32 DEBUG } 06:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712727872" 06:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727872" 06:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727872" 06:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727872" 06:44:32 INFO [http_cmd]: http status code OK 06:44:32 DEBUG --- status code --- 06:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:32 DEBUG --- http response --- 06:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:44:32 DEBUG ----- output ----- 06:44:32 DEBUG { 06:44:32 DEBUG "status": "success", 06:44:32 DEBUG "data": { 06:44:32 DEBUG "resultType": "vector", 06:44:32 DEBUG "result": [ 06:44:32 DEBUG { 06:44:32 DEBUG "metric": { 06:44:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:44:32 DEBUG }, 06:44:32 DEBUG "value": [ 06:44:32 DEBUG 06:44:32 DEBUG 06:44:32 DEBUG 06:44:32 DEBUG [...] 06:44:32 DEBUG 06:44:32 DEBUG 06:44:32 DEBUG 06:44:32 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:44:32 DEBUG }, 06:44:32 DEBUG "value": [ 06:44:32 DEBUG 1712727872, 06:44:32 DEBUG "0.0535" 06:44:32 DEBUG ] 06:44:32 DEBUG } 06:44:32 DEBUG ] 06:44:32 DEBUG } 06:44:32 DEBUG } 06:44:32 INFO [http_cmd]: http status code OK 06:44:32 INFO [http_cmd]: http status code OK 06:44:32 DEBUG --- status code --- 06:44:32 DEBUG --- status code --- 06:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:32 DEBUG --- http response --- 06:44:32 DEBUG --- http response --- 06:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:44:32 DEBUG ----- output ----- 06:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:44:32 DEBUG ----- output ----- 06:44:32 DEBUG { 06:44:32 DEBUG "status": "success", 06:44:32 DEBUG "data": { 06:44:32 DEBUG "resultType": "vector", 06:44:32 DEBUG "result": [ 06:44:32 DEBUG { 06:44:32 DEBUG "status": "success", 06:44:32 DEBUG "data": { 06:44:32 DEBUG "resultType": "vector", 06:44:32 DEBUG "result": [ 06:44:32 DEBUG { 06:44:32 DEBUG { 06:44:32 DEBUG "metric": { 06:44:32 DEBUG "pod": "ds-idrepo-2" 06:44:32 DEBUG }, 06:44:32 DEBUG "value": [ 06:44:32 DEBUG "metric": { 06:44:32 DEBUG 06:44:32 INFO [http_cmd]: http status code OK 06:44:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:44:32 DEBUG 06:44:32 DEBUG --- status code --- 06:44:32 DEBUG }, 06:44:32 DEBUG 06:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:32 DEBUG "value": [ 06:44:32 DEBUG [...] 06:44:32 DEBUG --- http response --- 06:44:32 DEBUG 06:44:32 DEBUG 06:44:32 DEBUG 06:44:32 DEBUG 06:44:32 DEBUG 06:44:32 DEBUG "pod": "ds-cts-2" 06:44:32 DEBUG }, 06:44:32 DEBUG "value": [ 06:44:32 DEBUG 06:44:32 DEBUG 1712727872, 06:44:32 DEBUG "0.0005278738792787389" 06:44:32 DEBUG ] 06:44:32 DEBUG [...] 06:44:32 DEBUG 06:44:32 DEBUG 06:44:32 DEBUG 06:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:44:32 DEBUG ----- output ----- 06:44:32 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:44:32 DEBUG { 06:44:32 DEBUG } 06:44:32 DEBUG }, 06:44:32 DEBUG "status": "success", 06:44:32 DEBUG ] 06:44:32 DEBUG "value": [ 06:44:32 DEBUG "data": { 06:44:32 DEBUG } 06:44:32 DEBUG 1712727872, 06:44:32 DEBUG "1401.2666666666667" 06:44:32 DEBUG ] 06:44:32 DEBUG } 06:44:32 DEBUG ] 06:44:32 DEBUG } 06:44:32 DEBUG } 06:44:32 DEBUG "resultType": "vector", 06:44:32 DEBUG "result": [ 06:44:32 DEBUG { 06:44:32 DEBUG "metric": { 06:44:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:44:32 DEBUG }, 06:44:32 DEBUG "value": [ 06:44:32 DEBUG 06:44:32 DEBUG 06:44:32 DEBUG 06:44:32 DEBUG [...] 06:44:32 DEBUG 06:44:32 DEBUG 06:44:32 DEBUG 06:44:32 DEBUG "pod": "ds-idrepo-0" 06:44:32 DEBUG }, 06:44:32 DEBUG "value": [ 06:44:32 DEBUG 1712727872, 06:44:32 DEBUG "10.799346096078615" 06:44:32 DEBUG ] 06:44:32 DEBUG } 06:44:32 DEBUG ] 06:44:32 DEBUG } 06:44:32 DEBUG } 06:44:32 DEBUG } 06:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727872" 06:44:32 INFO [http_cmd]: http status code OK 06:44:32 DEBUG --- status code --- 06:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:32 DEBUG --- http response --- 06:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:44:32 DEBUG ----- output ----- 06:44:32 DEBUG { 06:44:32 DEBUG "status": "success", 06:44:32 DEBUG "data": { 06:44:32 DEBUG "resultType": "vector", 06:44:32 DEBUG "result": [ 06:44:32 DEBUG { 06:44:32 DEBUG "metric": { 06:44:32 DEBUG "pod": "ds-idrepo-0" 06:44:32 DEBUG }, 06:44:32 DEBUG "value": [ 06:44:32 DEBUG 06:44:32 DEBUG 06:44:32 DEBUG 06:44:32 DEBUG [...] 06:44:32 DEBUG 06:44:32 DEBUG 06:44:32 DEBUG 06:44:32 DEBUG "pod": "ds-idrepo-1" 06:44:32 DEBUG }, 06:44:32 DEBUG "value": [ 06:44:32 DEBUG 1712727872, 06:44:32 DEBUG "0" 06:44:32 DEBUG ] 06:44:32 DEBUG } 06:44:32 DEBUG ] 06:44:32 DEBUG } 06:44:32 DEBUG } 06:44:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727872" 06:44:32 INFO [http_cmd]: http status code OK 06:44:32 DEBUG --- status code --- 06:44:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:32 DEBUG --- http response --- 06:44:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:44:32 DEBUG ----- output ----- 06:44:32 DEBUG { 06:44:32 DEBUG "status": "success", 06:44:32 DEBUG "data": { 06:44:32 DEBUG "resultType": "vector", 06:44:32 DEBUG "result": [ 06:44:32 DEBUG { 06:44:32 DEBUG "metric": { 06:44:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:44:32 DEBUG }, 06:44:32 DEBUG "value": [ 06:44:32 DEBUG 06:44:32 DEBUG 06:44:32 DEBUG 06:44:32 DEBUG [...] 06:44:32 DEBUG 06:44:32 DEBUG 06:44:32 DEBUG 06:44:32 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:44:32 DEBUG }, 06:44:32 DEBUG "value": [ 06:44:32 DEBUG 1712727872, 06:44:32 DEBUG "1422.6333333333332" 06:44:32 DEBUG ] 06:44:32 DEBUG } 06:44:32 DEBUG ] 06:44:32 DEBUG } 06:44:32 DEBUG } 06:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727873" 06:44:33 INFO [http_cmd]: http status code OK 06:44:33 DEBUG --- status code --- 06:44:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:33 DEBUG --- http response --- 06:44:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:44:33 DEBUG ----- output ----- 06:44:33 DEBUG { 06:44:33 DEBUG "status": "success", 06:44:33 DEBUG "data": { 06:44:33 DEBUG "resultType": "vector", 06:44:33 DEBUG "result": [ 06:44:33 DEBUG { 06:44:33 DEBUG "metric": { 06:44:33 DEBUG "pod": "ds-idrepo-0" 06:44:33 DEBUG }, 06:44:33 DEBUG "value": [ 06:44:33 DEBUG 06:44:33 DEBUG 06:44:33 DEBUG 06:44:33 DEBUG [...] 06:44:33 DEBUG 06:44:33 DEBUG 06:44:33 DEBUG 06:44:33 DEBUG "pod": "ds-idrepo-1" 06:44:33 DEBUG }, 06:44:33 DEBUG "value": [ 06:44:33 DEBUG 1712727873, 06:44:33 DEBUG "0" 06:44:33 DEBUG ] 06:44:33 DEBUG } 06:44:33 DEBUG ] 06:44:33 DEBUG } 06:44:33 DEBUG } 06:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727873" 06:44:33 INFO [http_cmd]: http status code OK 06:44:33 DEBUG --- status code --- 06:44:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:33 DEBUG --- http response --- 06:44:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:44:33 DEBUG ----- output ----- 06:44:33 DEBUG { 06:44:33 DEBUG "status": "success", 06:44:33 DEBUG "data": { 06:44:33 DEBUG "resultType": "vector", 06:44:33 DEBUG "result": [ 06:44:33 DEBUG { 06:44:33 DEBUG "metric": { 06:44:33 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:44:33 DEBUG }, 06:44:33 DEBUG "value": [ 06:44:33 DEBUG 06:44:33 DEBUG 06:44:33 DEBUG 06:44:33 DEBUG [...] 06:44:33 DEBUG 06:44:33 DEBUG 06:44:33 DEBUG 06:44:33 DEBUG "pod": "ds-idrepo-0" 06:44:33 DEBUG }, 06:44:33 DEBUG "value": [ 06:44:33 DEBUG 1712727873, 06:44:33 DEBUG "3749.485086581737" 06:44:33 DEBUG ] 06:44:33 DEBUG } 06:44:33 DEBUG ] 06:44:33 DEBUG } 06:44:33 DEBUG } 06:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712727873" 06:44:33 INFO [http_cmd]: http status code OK 06:44:33 DEBUG --- status code --- 06:44:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:33 DEBUG --- http response --- 06:44:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:44:33 DEBUG ----- output ----- 06:44:33 DEBUG { 06:44:33 DEBUG "status": "success", 06:44:33 DEBUG "data": { 06:44:33 DEBUG "resultType": "vector", 06:44:33 DEBUG "result": [ 06:44:33 DEBUG { 06:44:33 DEBUG "metric": { 06:44:33 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:44:33 DEBUG }, 06:44:33 DEBUG "value": [ 06:44:33 DEBUG 06:44:33 DEBUG 06:44:33 DEBUG 06:44:33 DEBUG [...] 06:44:33 DEBUG 06:44:33 DEBUG 06:44:33 DEBUG 06:44:33 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:44:33 DEBUG }, 06:44:33 DEBUG "value": [ 06:44:33 DEBUG 1712727873, 06:44:33 DEBUG "2485784576" 06:44:33 DEBUG ] 06:44:33 DEBUG } 06:44:33 DEBUG ] 06:44:33 DEBUG } 06:44:33 DEBUG } 06:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727873" 06:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727873" 06:44:33 INFO [http_cmd]: http status code OK 06:44:33 DEBUG --- status code --- 06:44:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:33 DEBUG --- http response --- 06:44:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:44:33 DEBUG ----- output ----- 06:44:33 DEBUG { 06:44:33 DEBUG "status": "success", 06:44:33 DEBUG "data": { 06:44:33 DEBUG "resultType": "vector", 06:44:33 DEBUG "result": [ 06:44:33 DEBUG { 06:44:33 DEBUG "metric": { 06:44:33 DEBUG "instance": "10.142.0.25:9100" 06:44:33 DEBUG }, 06:44:33 DEBUG "value": [ 06:44:33 DEBUG 06:44:33 DEBUG 06:44:33 DEBUG 06:44:33 DEBUG [...] 06:44:33 DEBUG 06:44:33 DEBUG 06:44:33 DEBUG 06:44:33 DEBUG "instance": "10.142.0.77:9100" 06:44:33 DEBUG }, 06:44:33 DEBUG "value": [ 06:44:33 DEBUG 1712727873, 06:44:33 DEBUG "42831944.13333333" 06:44:33 DEBUG ] 06:44:33 DEBUG } 06:44:33 DEBUG ] 06:44:33 DEBUG } 06:44:33 DEBUG } 06:44:33 INFO [http_cmd]: http status code OK 06:44:33 DEBUG --- status code --- 06:44:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:33 DEBUG --- http response --- 06:44:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:44:33 DEBUG ----- output ----- 06:44:33 DEBUG { 06:44:33 DEBUG "status": "success", 06:44:33 DEBUG "data": { 06:44:33 DEBUG "resultType": "vector", 06:44:33 DEBUG "result": [ 06:44:33 DEBUG { 06:44:33 DEBUG "metric": { 06:44:33 DEBUG "instance": "10.142.0.25:9100" 06:44:33 DEBUG }, 06:44:33 DEBUG "value": [ 06:44:33 DEBUG 06:44:33 DEBUG 06:44:33 DEBUG 06:44:33 DEBUG [...] 06:44:33 DEBUG 06:44:33 DEBUG 06:44:33 DEBUG 06:44:33 DEBUG "instance": "10.142.0.77:9100" 06:44:33 DEBUG }, 06:44:33 DEBUG "value": [ 06:44:33 DEBUG 1712727873, 06:44:33 DEBUG "0.05633333333333373" 06:44:33 DEBUG ] 06:44:33 DEBUG } 06:44:33 DEBUG ] 06:44:33 DEBUG } 06:44:33 DEBUG } 06:44:53 INFO 06:44:53 INFO [loop_until]: kubectl --namespace=xlou top pods 06:44:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:44:53 INFO [loop_until]: OK (rc = 0) 06:44:53 DEBUG --- stdout --- 06:44:53 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 11133m 9027Mi am-654c4fcbcf-9scrj 9828m 8461Mi am-654c4fcbcf-v5qb5 8394m 8490Mi ds-cts-0 8700m 13391Mi ds-cts-1 8639m 13395Mi ds-cts-2 7755m 13393Mi ds-idrepo-0 10912m 5634Mi ds-idrepo-1 395m 5673Mi ds-idrepo-2 379m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 3m 1191Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 2m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4461m 2618Mi 06:44:53 DEBUG --- stderr --- 06:44:53 DEBUG 06:45:02 INFO 06:45:02 INFO [loop_until]: kubectl --namespace=xlou top node 06:45:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:45:02 INFO [loop_until]: OK (rc = 0) 06:45:02 DEBUG --- stdout --- 06:45:02 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 95m 0% 2308Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9420m 59% 9364Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 78m 0% 3259Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 14267m 89% 6901Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 11064m 69% 9346Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11951m 75% 9836Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 75m 0% 854Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10640m 66% 6322Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8388m 52% 14116Mi 24% gke-xlou-cdm-ds-32e4dcb1-4hls 7764m 48% 14094Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 432m 2% 6394Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8825m 55% 14118Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 464m 2% 6361Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4547m 28% 6201Mi 10% 06:45:02 DEBUG --- stderr --- 06:45:02 DEBUG 06:45:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712727930" 06:45:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727930" 06:45:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727930" 06:45:30 ERROR [http_cmd]: ERROR 06:45:30 ERROR --- status code --- 06:45:30 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:45:30 ERROR --- http response --- 06:45:30 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:45:30 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:45:30 INFO [http_cmd]: http status code OK 06:45:30 DEBUG --- status code --- 06:45:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:30 DEBUG --- http response --- 06:45:30 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:45:30 INFO [http_cmd]: http status code OK 06:45:30 DEBUG --- status code --- 06:45:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:30 DEBUG --- http response --- 06:45:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:45:30 DEBUG ----- output ----- 06:45:30 DEBUG { 06:45:30 DEBUG "status": "success", 06:45:30 DEBUG "data": { 06:45:30 DEBUG "resultType": "vector", 06:45:30 DEBUG "result": [ 06:45:30 DEBUG { 06:45:30 DEBUG "metric": { 06:45:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:45:30 DEBUG }, 06:45:30 DEBUG "value": [ 06:45:30 DEBUG 06:45:30 DEBUG 06:45:30 DEBUG 06:45:30 DEBUG [...] 06:45:30 DEBUG 06:45:30 DEBUG 06:45:30 DEBUG 06:45:30 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:45:30 DEBUG }, 06:45:30 DEBUG "value": [ 06:45:30 DEBUG 1712727930, 06:45:30 DEBUG "0" 06:45:30 DEBUG ] 06:45:30 DEBUG } 06:45:30 DEBUG ] 06:45:30 DEBUG } 06:45:30 DEBUG } 06:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727931" 06:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727931" 06:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727931" 06:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727931" 06:45:31 INFO [http_cmd]: http status code OK 06:45:31 DEBUG --- status code --- 06:45:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:31 DEBUG --- http response --- 06:45:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:45:31 DEBUG ----- output ----- 06:45:31 DEBUG { 06:45:31 DEBUG "status": "success", 06:45:31 DEBUG "data": { 06:45:31 DEBUG "resultType": "vector", 06:45:31 DEBUG "result": [ 06:45:31 DEBUG { 06:45:31 DEBUG "metric": { 06:45:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:45:31 DEBUG }, 06:45:31 DEBUG "value": [ 06:45:31 DEBUG 06:45:31 DEBUG 06:45:31 DEBUG 06:45:31 DEBUG [...] 06:45:31 DEBUG 06:45:31 DEBUG 06:45:31 DEBUG 06:45:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:45:31 DEBUG }, 06:45:31 DEBUG "value": [ 06:45:31 DEBUG 1712727931, 06:45:31 DEBUG "1391.4855314041872" 06:45:31 DEBUG ] 06:45:31 DEBUG } 06:45:31 DEBUG ] 06:45:31 DEBUG } 06:45:31 DEBUG } 06:45:31 INFO [http_cmd]: http status code OK 06:45:31 DEBUG --- status code --- 06:45:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:31 DEBUG --- http response --- 06:45:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:45:31 DEBUG ----- output ----- 06:45:31 DEBUG { 06:45:31 DEBUG "status": "success", 06:45:31 DEBUG "data": { 06:45:31 DEBUG "resultType": "vector", 06:45:31 DEBUG "result": [ 06:45:31 DEBUG { 06:45:31 DEBUG "metric": { 06:45:31 DEBUG "instance": "10.142.0.25:9100" 06:45:31 DEBUG }, 06:45:31 DEBUG "value": [ 06:45:31 DEBUG 06:45:31 DEBUG 06:45:31 DEBUG 06:45:31 DEBUG [...] 06:45:31 DEBUG 06:45:31 DEBUG 06:45:31 DEBUG 06:45:31 DEBUG "instance": "10.142.0.77:9100" 06:45:31 DEBUG }, 06:45:31 DEBUG "value": [ 06:45:31 DEBUG 1712727931, 06:45:31 DEBUG "42712860.666666664" 06:45:31 DEBUG ] 06:45:31 DEBUG } 06:45:31 DEBUG ] 06:45:31 DEBUG } 06:45:31 DEBUG } 06:45:31 INFO [http_cmd]: http status code OK 06:45:31 DEBUG --- status code --- 06:45:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:31 DEBUG --- http response --- 06:45:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:45:31 DEBUG ----- output ----- 06:45:31 DEBUG { 06:45:31 DEBUG "status": "success", 06:45:31 DEBUG "data": { 06:45:31 DEBUG "resultType": "vector", 06:45:31 DEBUG "result": [ 06:45:31 DEBUG { 06:45:31 DEBUG "metric": { 06:45:31 DEBUG "pod": "ds-idrepo-0" 06:45:31 DEBUG }, 06:45:31 DEBUG "value": [ 06:45:31 DEBUG 06:45:31 DEBUG 06:45:31 DEBUG 06:45:31 DEBUG [...] 06:45:31 DEBUG 06:45:31 DEBUG 06:45:31 DEBUG 06:45:31 DEBUG "pod": "ds-idrepo-1" 06:45:31 DEBUG }, 06:45:31 DEBUG "value": [ 06:45:31 DEBUG 1712727931, 06:45:31 DEBUG "0" 06:45:31 INFO [http_cmd]: http status code OK 06:45:31 DEBUG ] 06:45:31 DEBUG --- status code --- 06:45:31 DEBUG } 06:45:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:31 DEBUG ] 06:45:31 DEBUG --- http response --- 06:45:31 DEBUG } 06:45:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:45:31 DEBUG ----- output ----- 06:45:31 DEBUG } 06:45:31 DEBUG { 06:45:31 DEBUG "status": "success", 06:45:31 DEBUG "data": { 06:45:31 DEBUG "resultType": "vector", 06:45:31 DEBUG "result": [ 06:45:31 DEBUG { 06:45:31 DEBUG "metric": { 06:45:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:45:31 DEBUG }, 06:45:31 DEBUG "value": [ 06:45:31 DEBUG 06:45:31 DEBUG 06:45:31 DEBUG 06:45:31 DEBUG [...] 06:45:31 DEBUG 06:45:31 DEBUG 06:45:31 DEBUG 06:45:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:45:31 DEBUG }, 06:45:31 DEBUG "value": [ 06:45:31 DEBUG 1712727931, 06:45:31 DEBUG "1395.0526736898253" 06:45:31 DEBUG ] 06:45:31 DEBUG } 06:45:31 DEBUG ] 06:45:31 DEBUG } 06:45:31 DEBUG } 06:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727931" 06:45:31 INFO [http_cmd]: http status code OK 06:45:31 DEBUG --- status code --- 06:45:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:31 DEBUG --- http response --- 06:45:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:45:31 DEBUG ----- output ----- 06:45:31 DEBUG { 06:45:31 DEBUG "status": "success", 06:45:31 DEBUG "data": { 06:45:31 DEBUG "resultType": "vector", 06:45:31 DEBUG "result": [ 06:45:31 DEBUG { 06:45:31 DEBUG "metric": { 06:45:31 DEBUG "instance": "10.142.0.25:9100" 06:45:31 DEBUG }, 06:45:31 DEBUG "value": [ 06:45:31 DEBUG 06:45:31 DEBUG 06:45:31 DEBUG 06:45:31 DEBUG [...] 06:45:31 DEBUG 06:45:31 DEBUG 06:45:31 DEBUG 06:45:31 DEBUG "instance": "10.142.0.77:9100" 06:45:31 DEBUG }, 06:45:31 DEBUG "value": [ 06:45:31 DEBUG 1712727931, 06:45:31 DEBUG "433850997760" 06:45:31 DEBUG ] 06:45:31 DEBUG } 06:45:31 DEBUG ] 06:45:31 DEBUG } 06:45:31 DEBUG } 06:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727931" 06:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727931" 06:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727931" 06:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727931" 06:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727931" 06:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727931" 06:45:32 INFO [http_cmd]: http status code OK 06:45:32 DEBUG --- status code --- 06:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:32 DEBUG --- http response --- 06:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:45:32 DEBUG ----- output ----- 06:45:32 DEBUG { 06:45:32 DEBUG "status": "success", 06:45:32 DEBUG "data": { 06:45:32 DEBUG "resultType": "vector", 06:45:32 DEBUG "result": [ 06:45:32 DEBUG { 06:45:32 DEBUG "metric": { 06:45:32 DEBUG "instance": "10.142.0.25:9100" 06:45:32 DEBUG }, 06:45:32 DEBUG "value": [ 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG [...] 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG "instance": "10.142.0.77:9100" 06:45:32 DEBUG }, 06:45:32 DEBUG "value": [ 06:45:32 DEBUG 1712727931, 06:45:32 DEBUG "0.15296666666666545" 06:45:32 DEBUG ] 06:45:32 DEBUG } 06:45:32 DEBUG ] 06:45:32 DEBUG } 06:45:32 DEBUG } 06:45:32 INFO [http_cmd]: http status code OK 06:45:32 DEBUG --- status code --- 06:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:32 DEBUG --- http response --- 06:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:45:32 DEBUG ----- output ----- 06:45:32 DEBUG { 06:45:32 DEBUG "status": "success", 06:45:32 DEBUG "data": { 06:45:32 DEBUG "resultType": "vector", 06:45:32 DEBUG "result": [ 06:45:32 DEBUG { 06:45:32 INFO [http_cmd]: http status code OK 06:45:32 INFO [http_cmd]: http status code OK 06:45:32 DEBUG --- status code --- 06:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:32 DEBUG --- status code --- 06:45:32 DEBUG --- http response --- 06:45:32 INFO [http_cmd]: http status code OK 06:45:32 DEBUG --- status code --- 06:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:32 DEBUG --- http response --- 06:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:32 DEBUG "metric": { 06:45:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:45:32 DEBUG --- http response --- 06:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:45:32 DEBUG ----- output ----- 06:45:32 DEBUG { 06:45:32 DEBUG "status": "success", 06:45:32 DEBUG ----- output ----- 06:45:32 INFO [http_cmd]: http status code OK 06:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:45:32 DEBUG ----- output ----- 06:45:32 DEBUG { 06:45:32 DEBUG { 06:45:32 DEBUG --- status code --- 06:45:32 DEBUG "data": { 06:45:32 DEBUG "resultType": "vector", 06:45:32 DEBUG "result": [ 06:45:32 DEBUG { 06:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:32 DEBUG --- http response --- 06:45:32 DEBUG "metric": { 06:45:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:45:32 DEBUG }, 06:45:32 DEBUG "status": "success", 06:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:45:32 DEBUG }, 06:45:32 DEBUG "status": "success", 06:45:32 DEBUG "data": { 06:45:32 DEBUG "data": { 06:45:32 DEBUG "resultType": "vector", 06:45:32 DEBUG "value": [ 06:45:32 DEBUG "resultType": "vector", 06:45:32 DEBUG ----- output ----- 06:45:32 DEBUG { 06:45:32 DEBUG "status": "success", 06:45:32 DEBUG 06:45:32 DEBUG "result": [ 06:45:32 DEBUG "data": { 06:45:32 DEBUG "resultType": "vector", 06:45:32 DEBUG "result": [ 06:45:32 DEBUG { 06:45:32 DEBUG "metric": { 06:45:32 DEBUG { 06:45:32 DEBUG "metric": { 06:45:32 DEBUG "pod": "ds-idrepo-0" 06:45:32 DEBUG }, 06:45:32 DEBUG "result": [ 06:45:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:45:32 DEBUG 06:45:32 DEBUG "value": [ 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG { 06:45:32 DEBUG "metric": { 06:45:32 DEBUG "pod": "ds-idrepo-0" 06:45:32 DEBUG }, 06:45:32 DEBUG [...] 06:45:32 DEBUG "value": [ 06:45:32 DEBUG "value": [ 06:45:32 DEBUG [...] 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG }, 06:45:32 DEBUG "value": [ 06:45:32 DEBUG 1712727931, 06:45:32 DEBUG "63406.51486349849" 06:45:32 DEBUG ] 06:45:32 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:45:32 DEBUG }, 06:45:32 DEBUG } 06:45:32 DEBUG ] 06:45:32 DEBUG } 06:45:32 DEBUG } 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG }, 06:45:32 DEBUG "value": [ 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG "value": [ 06:45:32 DEBUG 1712727931, 06:45:32 DEBUG [...] 06:45:32 DEBUG [...] 06:45:32 DEBUG 06:45:32 DEBUG "252445" 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG ] 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG } 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG [...] 06:45:32 DEBUG ] 06:45:32 DEBUG "pod": "ds-cts-2" 06:45:32 DEBUG }, 06:45:32 DEBUG 06:45:32 DEBUG } 06:45:32 DEBUG "value": [ 06:45:32 DEBUG 1712727931, 06:45:32 DEBUG 06:45:32 DEBUG } 06:45:32 DEBUG "0" 06:45:32 DEBUG ] 06:45:32 DEBUG 06:45:32 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:45:32 DEBUG } 06:45:32 DEBUG }, 06:45:32 DEBUG "value": [ 06:45:32 DEBUG ] 06:45:32 DEBUG 1712727931, 06:45:32 DEBUG "0" 06:45:32 DEBUG } 06:45:32 DEBUG ] 06:45:32 DEBUG } 06:45:32 DEBUG } 06:45:32 DEBUG ] 06:45:32 DEBUG } 06:45:32 DEBUG } 06:45:32 DEBUG "pod": "ds-cts-2" 06:45:32 DEBUG }, 06:45:32 DEBUG "value": [ 06:45:32 DEBUG 1712727931, 06:45:32 DEBUG "29.9" 06:45:32 DEBUG ] 06:45:32 DEBUG } 06:45:32 DEBUG ] 06:45:32 DEBUG } 06:45:32 DEBUG } 06:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712727932" 06:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712727932" 06:45:32 INFO [http_cmd]: http status code OK 06:45:32 DEBUG --- status code --- 06:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:32 DEBUG --- http response --- 06:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:45:32 DEBUG ----- output ----- 06:45:32 DEBUG { 06:45:32 DEBUG "status": "success", 06:45:32 DEBUG "data": { 06:45:32 DEBUG "resultType": "vector", 06:45:32 DEBUG "result": [ 06:45:32 DEBUG { 06:45:32 DEBUG "metric": { 06:45:32 DEBUG "pod": "ds-idrepo-0" 06:45:32 DEBUG }, 06:45:32 DEBUG "value": [ 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG [...] 06:45:32 INFO [http_cmd]: http status code OK 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG --- status code --- 06:45:32 DEBUG 06:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:32 DEBUG "pod": "ds-cts-2" 06:45:32 DEBUG --- http response --- 06:45:32 DEBUG }, 06:45:32 DEBUG "value": [ 06:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:45:32 DEBUG ----- output ----- 06:45:32 DEBUG 1712727932, 06:45:32 DEBUG { 06:45:32 DEBUG "0" 06:45:32 DEBUG "status": "success", 06:45:32 DEBUG "data": { 06:45:32 DEBUG ] 06:45:32 DEBUG "resultType": "vector", 06:45:32 DEBUG } 06:45:32 DEBUG "result": [ 06:45:32 DEBUG ] 06:45:32 DEBUG { 06:45:32 DEBUG } 06:45:32 DEBUG "metric": { 06:45:32 DEBUG } 06:45:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:45:32 DEBUG }, 06:45:32 DEBUG "value": [ 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG [...] 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:45:32 DEBUG }, 06:45:32 DEBUG "value": [ 06:45:32 DEBUG 1712727932, 06:45:32 DEBUG "4.414746889672387" 06:45:32 DEBUG ] 06:45:32 DEBUG } 06:45:32 DEBUG ] 06:45:32 DEBUG } 06:45:32 DEBUG } 06:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712727932" 06:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727932" 06:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727932" 06:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727932" 06:45:32 INFO [http_cmd]: http status code OK 06:45:32 DEBUG --- status code --- 06:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:32 DEBUG --- http response --- 06:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:45:32 DEBUG ----- output ----- 06:45:32 DEBUG { 06:45:32 DEBUG "status": "success", 06:45:32 DEBUG "data": { 06:45:32 DEBUG "resultType": "vector", 06:45:32 DEBUG "result": [ 06:45:32 DEBUG { 06:45:32 DEBUG "metric": { 06:45:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:45:32 DEBUG }, 06:45:32 DEBUG "value": [ 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG [...] 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:45:32 DEBUG }, 06:45:32 DEBUG "value": [ 06:45:32 DEBUG 1712727932, 06:45:32 DEBUG "0.055" 06:45:32 DEBUG ] 06:45:32 DEBUG } 06:45:32 DEBUG ] 06:45:32 DEBUG } 06:45:32 DEBUG } 06:45:32 INFO [http_cmd]: http status code OK 06:45:32 DEBUG --- status code --- 06:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:32 DEBUG --- http response --- 06:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:45:32 DEBUG ----- output ----- 06:45:32 DEBUG { 06:45:32 DEBUG "status": "success", 06:45:32 DEBUG "data": { 06:45:32 DEBUG "resultType": "vector", 06:45:32 DEBUG "result": [ 06:45:32 DEBUG { 06:45:32 DEBUG "metric": { 06:45:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:45:32 DEBUG }, 06:45:32 DEBUG "value": [ 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG [...] 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:45:32 DEBUG }, 06:45:32 DEBUG "value": [ 06:45:32 DEBUG 1712727932, 06:45:32 DEBUG "1390.95212695026" 06:45:32 DEBUG ] 06:45:32 INFO [http_cmd]: http status code OK 06:45:32 DEBUG --- status code --- 06:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:32 DEBUG --- http response --- 06:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:45:32 DEBUG ----- output ----- 06:45:32 DEBUG { 06:45:32 DEBUG "status": "success", 06:45:32 DEBUG "data": { 06:45:32 DEBUG "resultType": "vector", 06:45:32 DEBUG "result": [ 06:45:32 DEBUG { 06:45:32 DEBUG "metric": { 06:45:32 DEBUG "pod": "ds-idrepo-2" 06:45:32 DEBUG } 06:45:32 DEBUG ] 06:45:32 DEBUG } 06:45:32 DEBUG } 06:45:32 INFO [http_cmd]: http status code OK 06:45:32 DEBUG }, 06:45:32 DEBUG "value": [ 06:45:32 DEBUG --- status code --- 06:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:32 DEBUG --- http response --- 06:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:45:32 DEBUG ----- output ----- 06:45:32 DEBUG { 06:45:32 DEBUG "status": "success", 06:45:32 DEBUG "data": { 06:45:32 DEBUG "resultType": "vector", 06:45:32 DEBUG "result": [ 06:45:32 DEBUG { 06:45:32 DEBUG "metric": { 06:45:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:45:32 DEBUG }, 06:45:32 DEBUG "value": [ 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG [...] 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG "pod": "ds-cts-2" 06:45:32 DEBUG }, 06:45:32 DEBUG "value": [ 06:45:32 DEBUG 1712727932, 06:45:32 DEBUG "4.671606941120989" 06:45:32 DEBUG ] 06:45:32 DEBUG } 06:45:32 DEBUG ] 06:45:32 DEBUG } 06:45:32 DEBUG } 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG [...] 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG "pod": "ds-cts-2" 06:45:32 DEBUG }, 06:45:32 DEBUG "value": [ 06:45:32 DEBUG 1712727932, 06:45:32 DEBUG "0.00023262166666666664" 06:45:32 DEBUG ] 06:45:32 DEBUG } 06:45:32 DEBUG ] 06:45:32 DEBUG } 06:45:32 DEBUG } 06:45:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727932" 06:45:32 INFO [http_cmd]: http status code OK 06:45:32 DEBUG --- status code --- 06:45:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:32 DEBUG --- http response --- 06:45:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:45:32 DEBUG ----- output ----- 06:45:32 DEBUG { 06:45:32 DEBUG "status": "success", 06:45:32 DEBUG "data": { 06:45:32 DEBUG "resultType": "vector", 06:45:32 DEBUG "result": [ 06:45:32 DEBUG { 06:45:32 DEBUG "metric": { 06:45:32 DEBUG "pod": "ds-idrepo-0" 06:45:32 DEBUG }, 06:45:32 DEBUG "value": [ 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG [...] 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG 06:45:32 DEBUG "pod": "ds-idrepo-1" 06:45:32 DEBUG }, 06:45:32 DEBUG "value": [ 06:45:32 DEBUG 1712727932, 06:45:32 DEBUG "0" 06:45:32 DEBUG ] 06:45:32 DEBUG } 06:45:32 DEBUG ] 06:45:32 DEBUG } 06:45:32 DEBUG } 06:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727933" 06:45:33 INFO [http_cmd]: http status code OK 06:45:33 DEBUG --- status code --- 06:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:33 DEBUG --- http response --- 06:45:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:45:33 DEBUG ----- output ----- 06:45:33 DEBUG { 06:45:33 DEBUG "status": "success", 06:45:33 DEBUG "data": { 06:45:33 DEBUG "resultType": "vector", 06:45:33 DEBUG "result": [ 06:45:33 DEBUG { 06:45:33 DEBUG "metric": { 06:45:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:45:33 DEBUG }, 06:45:33 DEBUG "value": [ 06:45:33 DEBUG 06:45:33 DEBUG 06:45:33 DEBUG 06:45:33 DEBUG [...] 06:45:33 DEBUG 06:45:33 DEBUG 06:45:33 DEBUG 06:45:33 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:45:33 DEBUG }, 06:45:33 DEBUG "value": [ 06:45:33 DEBUG 1712727933, 06:45:33 DEBUG "1406.7875716762235" 06:45:33 DEBUG ] 06:45:33 DEBUG } 06:45:33 DEBUG ] 06:45:33 DEBUG } 06:45:33 DEBUG } 06:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727933" 06:45:33 INFO [http_cmd]: http status code OK 06:45:33 DEBUG --- status code --- 06:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:33 DEBUG --- http response --- 06:45:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:45:33 DEBUG ----- output ----- 06:45:33 DEBUG { 06:45:33 DEBUG "status": "success", 06:45:33 DEBUG "data": { 06:45:33 DEBUG "resultType": "vector", 06:45:33 DEBUG "result": [ 06:45:33 DEBUG { 06:45:33 DEBUG "metric": { 06:45:33 DEBUG "pod": "ds-idrepo-0" 06:45:33 DEBUG }, 06:45:33 DEBUG "value": [ 06:45:33 DEBUG 06:45:33 DEBUG 06:45:33 DEBUG 06:45:33 DEBUG [...] 06:45:33 DEBUG 06:45:33 DEBUG 06:45:33 DEBUG 06:45:33 DEBUG "pod": "ds-idrepo-1" 06:45:33 DEBUG }, 06:45:33 DEBUG "value": [ 06:45:33 DEBUG 1712727933, 06:45:33 DEBUG "0" 06:45:33 DEBUG ] 06:45:33 DEBUG } 06:45:33 DEBUG ] 06:45:33 DEBUG } 06:45:33 DEBUG } 06:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727933" 06:45:33 INFO [http_cmd]: http status code OK 06:45:33 DEBUG --- status code --- 06:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:33 DEBUG --- http response --- 06:45:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:45:33 DEBUG ----- output ----- 06:45:33 DEBUG { 06:45:33 DEBUG "status": "success", 06:45:33 DEBUG "data": { 06:45:33 DEBUG "resultType": "vector", 06:45:33 DEBUG "result": [ 06:45:33 DEBUG { 06:45:33 DEBUG "metric": { 06:45:33 DEBUG "pod": "ds-cts-0" 06:45:33 DEBUG }, 06:45:33 DEBUG "value": [ 06:45:33 DEBUG 06:45:33 DEBUG 06:45:33 DEBUG 06:45:33 DEBUG [...] 06:45:33 DEBUG 06:45:33 DEBUG 06:45:33 DEBUG 06:45:33 DEBUG "pod": "ds-cts-2" 06:45:33 DEBUG }, 06:45:33 DEBUG "value": [ 06:45:33 DEBUG 1712727933, 06:45:33 DEBUG "15767665.124711145" 06:45:33 DEBUG ] 06:45:33 DEBUG } 06:45:33 DEBUG ] 06:45:33 DEBUG } 06:45:33 DEBUG } 06:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712727933" 06:45:33 INFO [http_cmd]: http status code OK 06:45:33 DEBUG --- status code --- 06:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:33 DEBUG --- http response --- 06:45:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:45:33 DEBUG ----- output ----- 06:45:33 DEBUG { 06:45:33 DEBUG "status": "success", 06:45:33 DEBUG "data": { 06:45:33 DEBUG "resultType": "vector", 06:45:33 DEBUG "result": [ 06:45:33 DEBUG { 06:45:33 DEBUG "metric": { 06:45:33 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:45:33 DEBUG }, 06:45:33 DEBUG "value": [ 06:45:33 DEBUG 06:45:33 DEBUG 06:45:33 DEBUG 06:45:33 DEBUG [...] 06:45:33 DEBUG 06:45:33 DEBUG 06:45:33 DEBUG 06:45:33 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:45:33 DEBUG }, 06:45:33 DEBUG "value": [ 06:45:33 DEBUG 1712727933, 06:45:33 DEBUG "2485809152" 06:45:33 DEBUG ] 06:45:33 DEBUG } 06:45:33 DEBUG ] 06:45:33 DEBUG } 06:45:33 DEBUG } 06:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727933" 06:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727933" 06:45:33 INFO [http_cmd]: http status code OK 06:45:33 DEBUG --- status code --- 06:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:33 DEBUG --- http response --- 06:45:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:45:33 DEBUG ----- output ----- 06:45:33 DEBUG { 06:45:33 DEBUG "status": "success", 06:45:33 DEBUG "data": { 06:45:33 DEBUG "resultType": "vector", 06:45:33 DEBUG "result": [ 06:45:33 DEBUG { 06:45:33 DEBUG "metric": { 06:45:33 DEBUG "instance": "10.142.0.25:9100" 06:45:33 DEBUG }, 06:45:33 DEBUG "value": [ 06:45:33 DEBUG 06:45:33 DEBUG 06:45:33 DEBUG 06:45:33 DEBUG [...] 06:45:33 DEBUG 06:45:33 DEBUG 06:45:33 DEBUG 06:45:33 DEBUG "instance": "10.142.0.77:9100" 06:45:33 DEBUG }, 06:45:33 DEBUG "value": [ 06:45:33 DEBUG 1712727933, 06:45:33 DEBUG "42714701.266666666" 06:45:33 DEBUG ] 06:45:33 DEBUG } 06:45:33 DEBUG ] 06:45:33 DEBUG } 06:45:33 DEBUG } 06:45:33 INFO [http_cmd]: http status code OK 06:45:33 DEBUG --- status code --- 06:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:33 DEBUG --- http response --- 06:45:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:45:33 DEBUG ----- output ----- 06:45:33 DEBUG { 06:45:33 DEBUG "status": "success", 06:45:33 DEBUG "data": { 06:45:33 DEBUG "resultType": "vector", 06:45:33 DEBUG "result": [ 06:45:33 DEBUG { 06:45:33 DEBUG "metric": { 06:45:33 DEBUG "instance": "10.142.0.25:9100" 06:45:33 DEBUG }, 06:45:33 DEBUG "value": [ 06:45:33 DEBUG 06:45:33 DEBUG 06:45:33 DEBUG 06:45:33 DEBUG [...] 06:45:33 DEBUG 06:45:33 DEBUG 06:45:33 DEBUG 06:45:33 DEBUG "instance": "10.142.0.77:9100" 06:45:33 DEBUG }, 06:45:33 DEBUG "value": [ 06:45:33 DEBUG 1712727933, 06:45:33 DEBUG "0.055666666666666365" 06:45:33 DEBUG ] 06:45:33 DEBUG } 06:45:33 DEBUG ] 06:45:33 DEBUG } 06:45:33 DEBUG } 06:45:53 INFO 06:45:53 INFO [loop_until]: kubectl --namespace=xlou top pods 06:45:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:45:54 INFO [loop_until]: OK (rc = 0) 06:45:54 DEBUG --- stdout --- 06:45:54 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 11006m 9024Mi am-654c4fcbcf-9scrj 9669m 8461Mi am-654c4fcbcf-v5qb5 8294m 8491Mi ds-cts-0 8313m 12756Mi ds-cts-1 8527m 12808Mi ds-cts-2 7402m 12884Mi ds-idrepo-0 10920m 5634Mi ds-idrepo-1 368m 5672Mi ds-idrepo-2 389m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 5m 1191Mi idm-6bfb7c96cc-vsb58 3m 2213Mi lodemon-b776b7d88-thrqf 11m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 4379m 2658Mi 06:45:54 DEBUG --- stderr --- 06:45:54 DEBUG 06:46:02 INFO 06:46:02 INFO [loop_until]: kubectl --namespace=xlou top node 06:46:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:46:02 INFO [loop_until]: OK (rc = 0) 06:46:02 DEBUG --- stdout --- 06:46:02 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 95m 0% 2294Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 9282m 58% 9364Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 80m 0% 3258Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 13572m 85% 6923Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 11036m 69% 9342Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 11991m 75% 9835Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 76m 0% 859Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 10923m 68% 6322Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 8417m 52% 13522Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 7567m 47% 13673Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 429m 2% 6399Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 8481m 53% 13543Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 431m 2% 6362Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4508m 28% 6241Mi 10% 06:46:02 DEBUG --- stderr --- 06:46:02 DEBUG 06:46:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727990" 06:46:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727990" 06:46:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712727990" 06:46:30 INFO [http_cmd]: http status code OK 06:46:30 INFO [http_cmd]: http status code OK 06:46:30 DEBUG --- status code --- 06:46:30 DEBUG --- status code --- 06:46:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:30 DEBUG --- http response --- 06:46:30 DEBUG --- http response --- 06:46:30 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:46:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:46:30 ERROR [http_cmd]: ERROR 06:46:30 DEBUG ----- output ----- 06:46:30 ERROR --- status code --- 06:46:30 DEBUG { 06:46:30 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:46:30 DEBUG "status": "success", 06:46:30 ERROR --- http response --- 06:46:30 DEBUG "data": { 06:46:30 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:46:30 DEBUG "resultType": "vector", 06:46:30 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:46:30 DEBUG "result": [ 06:46:30 DEBUG { 06:46:30 DEBUG "metric": { 06:46:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:46:30 DEBUG }, 06:46:30 DEBUG "value": [ 06:46:30 DEBUG 06:46:30 DEBUG 06:46:30 DEBUG 06:46:30 DEBUG [...] 06:46:30 DEBUG 06:46:30 DEBUG 06:46:30 DEBUG 06:46:30 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:46:30 DEBUG }, 06:46:30 DEBUG "value": [ 06:46:30 DEBUG 1712727990, 06:46:30 DEBUG "0" 06:46:30 DEBUG ] 06:46:30 DEBUG } 06:46:30 DEBUG ] 06:46:30 DEBUG } 06:46:30 DEBUG } 06:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727991" 06:46:31 INFO [http_cmd]: http status code OK 06:46:31 DEBUG --- status code --- 06:46:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:31 DEBUG --- http response --- 06:46:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:46:31 DEBUG ----- output ----- 06:46:31 DEBUG { 06:46:31 DEBUG "status": "success", 06:46:31 DEBUG "data": { 06:46:31 DEBUG "resultType": "vector", 06:46:31 DEBUG "result": [ 06:46:31 DEBUG { 06:46:31 DEBUG "metric": { 06:46:31 DEBUG "instance": "10.142.0.25:9100" 06:46:31 DEBUG }, 06:46:31 DEBUG "value": [ 06:46:31 DEBUG 06:46:31 DEBUG 06:46:31 DEBUG 06:46:31 DEBUG [...] 06:46:31 DEBUG 06:46:31 DEBUG 06:46:31 DEBUG 06:46:31 DEBUG "instance": "10.142.0.77:9100" 06:46:31 DEBUG }, 06:46:31 DEBUG "value": [ 06:46:31 DEBUG 1712727991, 06:46:31 DEBUG "36878038.06666666" 06:46:31 DEBUG ] 06:46:31 DEBUG } 06:46:31 DEBUG ] 06:46:31 DEBUG } 06:46:31 DEBUG } 06:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727991" 06:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727991" 06:46:31 INFO [http_cmd]: http status code OK 06:46:31 INFO [http_cmd]: http status code OK 06:46:31 DEBUG --- status code --- 06:46:31 DEBUG --- status code --- 06:46:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:31 DEBUG --- http response --- 06:46:31 DEBUG --- http response --- 06:46:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:46:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:46:31 DEBUG ----- output ----- 06:46:31 DEBUG ----- output ----- 06:46:31 DEBUG { 06:46:31 DEBUG { 06:46:31 DEBUG "status": "success", 06:46:31 DEBUG "status": "success", 06:46:31 DEBUG "data": { 06:46:31 DEBUG "data": { 06:46:31 DEBUG "resultType": "vector", 06:46:31 DEBUG "resultType": "vector", 06:46:31 DEBUG "result": [ 06:46:31 DEBUG "result": [ 06:46:31 DEBUG { 06:46:31 DEBUG { 06:46:31 DEBUG "metric": { 06:46:31 DEBUG "metric": { 06:46:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:46:31 DEBUG "pod": "ds-idrepo-0" 06:46:31 DEBUG }, 06:46:31 DEBUG }, 06:46:31 DEBUG "value": [ 06:46:31 DEBUG "value": [ 06:46:31 DEBUG 06:46:31 DEBUG 06:46:31 DEBUG 06:46:31 DEBUG 06:46:31 DEBUG 06:46:31 DEBUG 06:46:31 DEBUG [...] 06:46:31 DEBUG [...] 06:46:31 DEBUG 06:46:31 DEBUG 06:46:31 DEBUG 06:46:31 DEBUG 06:46:31 DEBUG 06:46:31 DEBUG 06:46:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:46:31 DEBUG "pod": "ds-idrepo-1" 06:46:31 DEBUG }, 06:46:31 DEBUG }, 06:46:31 DEBUG "value": [ 06:46:31 DEBUG "value": [ 06:46:31 DEBUG 1712727991, 06:46:31 DEBUG 1712727991, 06:46:31 DEBUG "133.43333333333334" 06:46:31 DEBUG "0" 06:46:31 DEBUG ] 06:46:31 DEBUG ] 06:46:31 DEBUG } 06:46:31 DEBUG } 06:46:31 DEBUG ] 06:46:31 DEBUG ] 06:46:31 DEBUG } 06:46:31 DEBUG } 06:46:31 DEBUG } 06:46:31 DEBUG } 06:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727991" 06:46:31 INFO [http_cmd]: http status code OK 06:46:31 DEBUG --- status code --- 06:46:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:31 DEBUG --- http response --- 06:46:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:46:31 DEBUG ----- output ----- 06:46:31 DEBUG { 06:46:31 DEBUG "status": "success", 06:46:31 DEBUG "data": { 06:46:31 DEBUG "resultType": "vector", 06:46:31 DEBUG "result": [ 06:46:31 DEBUG { 06:46:31 DEBUG "metric": { 06:46:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:46:31 DEBUG }, 06:46:31 DEBUG "value": [ 06:46:31 DEBUG 06:46:31 DEBUG 06:46:31 DEBUG 06:46:31 DEBUG [...] 06:46:31 DEBUG 06:46:31 DEBUG 06:46:31 DEBUG 06:46:31 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:46:31 DEBUG }, 06:46:31 DEBUG "value": [ 06:46:31 DEBUG 1712727991, 06:46:31 DEBUG "124.8" 06:46:31 DEBUG ] 06:46:31 DEBUG } 06:46:31 DEBUG ] 06:46:31 DEBUG } 06:46:31 DEBUG } 06:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727992" 06:46:32 INFO [http_cmd]: http status code OK 06:46:32 DEBUG --- status code --- 06:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:32 DEBUG --- http response --- 06:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:46:32 DEBUG ----- output ----- 06:46:32 DEBUG { 06:46:32 DEBUG "status": "success", 06:46:32 DEBUG "data": { 06:46:32 DEBUG "resultType": "vector", 06:46:32 DEBUG "result": [ 06:46:32 DEBUG { 06:46:32 DEBUG "metric": { 06:46:32 DEBUG "instance": "10.142.0.25:9100" 06:46:32 DEBUG }, 06:46:32 DEBUG "value": [ 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG [...] 06:46:32 DEBUG 06:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727992" 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG "instance": "10.142.0.77:9100" 06:46:32 DEBUG }, 06:46:32 DEBUG "value": [ 06:46:32 DEBUG 1712727992, 06:46:32 DEBUG "435678710784" 06:46:32 DEBUG ] 06:46:32 DEBUG } 06:46:32 DEBUG ] 06:46:32 DEBUG } 06:46:32 DEBUG } 06:46:32 INFO [http_cmd]: http status code OK 06:46:32 DEBUG --- status code --- 06:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:32 DEBUG --- http response --- 06:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:46:32 DEBUG ----- output ----- 06:46:32 DEBUG { 06:46:32 DEBUG "status": "success", 06:46:32 DEBUG "data": { 06:46:32 DEBUG "resultType": "vector", 06:46:32 DEBUG "result": [ 06:46:32 DEBUG { 06:46:32 DEBUG "metric": { 06:46:32 DEBUG "instance": "10.142.0.25:9100" 06:46:32 DEBUG }, 06:46:32 DEBUG "value": [ 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG [...] 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG "instance": "10.142.0.77:9100" 06:46:32 DEBUG }, 06:46:32 DEBUG "value": [ 06:46:32 DEBUG 1712727992, 06:46:32 DEBUG "0.1532000000000001" 06:46:32 DEBUG ] 06:46:32 DEBUG } 06:46:32 DEBUG ] 06:46:32 DEBUG } 06:46:32 DEBUG } 06:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727992" 06:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727992" 06:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727992" 06:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727992" 06:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727992" 06:46:32 INFO [http_cmd]: http status code OK 06:46:32 INFO [http_cmd]: http status code OK 06:46:32 INFO [http_cmd]: http status code OK 06:46:32 DEBUG --- status code --- 06:46:32 DEBUG --- status code --- 06:46:32 DEBUG --- status code --- 06:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:32 DEBUG --- http response --- 06:46:32 DEBUG --- http response --- 06:46:32 DEBUG --- http response --- 06:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:46:32 DEBUG ----- output ----- 06:46:32 DEBUG ----- output ----- 06:46:32 DEBUG ----- output ----- 06:46:32 DEBUG { 06:46:32 DEBUG { 06:46:32 DEBUG { 06:46:32 DEBUG "status": "success", 06:46:32 INFO [http_cmd]: http status code OK 06:46:32 DEBUG "status": "success", 06:46:32 DEBUG "status": "success", 06:46:32 DEBUG "data": { 06:46:32 DEBUG --- status code --- 06:46:32 DEBUG "data": { 06:46:32 INFO [http_cmd]: http status code OK 06:46:32 DEBUG "data": { 06:46:32 DEBUG "resultType": "vector", 06:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:32 DEBUG "resultType": "vector", 06:46:32 DEBUG --- status code --- 06:46:32 DEBUG "resultType": "vector", 06:46:32 DEBUG "result": [ 06:46:32 DEBUG --- http response --- 06:46:32 DEBUG "result": [ 06:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:32 DEBUG "result": [ 06:46:32 DEBUG { 06:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:46:32 DEBUG { 06:46:32 DEBUG --- http response --- 06:46:32 DEBUG { 06:46:32 DEBUG "metric": { 06:46:32 DEBUG ----- output ----- 06:46:32 DEBUG "metric": { 06:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 06:46:32 DEBUG "metric": { 06:46:32 DEBUG "pod": "ds-idrepo-0" 06:46:32 DEBUG { 06:46:32 DEBUG "pod": "ds-idrepo-0" 06:46:32 DEBUG ----- output ----- 06:46:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:46:32 DEBUG }, 06:46:32 DEBUG "status": "success", 06:46:32 DEBUG }, 06:46:32 DEBUG { 06:46:32 DEBUG }, 06:46:32 DEBUG "value": [ 06:46:32 DEBUG "data": { 06:46:32 DEBUG "value": [ 06:46:32 DEBUG "status": "success", 06:46:32 DEBUG "value": [ 06:46:32 DEBUG 06:46:32 DEBUG "resultType": "vector", 06:46:32 DEBUG 06:46:32 DEBUG "data": { 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG "result": [ 06:46:32 DEBUG 06:46:32 DEBUG "resultType": "vector", 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG { 06:46:32 DEBUG 06:46:32 DEBUG "result": [ 06:46:32 DEBUG 06:46:32 DEBUG [...] 06:46:32 DEBUG "metric": { 06:46:32 DEBUG [...] 06:46:32 DEBUG { 06:46:32 DEBUG [...] 06:46:32 DEBUG 06:46:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:46:32 DEBUG 06:46:32 DEBUG "metric": { 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG }, 06:46:32 DEBUG 06:46:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG "value": [ 06:46:32 DEBUG 06:46:32 DEBUG }, 06:46:32 DEBUG 06:46:32 DEBUG "pod": "ds-cts-2" 06:46:32 DEBUG 06:46:32 DEBUG "pod": "ds-cts-2" 06:46:32 DEBUG "value": [ 06:46:32 DEBUG "pod": "ds-cts-0" 06:46:32 DEBUG }, 06:46:32 DEBUG 06:46:32 DEBUG }, 06:46:32 DEBUG 06:46:32 DEBUG }, 06:46:32 DEBUG "value": [ 06:46:32 DEBUG 06:46:32 DEBUG "value": [ 06:46:32 DEBUG 06:46:32 DEBUG "value": [ 06:46:32 DEBUG 1712727992, 06:46:32 DEBUG [...] 06:46:32 DEBUG 1712727992, 06:46:32 DEBUG 06:46:32 DEBUG 1712727992, 06:46:32 DEBUG "0" 06:46:32 DEBUG 06:46:32 DEBUG "29.2" 06:46:32 DEBUG [...] 06:46:32 DEBUG "0" 06:46:32 DEBUG ] 06:46:32 DEBUG 06:46:32 DEBUG ] 06:46:32 DEBUG 06:46:32 DEBUG ] 06:46:32 DEBUG } 06:46:32 DEBUG 06:46:32 DEBUG } 06:46:32 DEBUG 06:46:32 DEBUG } 06:46:32 DEBUG ] 06:46:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:46:32 DEBUG ] 06:46:32 DEBUG 06:46:32 DEBUG ] 06:46:32 DEBUG } 06:46:32 DEBUG }, 06:46:32 DEBUG } 06:46:32 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:46:32 DEBUG } 06:46:32 DEBUG } 06:46:32 DEBUG "value": [ 06:46:32 DEBUG } 06:46:32 DEBUG }, 06:46:32 DEBUG } 06:46:32 DEBUG 1712727992, 06:46:32 DEBUG "value": [ 06:46:32 DEBUG "259300" 06:46:32 DEBUG 1712727992, 06:46:32 DEBUG ] 06:46:32 DEBUG "0" 06:46:32 DEBUG } 06:46:32 DEBUG ] 06:46:32 DEBUG ] 06:46:32 DEBUG } 06:46:32 DEBUG } 06:46:32 DEBUG ] 06:46:32 DEBUG } 06:46:32 DEBUG } 06:46:32 DEBUG } 06:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712727992" 06:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712727992" 06:46:32 INFO [http_cmd]: http status code OK 06:46:32 DEBUG --- status code --- 06:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:32 DEBUG --- http response --- 06:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:46:32 DEBUG ----- output ----- 06:46:32 DEBUG { 06:46:32 DEBUG "status": "success", 06:46:32 DEBUG "data": { 06:46:32 DEBUG "resultType": "vector", 06:46:32 DEBUG "result": [ 06:46:32 DEBUG { 06:46:32 DEBUG "metric": { 06:46:32 DEBUG "pod": "ds-idrepo-0" 06:46:32 DEBUG }, 06:46:32 DEBUG "value": [ 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG [...] 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG "pod": "ds-cts-2" 06:46:32 DEBUG }, 06:46:32 DEBUG "value": [ 06:46:32 DEBUG 1712727992, 06:46:32 DEBUG "0" 06:46:32 DEBUG ] 06:46:32 DEBUG } 06:46:32 DEBUG ] 06:46:32 DEBUG } 06:46:32 DEBUG } 06:46:32 INFO [http_cmd]: http status code OK 06:46:32 DEBUG --- status code --- 06:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:32 DEBUG --- http response --- 06:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:46:32 DEBUG ----- output ----- 06:46:32 DEBUG { 06:46:32 DEBUG "status": "success", 06:46:32 DEBUG "data": { 06:46:32 DEBUG "resultType": "vector", 06:46:32 DEBUG "result": [ 06:46:32 DEBUG { 06:46:32 DEBUG "metric": { 06:46:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:46:32 DEBUG }, 06:46:32 DEBUG "value": [ 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG [...] 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:46:32 DEBUG }, 06:46:32 DEBUG "value": [ 06:46:32 DEBUG 1712727992, 06:46:32 DEBUG "4.366037184686735" 06:46:32 DEBUG ] 06:46:32 DEBUG } 06:46:32 DEBUG ] 06:46:32 DEBUG } 06:46:32 DEBUG } 06:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727992" 06:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727992" 06:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727992" 06:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712727992" 06:46:32 INFO [http_cmd]: http status code OK 06:46:32 DEBUG --- status code --- 06:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:32 DEBUG --- http response --- 06:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:46:32 DEBUG ----- output ----- 06:46:32 DEBUG { 06:46:32 DEBUG "status": "success", 06:46:32 DEBUG "data": { 06:46:32 DEBUG "resultType": "vector", 06:46:32 DEBUG "result": [ 06:46:32 DEBUG { 06:46:32 DEBUG "metric": { 06:46:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:46:32 DEBUG }, 06:46:32 DEBUG "value": [ 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG [...] 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:46:32 DEBUG }, 06:46:32 DEBUG "value": [ 06:46:32 DEBUG 1712727992, 06:46:32 DEBUG "134.16666666666666" 06:46:32 DEBUG ] 06:46:32 DEBUG } 06:46:32 DEBUG ] 06:46:32 DEBUG } 06:46:32 DEBUG } 06:46:32 INFO [http_cmd]: http status code OK 06:46:32 DEBUG --- status code --- 06:46:32 INFO [http_cmd]: http status code OK 06:46:32 INFO [http_cmd]: http status code OK 06:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:32 DEBUG --- status code --- 06:46:32 DEBUG --- status code --- 06:46:32 DEBUG --- http response --- 06:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:46:32 DEBUG --- http response --- 06:46:32 DEBUG --- http response --- 06:46:32 DEBUG ----- output ----- 06:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:46:32 DEBUG { 06:46:32 DEBUG ----- output ----- 06:46:32 DEBUG ----- output ----- 06:46:32 DEBUG "status": "success", 06:46:32 DEBUG { 06:46:32 DEBUG { 06:46:32 DEBUG "data": { 06:46:32 DEBUG "status": "success", 06:46:32 DEBUG "status": "success", 06:46:32 DEBUG "resultType": "vector", 06:46:32 DEBUG "data": { 06:46:32 DEBUG "data": { 06:46:32 DEBUG "result": [ 06:46:32 DEBUG "resultType": "vector", 06:46:32 DEBUG "resultType": "vector", 06:46:32 DEBUG { 06:46:32 DEBUG "result": [ 06:46:32 DEBUG "result": [ 06:46:32 DEBUG "metric": { 06:46:32 DEBUG { 06:46:32 DEBUG { 06:46:32 DEBUG "pod": "ds-idrepo-2" 06:46:32 DEBUG "metric": { 06:46:32 DEBUG "metric": { 06:46:32 DEBUG }, 06:46:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:46:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:46:32 DEBUG "value": [ 06:46:32 DEBUG }, 06:46:32 DEBUG }, 06:46:32 DEBUG 06:46:32 DEBUG "value": [ 06:46:32 DEBUG "value": [ 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG [...] 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG [...] 06:46:32 DEBUG [...] 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG "pod": "ds-cts-2" 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG }, 06:46:32 DEBUG "pod": "ds-cts-0" 06:46:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:46:32 DEBUG "value": [ 06:46:32 DEBUG }, 06:46:32 DEBUG }, 06:46:32 DEBUG 1712727992, 06:46:32 DEBUG "value": [ 06:46:32 DEBUG "value": [ 06:46:32 DEBUG "0" 06:46:32 DEBUG 1712727992, 06:46:32 DEBUG 1712727992, 06:46:32 DEBUG ] 06:46:32 DEBUG "0" 06:46:32 DEBUG "0.052" 06:46:32 DEBUG } 06:46:32 DEBUG ] 06:46:32 DEBUG ] 06:46:32 DEBUG ] 06:46:32 DEBUG } 06:46:32 DEBUG } 06:46:32 DEBUG } 06:46:32 DEBUG ] 06:46:32 DEBUG ] 06:46:32 DEBUG } 06:46:32 DEBUG } 06:46:32 DEBUG } 06:46:32 DEBUG } 06:46:32 DEBUG } 06:46:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727992" 06:46:32 INFO [http_cmd]: http status code OK 06:46:32 DEBUG --- status code --- 06:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:32 DEBUG --- http response --- 06:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:46:32 DEBUG ----- output ----- 06:46:32 DEBUG { 06:46:32 DEBUG "status": "success", 06:46:32 DEBUG "data": { 06:46:32 DEBUG "resultType": "vector", 06:46:32 DEBUG "result": [ 06:46:32 DEBUG { 06:46:32 DEBUG "metric": { 06:46:32 DEBUG "pod": "ds-idrepo-0" 06:46:32 DEBUG }, 06:46:32 DEBUG "value": [ 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG [...] 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG 06:46:32 DEBUG "pod": "ds-idrepo-1" 06:46:32 DEBUG }, 06:46:32 DEBUG "value": [ 06:46:32 DEBUG 1712727992, 06:46:32 DEBUG "0" 06:46:32 DEBUG ] 06:46:32 DEBUG } 06:46:32 DEBUG ] 06:46:32 DEBUG } 06:46:32 DEBUG } 06:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727993" 06:46:33 INFO [http_cmd]: http status code OK 06:46:33 DEBUG --- status code --- 06:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:33 DEBUG --- http response --- 06:46:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 06:46:33 DEBUG ----- output ----- 06:46:33 DEBUG { 06:46:33 DEBUG "status": "success", 06:46:33 DEBUG "data": { 06:46:33 DEBUG "resultType": "vector", 06:46:33 DEBUG "result": [ 06:46:33 DEBUG { 06:46:33 DEBUG "metric": { 06:46:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:46:33 DEBUG }, 06:46:33 DEBUG "value": [ 06:46:33 DEBUG 06:46:33 DEBUG 06:46:33 DEBUG 06:46:33 DEBUG [...] 06:46:33 DEBUG 06:46:33 DEBUG 06:46:33 DEBUG 06:46:33 DEBUG "pod": "am-654c4fcbcf-9scrj" 06:46:33 DEBUG }, 06:46:33 DEBUG "value": [ 06:46:33 DEBUG 1712727993, 06:46:33 DEBUG "1402.6333333333332" 06:46:33 DEBUG ] 06:46:33 DEBUG } 06:46:33 DEBUG ] 06:46:33 DEBUG } 06:46:33 DEBUG } 06:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727993" 06:46:33 INFO [http_cmd]: http status code OK 06:46:33 DEBUG --- status code --- 06:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:33 DEBUG --- http response --- 06:46:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:46:33 DEBUG ----- output ----- 06:46:33 DEBUG { 06:46:33 DEBUG "status": "success", 06:46:33 DEBUG "data": { 06:46:33 DEBUG "resultType": "vector", 06:46:33 DEBUG "result": [ 06:46:33 DEBUG { 06:46:33 DEBUG "metric": { 06:46:33 DEBUG "pod": "ds-idrepo-0" 06:46:33 DEBUG }, 06:46:33 DEBUG "value": [ 06:46:33 DEBUG 06:46:33 DEBUG 06:46:33 DEBUG 06:46:33 DEBUG [...] 06:46:33 DEBUG 06:46:33 DEBUG 06:46:33 DEBUG 06:46:33 DEBUG "pod": "ds-idrepo-1" 06:46:33 DEBUG }, 06:46:33 DEBUG "value": [ 06:46:33 DEBUG 1712727993, 06:46:33 DEBUG "0" 06:46:33 DEBUG ] 06:46:33 DEBUG } 06:46:33 DEBUG ] 06:46:33 DEBUG } 06:46:33 DEBUG } 06:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712727993" 06:46:33 INFO [http_cmd]: http status code OK 06:46:33 DEBUG --- status code --- 06:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:33 DEBUG --- http response --- 06:46:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:46:33 DEBUG ----- output ----- 06:46:33 DEBUG { 06:46:33 DEBUG "status": "success", 06:46:33 DEBUG "data": { 06:46:33 DEBUG "resultType": "vector", 06:46:33 DEBUG "result": [ 06:46:33 DEBUG { 06:46:33 DEBUG "metric": { 06:46:33 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:46:33 DEBUG }, 06:46:33 DEBUG "value": [ 06:46:33 DEBUG 06:46:33 DEBUG 06:46:33 DEBUG 06:46:33 DEBUG [...] 06:46:33 DEBUG 06:46:33 DEBUG 06:46:33 DEBUG 06:46:33 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:46:33 DEBUG }, 06:46:33 DEBUG "value": [ 06:46:33 DEBUG 1712727993, 06:46:33 DEBUG "33616.15937154142" 06:46:33 DEBUG ] 06:46:33 DEBUG } 06:46:33 DEBUG ] 06:46:33 DEBUG } 06:46:33 DEBUG } 06:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712727993" 06:46:33 INFO [http_cmd]: http status code OK 06:46:33 DEBUG --- status code --- 06:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:33 DEBUG --- http response --- 06:46:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:46:33 DEBUG ----- output ----- 06:46:33 DEBUG { 06:46:33 DEBUG "status": "success", 06:46:33 DEBUG "data": { 06:46:33 DEBUG "resultType": "vector", 06:46:33 DEBUG "result": [ 06:46:33 DEBUG { 06:46:33 DEBUG "metric": { 06:46:33 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:46:33 DEBUG }, 06:46:33 DEBUG "value": [ 06:46:33 DEBUG 06:46:33 DEBUG 06:46:33 DEBUG 06:46:33 DEBUG [...] 06:46:33 DEBUG 06:46:33 DEBUG 06:46:33 DEBUG 06:46:33 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:46:33 DEBUG }, 06:46:33 DEBUG "value": [ 06:46:33 DEBUG 1712727993, 06:46:33 DEBUG "2487132160" 06:46:33 DEBUG ] 06:46:33 DEBUG } 06:46:33 DEBUG ] 06:46:33 DEBUG } 06:46:33 DEBUG } 06:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727993" 06:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712727993" 06:46:33 INFO [http_cmd]: http status code OK 06:46:33 DEBUG --- status code --- 06:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:33 DEBUG --- http response --- 06:46:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:46:33 DEBUG ----- output ----- 06:46:33 DEBUG { 06:46:33 DEBUG "status": "success", 06:46:33 DEBUG "data": { 06:46:33 DEBUG "resultType": "vector", 06:46:33 DEBUG "result": [ 06:46:33 DEBUG { 06:46:33 DEBUG "metric": { 06:46:33 DEBUG "instance": "10.142.0.25:9100" 06:46:33 DEBUG }, 06:46:33 DEBUG "value": [ 06:46:33 DEBUG 06:46:33 DEBUG 06:46:33 DEBUG 06:46:33 DEBUG [...] 06:46:33 DEBUG 06:46:33 DEBUG 06:46:33 DEBUG 06:46:33 DEBUG "instance": "10.142.0.77:9100" 06:46:33 DEBUG }, 06:46:33 DEBUG "value": [ 06:46:33 DEBUG 1712727993, 06:46:33 DEBUG "36879510.3" 06:46:33 DEBUG ] 06:46:33 DEBUG } 06:46:33 DEBUG ] 06:46:33 DEBUG } 06:46:33 DEBUG } 06:46:33 INFO [http_cmd]: http status code OK 06:46:33 DEBUG --- status code --- 06:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:33 DEBUG --- http response --- 06:46:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:46:33 DEBUG ----- output ----- 06:46:33 DEBUG { 06:46:33 DEBUG "status": "success", 06:46:33 DEBUG "data": { 06:46:33 DEBUG "resultType": "vector", 06:46:33 DEBUG "result": [ 06:46:33 DEBUG { 06:46:33 DEBUG "metric": { 06:46:33 DEBUG "instance": "10.142.0.25:9100" 06:46:33 DEBUG }, 06:46:33 DEBUG "value": [ 06:46:33 DEBUG 06:46:33 DEBUG 06:46:33 DEBUG 06:46:33 DEBUG [...] 06:46:33 DEBUG 06:46:33 DEBUG 06:46:33 DEBUG 06:46:33 DEBUG "instance": "10.142.0.77:9100" 06:46:33 DEBUG }, 06:46:33 DEBUG "value": [ 06:46:33 DEBUG 1712727993, 06:46:33 DEBUG "0.05099999999999981" 06:46:33 DEBUG ] 06:46:33 DEBUG } 06:46:33 DEBUG ] 06:46:33 DEBUG } 06:46:33 DEBUG } 06:46:54 INFO 06:46:54 INFO [loop_until]: kubectl --namespace=xlou top pods 06:46:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:46:54 INFO [loop_until]: OK (rc = 0) 06:46:54 DEBUG --- stdout --- 06:46:54 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 3511m 9020Mi am-654c4fcbcf-9scrj 3282m 8465Mi am-654c4fcbcf-v5qb5 1882m 8493Mi ds-cts-0 5997m 13113Mi ds-cts-1 4178m 13382Mi ds-cts-2 4424m 13393Mi ds-idrepo-0 3997m 5634Mi ds-idrepo-1 125m 5672Mi ds-idrepo-2 37m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1191Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 7m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 47m 1189Mi 06:46:54 DEBUG --- stderr --- 06:46:54 DEBUG 06:47:02 INFO 06:47:02 INFO [loop_until]: kubectl --namespace=xlou top node 06:47:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:47:03 INFO [loop_until]: OK (rc = 0) 06:47:03 DEBUG --- stdout --- 06:47:03 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 96m 0% 2307Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 3351m 21% 9366Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 77m 0% 3259Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 623m 3% 6525Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 1990m 12% 9347Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 5837m 36% 9833Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 76m 0% 857Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 2623m 16% 6323Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 4017m 25% 13836Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 3791m 23% 14096Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 141m 0% 6398Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 4746m 29% 14109Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 130m 0% 6360Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 792m 4% 4812Mi 8% 06:47:03 DEBUG --- stderr --- 06:47:03 DEBUG 06:47:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728050" 06:47:30 INFO [http_cmd]: http status code OK 06:47:30 DEBUG --- status code --- 06:47:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:30 DEBUG --- http response --- 06:47:30 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:47:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728050" 06:47:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712728050" 06:47:30 INFO [http_cmd]: http status code OK 06:47:30 DEBUG --- status code --- 06:47:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:30 DEBUG --- http response --- 06:47:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:47:30 DEBUG ----- output ----- 06:47:30 DEBUG { 06:47:30 DEBUG "status": "success", 06:47:30 DEBUG "data": { 06:47:30 DEBUG "resultType": "vector", 06:47:30 DEBUG "result": [ 06:47:30 DEBUG { 06:47:30 DEBUG "metric": { 06:47:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:47:30 DEBUG }, 06:47:30 DEBUG "value": [ 06:47:30 DEBUG 06:47:30 DEBUG 06:47:30 DEBUG 06:47:30 DEBUG [...] 06:47:30 DEBUG 06:47:30 DEBUG 06:47:30 DEBUG 06:47:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:47:30 DEBUG }, 06:47:30 DEBUG "value": [ 06:47:30 DEBUG 1712728050, 06:47:30 DEBUG "0" 06:47:30 DEBUG ] 06:47:30 DEBUG } 06:47:30 DEBUG ] 06:47:30 DEBUG } 06:47:30 DEBUG } 06:47:30 ERROR [http_cmd]: ERROR 06:47:30 ERROR --- status code --- 06:47:30 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:47:30 ERROR --- http response --- 06:47:30 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:47:30 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728051" 06:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728051" 06:47:31 INFO [http_cmd]: http status code OK 06:47:31 DEBUG --- status code --- 06:47:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:31 DEBUG --- http response --- 06:47:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:47:31 DEBUG ----- output ----- 06:47:31 DEBUG { 06:47:31 DEBUG "status": "success", 06:47:31 DEBUG "data": { 06:47:31 DEBUG "resultType": "vector", 06:47:31 DEBUG "result": [ 06:47:31 DEBUG { 06:47:31 DEBUG "metric": { 06:47:31 DEBUG "instance": "10.142.0.25:9100" 06:47:31 DEBUG }, 06:47:31 DEBUG "value": [ 06:47:31 DEBUG 06:47:31 DEBUG 06:47:31 DEBUG 06:47:31 DEBUG [...] 06:47:31 DEBUG 06:47:31 DEBUG 06:47:31 DEBUG 06:47:31 DEBUG "instance": "10.142.0.77:9100" 06:47:31 DEBUG }, 06:47:31 DEBUG "value": [ 06:47:31 DEBUG 1712728051, 06:47:31 DEBUG "9647526.666666668" 06:47:31 DEBUG ] 06:47:31 DEBUG } 06:47:31 DEBUG ] 06:47:31 DEBUG } 06:47:31 DEBUG } 06:47:31 INFO [http_cmd]: http status code OK 06:47:31 DEBUG --- status code --- 06:47:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:31 DEBUG --- http response --- 06:47:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:47:31 DEBUG ----- output ----- 06:47:31 DEBUG { 06:47:31 DEBUG "status": "success", 06:47:31 DEBUG "data": { 06:47:31 DEBUG "resultType": "vector", 06:47:31 DEBUG "result": [ 06:47:31 DEBUG { 06:47:31 DEBUG "metric": { 06:47:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:47:31 DEBUG }, 06:47:31 DEBUG "value": [ 06:47:31 DEBUG 06:47:31 DEBUG 06:47:31 DEBUG 06:47:31 DEBUG [...] 06:47:31 DEBUG 06:47:31 DEBUG 06:47:31 DEBUG 06:47:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:47:31 DEBUG }, 06:47:31 DEBUG "value": [ 06:47:31 DEBUG 1712728051, 06:47:31 DEBUG "0" 06:47:31 DEBUG ] 06:47:31 DEBUG } 06:47:31 DEBUG ] 06:47:31 DEBUG } 06:47:31 DEBUG } 06:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728051" 06:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728051" 06:47:31 INFO [http_cmd]: http status code OK 06:47:31 DEBUG --- status code --- 06:47:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:31 DEBUG --- http response --- 06:47:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:47:31 DEBUG ----- output ----- 06:47:31 DEBUG { 06:47:31 DEBUG "status": "success", 06:47:31 DEBUG "data": { 06:47:31 DEBUG "resultType": "vector", 06:47:31 DEBUG "result": [ 06:47:31 DEBUG { 06:47:31 DEBUG "metric": { 06:47:31 DEBUG "pod": "ds-idrepo-0" 06:47:31 DEBUG }, 06:47:31 DEBUG "value": [ 06:47:31 DEBUG 06:47:31 DEBUG 06:47:31 DEBUG 06:47:31 DEBUG [...] 06:47:31 DEBUG 06:47:31 DEBUG 06:47:31 DEBUG 06:47:31 DEBUG "pod": "ds-idrepo-1" 06:47:31 DEBUG }, 06:47:31 DEBUG "value": [ 06:47:31 DEBUG 1712728051, 06:47:31 DEBUG "0" 06:47:31 DEBUG ] 06:47:31 DEBUG } 06:47:31 DEBUG ] 06:47:31 DEBUG } 06:47:31 DEBUG } 06:47:31 INFO [http_cmd]: http status code OK 06:47:31 DEBUG --- status code --- 06:47:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:31 DEBUG --- http response --- 06:47:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:47:31 DEBUG ----- output ----- 06:47:31 DEBUG { 06:47:31 DEBUG "status": "success", 06:47:31 DEBUG "data": { 06:47:31 DEBUG "resultType": "vector", 06:47:31 DEBUG "result": [ 06:47:31 DEBUG { 06:47:31 DEBUG "metric": { 06:47:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:47:31 DEBUG }, 06:47:31 DEBUG "value": [ 06:47:31 DEBUG 06:47:31 DEBUG 06:47:31 DEBUG 06:47:31 DEBUG [...] 06:47:31 DEBUG 06:47:31 DEBUG 06:47:31 DEBUG 06:47:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:47:31 DEBUG }, 06:47:31 DEBUG "value": [ 06:47:31 DEBUG 1712728051, 06:47:31 DEBUG "0" 06:47:31 DEBUG ] 06:47:31 DEBUG } 06:47:31 DEBUG ] 06:47:31 DEBUG } 06:47:31 DEBUG } 06:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728052" 06:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728052" 06:47:32 INFO [http_cmd]: http status code OK 06:47:32 DEBUG --- status code --- 06:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:32 DEBUG --- http response --- 06:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:47:32 DEBUG ----- output ----- 06:47:32 DEBUG { 06:47:32 DEBUG "status": "success", 06:47:32 DEBUG "data": { 06:47:32 DEBUG "resultType": "vector", 06:47:32 DEBUG "result": [ 06:47:32 DEBUG { 06:47:32 DEBUG "metric": { 06:47:32 DEBUG "instance": "10.142.0.25:9100" 06:47:32 DEBUG }, 06:47:32 DEBUG "value": [ 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG [...] 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG "instance": "10.142.0.77:9100" 06:47:32 DEBUG }, 06:47:32 DEBUG "value": [ 06:47:32 DEBUG 1712728052, 06:47:32 DEBUG "436803986432" 06:47:32 DEBUG ] 06:47:32 DEBUG } 06:47:32 DEBUG ] 06:47:32 DEBUG } 06:47:32 DEBUG } 06:47:32 INFO [http_cmd]: http status code OK 06:47:32 DEBUG --- status code --- 06:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:32 DEBUG --- http response --- 06:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:47:32 DEBUG ----- output ----- 06:47:32 DEBUG { 06:47:32 DEBUG "status": "success", 06:47:32 DEBUG "data": { 06:47:32 DEBUG "resultType": "vector", 06:47:32 DEBUG "result": [ 06:47:32 DEBUG { 06:47:32 DEBUG "metric": { 06:47:32 DEBUG "instance": "10.142.0.25:9100" 06:47:32 DEBUG }, 06:47:32 DEBUG "value": [ 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG [...] 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG "instance": "10.142.0.77:9100" 06:47:32 DEBUG }, 06:47:32 DEBUG "value": [ 06:47:32 DEBUG 1712728052, 06:47:32 DEBUG "0.23843333333333425" 06:47:32 DEBUG ] 06:47:32 DEBUG } 06:47:32 DEBUG ] 06:47:32 DEBUG } 06:47:32 DEBUG } 06:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728052" 06:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728052" 06:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728052" 06:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728052" 06:47:32 INFO [http_cmd]: http status code OK 06:47:32 DEBUG --- status code --- 06:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:32 DEBUG --- http response --- 06:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:47:32 DEBUG ----- output ----- 06:47:32 DEBUG { 06:47:32 DEBUG "status": "success", 06:47:32 DEBUG "data": { 06:47:32 DEBUG "resultType": "vector", 06:47:32 DEBUG "result": [ 06:47:32 DEBUG { 06:47:32 DEBUG "metric": { 06:47:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:47:32 DEBUG }, 06:47:32 DEBUG "value": [ 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG [...] 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG "pod": "ds-idrepo-0" 06:47:32 DEBUG }, 06:47:32 DEBUG "value": [ 06:47:32 DEBUG 1712728052, 06:47:32 DEBUG "0" 06:47:32 DEBUG ] 06:47:32 DEBUG } 06:47:32 DEBUG ] 06:47:32 DEBUG } 06:47:32 DEBUG } 06:47:32 INFO [http_cmd]: http status code OK 06:47:32 INFO [http_cmd]: http status code OK 06:47:32 DEBUG --- status code --- 06:47:32 DEBUG --- status code --- 06:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:32 DEBUG --- http response --- 06:47:32 DEBUG --- http response --- 06:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:47:32 DEBUG ----- output ----- 06:47:32 DEBUG ----- output ----- 06:47:32 DEBUG { 06:47:32 DEBUG { 06:47:32 DEBUG "status": "success", 06:47:32 DEBUG "status": "success", 06:47:32 DEBUG "data": { 06:47:32 DEBUG "data": { 06:47:32 DEBUG "resultType": "vector", 06:47:32 DEBUG "resultType": "vector", 06:47:32 DEBUG "result": [ 06:47:32 DEBUG "result": [ 06:47:32 DEBUG { 06:47:32 DEBUG { 06:47:32 DEBUG "metric": { 06:47:32 DEBUG "metric": { 06:47:32 DEBUG "pod": "ds-idrepo-0" 06:47:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:47:32 DEBUG }, 06:47:32 DEBUG }, 06:47:32 DEBUG "value": [ 06:47:32 DEBUG "value": [ 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 INFO [http_cmd]: http status code OK 06:47:32 DEBUG [...] 06:47:32 DEBUG [...] 06:47:32 DEBUG --- status code --- 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG --- http response --- 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:47:32 DEBUG "pod": "ds-cts-2" 06:47:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:47:32 DEBUG ----- output ----- 06:47:32 DEBUG }, 06:47:32 DEBUG }, 06:47:32 DEBUG { 06:47:32 DEBUG "value": [ 06:47:32 DEBUG "value": [ 06:47:32 DEBUG "status": "success", 06:47:32 DEBUG 1712728052, 06:47:32 DEBUG 1712728052, 06:47:32 DEBUG "data": { 06:47:32 DEBUG "29.933333333333334" 06:47:32 DEBUG "197756" 06:47:32 DEBUG "resultType": "vector", 06:47:32 DEBUG ] 06:47:32 DEBUG ] 06:47:32 DEBUG "result": [ 06:47:32 DEBUG } 06:47:32 DEBUG } 06:47:32 DEBUG { 06:47:32 DEBUG ] 06:47:32 DEBUG ] 06:47:32 DEBUG "metric": { 06:47:32 DEBUG } 06:47:32 DEBUG } 06:47:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:47:32 DEBUG } 06:47:32 DEBUG } 06:47:32 DEBUG }, 06:47:32 DEBUG "value": [ 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG [...] 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG "pod": "ds-idrepo-0" 06:47:32 DEBUG }, 06:47:32 DEBUG "value": [ 06:47:32 DEBUG 1712728052, 06:47:32 DEBUG "216989714.76980487" 06:47:32 DEBUG ] 06:47:32 DEBUG } 06:47:32 DEBUG ] 06:47:32 DEBUG } 06:47:32 DEBUG } 06:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728052" 06:47:32 INFO [http_cmd]: http status code OK 06:47:32 DEBUG --- status code --- 06:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:32 DEBUG --- http response --- 06:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:47:32 DEBUG ----- output ----- 06:47:32 DEBUG { 06:47:32 DEBUG "status": "success", 06:47:32 DEBUG "data": { 06:47:32 DEBUG "resultType": "vector", 06:47:32 DEBUG "result": [ 06:47:32 DEBUG { 06:47:32 DEBUG "metric": { 06:47:32 DEBUG "pod": "ds-idrepo-0" 06:47:32 DEBUG }, 06:47:32 DEBUG "value": [ 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG [...] 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG "pod": "ds-cts-2" 06:47:32 DEBUG }, 06:47:32 DEBUG "value": [ 06:47:32 DEBUG 1712728052, 06:47:32 DEBUG "0" 06:47:32 DEBUG ] 06:47:32 DEBUG } 06:47:32 DEBUG ] 06:47:32 DEBUG } 06:47:32 DEBUG } 06:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712728052" 06:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712728052" 06:47:32 INFO [http_cmd]: http status code OK 06:47:32 DEBUG --- status code --- 06:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:32 DEBUG --- http response --- 06:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:47:32 DEBUG ----- output ----- 06:47:32 DEBUG { 06:47:32 DEBUG "status": "success", 06:47:32 DEBUG "data": { 06:47:32 DEBUG "resultType": "vector", 06:47:32 DEBUG "result": [ 06:47:32 DEBUG { 06:47:32 DEBUG "metric": { 06:47:32 DEBUG "pod": "ds-idrepo-0" 06:47:32 DEBUG }, 06:47:32 DEBUG "value": [ 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG [...] 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG "pod": "ds-cts-2" 06:47:32 DEBUG }, 06:47:32 DEBUG "value": [ 06:47:32 DEBUG 1712728052, 06:47:32 DEBUG "0" 06:47:32 DEBUG ] 06:47:32 DEBUG } 06:47:32 DEBUG ] 06:47:32 DEBUG } 06:47:32 DEBUG } 06:47:32 INFO [http_cmd]: http status code OK 06:47:32 DEBUG --- status code --- 06:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:32 DEBUG --- http response --- 06:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:47:32 DEBUG ----- output ----- 06:47:32 DEBUG { 06:47:32 DEBUG "status": "success", 06:47:32 DEBUG "data": { 06:47:32 DEBUG "resultType": "vector", 06:47:32 DEBUG "result": [ 06:47:32 DEBUG { 06:47:32 DEBUG "metric": { 06:47:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:47:32 DEBUG }, 06:47:32 DEBUG "value": [ 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG [...] 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:47:32 DEBUG }, 06:47:32 DEBUG "value": [ 06:47:32 DEBUG 1712728052, 06:47:32 DEBUG "0.027686478301104288" 06:47:32 DEBUG ] 06:47:32 DEBUG } 06:47:32 DEBUG ] 06:47:32 DEBUG } 06:47:32 DEBUG } 06:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728052" 06:47:32 INFO [http_cmd]: http status code OK 06:47:32 DEBUG --- status code --- 06:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:32 DEBUG --- http response --- 06:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:47:32 DEBUG ----- output ----- 06:47:32 DEBUG { 06:47:32 DEBUG "status": "success", 06:47:32 DEBUG "data": { 06:47:32 DEBUG "resultType": "vector", 06:47:32 DEBUG "result": [ 06:47:32 DEBUG { 06:47:32 DEBUG "metric": { 06:47:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:47:32 DEBUG }, 06:47:32 DEBUG "value": [ 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG [...] 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:47:32 DEBUG }, 06:47:32 DEBUG "value": [ 06:47:32 DEBUG 1712728052, 06:47:32 DEBUG "0" 06:47:32 DEBUG ] 06:47:32 DEBUG } 06:47:32 DEBUG ] 06:47:32 DEBUG } 06:47:32 DEBUG } 06:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712728052" 06:47:32 INFO [http_cmd]: http status code OK 06:47:32 DEBUG --- status code --- 06:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:32 DEBUG --- http response --- 06:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:47:32 DEBUG ----- output ----- 06:47:32 DEBUG { 06:47:32 DEBUG "status": "success", 06:47:32 DEBUG "data": { 06:47:32 DEBUG "resultType": "vector", 06:47:32 DEBUG "result": [ 06:47:32 DEBUG { 06:47:32 DEBUG "metric": { 06:47:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:47:32 DEBUG }, 06:47:32 DEBUG "value": [ 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG [...] 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:47:32 DEBUG }, 06:47:32 DEBUG "value": [ 06:47:32 DEBUG 1712728052, 06:47:32 DEBUG "0.052" 06:47:32 DEBUG ] 06:47:32 DEBUG } 06:47:32 DEBUG ] 06:47:32 DEBUG } 06:47:32 DEBUG } 06:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728052" 06:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728052" 06:47:32 INFO [http_cmd]: http status code OK 06:47:32 INFO [http_cmd]: http status code OK 06:47:32 DEBUG --- status code --- 06:47:32 DEBUG --- status code --- 06:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:32 DEBUG --- http response --- 06:47:32 DEBUG --- http response --- 06:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:47:32 DEBUG ----- output ----- 06:47:32 DEBUG ----- output ----- 06:47:32 DEBUG { 06:47:32 DEBUG { 06:47:32 DEBUG "status": "success", 06:47:32 DEBUG "status": "success", 06:47:32 DEBUG "data": { 06:47:32 DEBUG "data": { 06:47:32 DEBUG "resultType": "vector", 06:47:32 DEBUG "resultType": "vector", 06:47:32 DEBUG "result": [ 06:47:32 DEBUG "result": [ 06:47:32 DEBUG { 06:47:32 DEBUG { 06:47:32 DEBUG "metric": { 06:47:32 DEBUG "metric": { 06:47:32 DEBUG "pod": "ds-idrepo-2" 06:47:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:47:32 DEBUG }, 06:47:32 DEBUG }, 06:47:32 DEBUG "value": [ 06:47:32 DEBUG "value": [ 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG [...] 06:47:32 DEBUG [...] 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG "pod": "ds-cts-2" 06:47:32 DEBUG "pod": "ds-idrepo-0" 06:47:32 DEBUG }, 06:47:32 DEBUG }, 06:47:32 DEBUG "value": [ 06:47:32 DEBUG "value": [ 06:47:32 DEBUG 1712728052, 06:47:32 DEBUG 1712728052, 06:47:32 DEBUG "0.00013333333333333334" 06:47:32 DEBUG "0.9609958221586514" 06:47:32 DEBUG ] 06:47:32 DEBUG ] 06:47:32 DEBUG } 06:47:32 DEBUG } 06:47:32 DEBUG ] 06:47:32 DEBUG ] 06:47:32 DEBUG } 06:47:32 DEBUG } 06:47:32 DEBUG } 06:47:32 DEBUG } 06:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728052" 06:47:32 INFO [http_cmd]: http status code OK 06:47:32 DEBUG --- status code --- 06:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:32 DEBUG --- http response --- 06:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:47:32 DEBUG ----- output ----- 06:47:32 DEBUG { 06:47:32 DEBUG "status": "success", 06:47:32 DEBUG "data": { 06:47:32 DEBUG "resultType": "vector", 06:47:32 DEBUG "result": [ 06:47:32 DEBUG { 06:47:32 DEBUG "metric": { 06:47:32 DEBUG "pod": "ds-idrepo-0" 06:47:32 DEBUG }, 06:47:32 DEBUG "value": [ 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG [...] 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG 06:47:32 DEBUG "pod": "ds-idrepo-1" 06:47:32 DEBUG }, 06:47:32 DEBUG "value": [ 06:47:32 DEBUG 1712728052, 06:47:32 DEBUG "0" 06:47:32 DEBUG ] 06:47:32 DEBUG } 06:47:32 DEBUG ] 06:47:32 DEBUG } 06:47:32 DEBUG } 06:47:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728053" 06:47:33 INFO [http_cmd]: http status code OK 06:47:33 DEBUG --- status code --- 06:47:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:33 DEBUG --- http response --- 06:47:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:47:33 DEBUG ----- output ----- 06:47:33 DEBUG { 06:47:33 DEBUG "status": "success", 06:47:33 DEBUG "data": { 06:47:33 DEBUG "resultType": "vector", 06:47:33 DEBUG "result": [ 06:47:33 DEBUG { 06:47:33 DEBUG "metric": { 06:47:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:47:33 DEBUG }, 06:47:33 DEBUG "value": [ 06:47:33 DEBUG 06:47:33 DEBUG 06:47:33 DEBUG 06:47:33 DEBUG [...] 06:47:33 DEBUG 06:47:33 DEBUG 06:47:33 DEBUG 06:47:33 DEBUG "pod": "am-654c4fcbcf-487cv" 06:47:33 DEBUG }, 06:47:33 DEBUG "value": [ 06:47:33 DEBUG 1712728053, 06:47:33 DEBUG "1430.4" 06:47:33 DEBUG ] 06:47:33 DEBUG } 06:47:33 DEBUG ] 06:47:33 DEBUG } 06:47:33 DEBUG } 06:47:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728053" 06:47:33 INFO [http_cmd]: http status code OK 06:47:33 DEBUG --- status code --- 06:47:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:33 DEBUG --- http response --- 06:47:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:47:33 DEBUG ----- output ----- 06:47:33 DEBUG { 06:47:33 DEBUG "status": "success", 06:47:33 DEBUG "data": { 06:47:33 DEBUG "resultType": "vector", 06:47:33 DEBUG "result": [ 06:47:33 DEBUG { 06:47:33 DEBUG "metric": { 06:47:33 DEBUG "pod": "ds-idrepo-0" 06:47:33 DEBUG }, 06:47:33 DEBUG "value": [ 06:47:33 DEBUG 06:47:33 DEBUG 06:47:33 DEBUG 06:47:33 DEBUG [...] 06:47:33 DEBUG 06:47:33 DEBUG 06:47:33 DEBUG 06:47:33 DEBUG "pod": "ds-idrepo-1" 06:47:33 DEBUG }, 06:47:33 DEBUG "value": [ 06:47:33 DEBUG 1712728053, 06:47:33 DEBUG "0" 06:47:33 DEBUG ] 06:47:33 DEBUG } 06:47:33 DEBUG ] 06:47:33 DEBUG } 06:47:33 DEBUG } 06:47:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728053" 06:47:33 INFO [http_cmd]: http status code OK 06:47:33 DEBUG --- status code --- 06:47:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:33 DEBUG --- http response --- 06:47:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:47:33 DEBUG ----- output ----- 06:47:33 DEBUG { 06:47:33 DEBUG "status": "success", 06:47:33 DEBUG "data": { 06:47:33 DEBUG "resultType": "vector", 06:47:33 DEBUG "result": [ 06:47:33 DEBUG { 06:47:33 DEBUG "metric": { 06:47:33 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:47:33 DEBUG }, 06:47:33 DEBUG "value": [ 06:47:33 DEBUG 06:47:33 DEBUG 06:47:33 DEBUG 06:47:33 DEBUG [...] 06:47:33 DEBUG 06:47:33 DEBUG 06:47:33 DEBUG 06:47:33 DEBUG "pod": "ds-idrepo-0" 06:47:33 DEBUG }, 06:47:33 DEBUG "value": [ 06:47:33 DEBUG 1712728053, 06:47:33 DEBUG "2645.0708486712724" 06:47:33 DEBUG ] 06:47:33 DEBUG } 06:47:33 DEBUG ] 06:47:33 DEBUG } 06:47:33 DEBUG } 06:47:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712728053" 06:47:33 INFO [http_cmd]: http status code OK 06:47:33 DEBUG --- status code --- 06:47:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:33 DEBUG --- http response --- 06:47:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:47:33 DEBUG ----- output ----- 06:47:33 DEBUG { 06:47:33 DEBUG "status": "success", 06:47:33 DEBUG "data": { 06:47:33 DEBUG "resultType": "vector", 06:47:33 DEBUG "result": [ 06:47:33 DEBUG { 06:47:33 DEBUG "metric": { 06:47:33 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:47:33 DEBUG }, 06:47:33 DEBUG "value": [ 06:47:33 DEBUG 06:47:33 DEBUG 06:47:33 DEBUG 06:47:33 DEBUG [...] 06:47:33 DEBUG 06:47:33 DEBUG 06:47:33 DEBUG 06:47:33 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:47:33 DEBUG }, 06:47:33 DEBUG "value": [ 06:47:33 DEBUG 1712728053, 06:47:33 DEBUG "2487963648" 06:47:33 DEBUG ] 06:47:33 DEBUG } 06:47:33 DEBUG ] 06:47:33 DEBUG } 06:47:33 DEBUG } 06:47:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728053" 06:47:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728053" 06:47:33 INFO [http_cmd]: http status code OK 06:47:33 DEBUG --- status code --- 06:47:33 INFO [http_cmd]: http status code OK 06:47:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:33 DEBUG --- status code --- 06:47:33 DEBUG --- http response --- 06:47:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:47:33 DEBUG --- http response --- 06:47:33 DEBUG ----- output ----- 06:47:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:47:33 DEBUG { 06:47:33 DEBUG ----- output ----- 06:47:33 DEBUG "status": "success", 06:47:33 DEBUG { 06:47:33 DEBUG "data": { 06:47:33 DEBUG "status": "success", 06:47:33 DEBUG "resultType": "vector", 06:47:33 DEBUG "data": { 06:47:33 DEBUG "result": [ 06:47:33 DEBUG "resultType": "vector", 06:47:33 DEBUG { 06:47:33 DEBUG "result": [ 06:47:33 DEBUG "metric": { 06:47:33 DEBUG { 06:47:33 DEBUG "instance": "10.142.0.25:9100" 06:47:33 DEBUG "metric": { 06:47:33 DEBUG }, 06:47:33 DEBUG "instance": "10.142.0.25:9100" 06:47:33 DEBUG "value": [ 06:47:33 DEBUG }, 06:47:33 DEBUG 06:47:33 DEBUG "value": [ 06:47:33 DEBUG 06:47:33 DEBUG 06:47:33 DEBUG 06:47:33 DEBUG 06:47:33 DEBUG [...] 06:47:33 DEBUG 06:47:33 DEBUG 06:47:33 DEBUG [...] 06:47:33 DEBUG 06:47:33 DEBUG 06:47:33 DEBUG 06:47:33 DEBUG 06:47:33 DEBUG "instance": "10.142.0.77:9100" 06:47:33 DEBUG 06:47:33 DEBUG }, 06:47:33 DEBUG "instance": "10.142.0.77:9100" 06:47:33 DEBUG "value": [ 06:47:33 DEBUG }, 06:47:33 DEBUG 1712728053, 06:47:33 DEBUG "value": [ 06:47:33 DEBUG "0.0506666666666663" 06:47:33 DEBUG 1712728053, 06:47:33 DEBUG ] 06:47:33 DEBUG "9650286.666666666" 06:47:33 DEBUG } 06:47:33 DEBUG ] 06:47:33 DEBUG ] 06:47:33 DEBUG } 06:47:33 DEBUG } 06:47:33 DEBUG ] 06:47:33 DEBUG } 06:47:33 DEBUG } 06:47:33 DEBUG } 06:47:54 INFO 06:47:54 INFO [loop_until]: kubectl --namespace=xlou top pods 06:47:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:47:54 INFO [loop_until]: OK (rc = 0) 06:47:54 DEBUG --- stdout --- 06:47:54 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 1732m 8957Mi am-654c4fcbcf-9scrj 1680m 8389Mi am-654c4fcbcf-v5qb5 1445m 8089Mi ds-cts-0 3864m 13123Mi ds-cts-1 3983m 13387Mi ds-cts-2 3740m 13297Mi ds-idrepo-0 1275m 5634Mi ds-idrepo-1 11m 5672Mi ds-idrepo-2 12m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1191Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 4m 72Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 1m 1189Mi 06:47:54 DEBUG --- stderr --- 06:47:54 DEBUG 06:48:03 INFO 06:48:03 INFO [loop_until]: kubectl --namespace=xlou top node 06:48:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:48:03 INFO [loop_until]: OK (rc = 0) 06:48:03 DEBUG --- stdout --- 06:48:03 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 97m 0% 2308Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 1623m 10% 8980Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 78m 0% 3261Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 214m 1% 6526Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 1920m 12% 9309Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 1916m 12% 9769Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 77m 0% 858Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 1351m 8% 6324Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 3872m 24% 13848Mi 23% gke-xlou-cdm-ds-32e4dcb1-4hls 3871m 24% 14050Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 66m 0% 6394Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 4018m 25% 14142Mi 24% gke-xlou-cdm-ds-32e4dcb1-rr7g 63m 0% 6366Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 68m 0% 4816Mi 8% 06:48:03 DEBUG --- stderr --- 06:48:03 DEBUG 06:48:09 WARNING File /tmp/lodemon_data-20240409_230625/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [10/Apr/2024 06:48:09] "GET /monitoring/average?start_time=20240410_044722&stop_time=20240410_054608 HTTP/1.1" 200 - 06:48:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728110" 06:48:30 INFO [http_cmd]: http status code OK 06:48:30 DEBUG --- status code --- 06:48:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:30 DEBUG --- http response --- 06:48:30 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:48:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712728110" 06:48:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728110" 06:48:30 ERROR [http_cmd]: ERROR 06:48:30 INFO [http_cmd]: http status code OK 06:48:30 ERROR --- status code --- 06:48:30 DEBUG --- status code --- 06:48:30 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:48:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:30 ERROR --- http response --- 06:48:30 DEBUG --- http response --- 06:48:30 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:48:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:48:30 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:48:30 DEBUG ----- output ----- 06:48:30 DEBUG { 06:48:30 DEBUG "status": "success", 06:48:30 DEBUG "data": { 06:48:30 DEBUG "resultType": "vector", 06:48:30 DEBUG "result": [ 06:48:30 DEBUG { 06:48:30 DEBUG "metric": { 06:48:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:48:30 DEBUG }, 06:48:30 DEBUG "value": [ 06:48:30 DEBUG 06:48:30 DEBUG 06:48:30 DEBUG 06:48:30 DEBUG [...] 06:48:30 DEBUG 06:48:30 DEBUG 06:48:30 DEBUG 06:48:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:48:30 DEBUG }, 06:48:30 DEBUG "value": [ 06:48:30 DEBUG 1712728110, 06:48:30 DEBUG "0" 06:48:30 DEBUG ] 06:48:30 DEBUG } 06:48:30 DEBUG ] 06:48:30 DEBUG } 06:48:30 DEBUG } 06:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728111" 06:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728111" 06:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728111" 06:48:31 INFO [http_cmd]: http status code OK 06:48:31 DEBUG --- status code --- 06:48:31 INFO [http_cmd]: http status code OK 06:48:31 INFO [http_cmd]: http status code OK 06:48:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:31 DEBUG --- status code --- 06:48:31 DEBUG --- status code --- 06:48:31 DEBUG --- http response --- 06:48:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:48:31 DEBUG --- http response --- 06:48:31 DEBUG --- http response --- 06:48:31 DEBUG ----- output ----- 06:48:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:48:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:48:31 DEBUG { 06:48:31 DEBUG ----- output ----- 06:48:31 DEBUG ----- output ----- 06:48:31 DEBUG "status": "success", 06:48:31 DEBUG { 06:48:31 DEBUG { 06:48:31 DEBUG "data": { 06:48:31 DEBUG "status": "success", 06:48:31 DEBUG "status": "success", 06:48:31 DEBUG "resultType": "vector", 06:48:31 DEBUG "data": { 06:48:31 DEBUG "data": { 06:48:31 DEBUG "result": [ 06:48:31 DEBUG "resultType": "vector", 06:48:31 DEBUG "resultType": "vector", 06:48:31 DEBUG { 06:48:31 DEBUG "result": [ 06:48:31 DEBUG "result": [ 06:48:31 DEBUG "metric": { 06:48:31 DEBUG { 06:48:31 DEBUG { 06:48:31 DEBUG "pod": "ds-idrepo-0" 06:48:31 DEBUG "metric": { 06:48:31 DEBUG "metric": { 06:48:31 DEBUG }, 06:48:31 DEBUG "instance": "10.142.0.25:9100" 06:48:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:48:31 DEBUG "value": [ 06:48:31 DEBUG }, 06:48:31 DEBUG }, 06:48:31 DEBUG 06:48:31 DEBUG "value": [ 06:48:31 DEBUG "value": [ 06:48:31 DEBUG 06:48:31 DEBUG 06:48:31 DEBUG 06:48:31 DEBUG 06:48:31 DEBUG 06:48:31 DEBUG 06:48:31 DEBUG [...] 06:48:31 DEBUG 06:48:31 DEBUG 06:48:31 DEBUG 06:48:31 DEBUG [...] 06:48:31 DEBUG [...] 06:48:31 DEBUG 06:48:31 DEBUG 06:48:31 DEBUG 06:48:31 DEBUG 06:48:31 DEBUG 06:48:31 DEBUG 06:48:31 DEBUG "pod": "ds-idrepo-1" 06:48:31 DEBUG 06:48:31 DEBUG 06:48:31 DEBUG }, 06:48:31 DEBUG "instance": "10.142.0.77:9100" 06:48:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:48:31 DEBUG "value": [ 06:48:31 DEBUG }, 06:48:31 DEBUG }, 06:48:31 DEBUG 1712728111, 06:48:31 DEBUG "value": [ 06:48:31 DEBUG "value": [ 06:48:31 DEBUG "0" 06:48:31 DEBUG 1712728111, 06:48:31 DEBUG 1712728111, 06:48:31 DEBUG ] 06:48:31 DEBUG "9587957.433333334" 06:48:31 DEBUG "0" 06:48:31 DEBUG } 06:48:31 DEBUG ] 06:48:31 DEBUG ] 06:48:31 DEBUG ] 06:48:31 DEBUG } 06:48:31 DEBUG } 06:48:31 DEBUG } 06:48:31 DEBUG ] 06:48:31 DEBUG ] 06:48:31 DEBUG } 06:48:31 DEBUG } 06:48:31 DEBUG } 06:48:31 DEBUG } 06:48:31 DEBUG } 06:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728111" 06:48:31 INFO [http_cmd]: http status code OK 06:48:31 DEBUG --- status code --- 06:48:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:31 DEBUG --- http response --- 06:48:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:48:31 DEBUG ----- output ----- 06:48:31 DEBUG { 06:48:31 DEBUG "status": "success", 06:48:31 DEBUG "data": { 06:48:31 DEBUG "resultType": "vector", 06:48:31 DEBUG "result": [ 06:48:31 DEBUG { 06:48:31 DEBUG "metric": { 06:48:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:48:31 DEBUG }, 06:48:31 DEBUG "value": [ 06:48:31 DEBUG 06:48:31 DEBUG 06:48:31 DEBUG 06:48:31 DEBUG [...] 06:48:31 DEBUG 06:48:31 DEBUG 06:48:31 DEBUG 06:48:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:48:31 DEBUG }, 06:48:31 DEBUG "value": [ 06:48:31 DEBUG 1712728111, 06:48:31 DEBUG "0" 06:48:31 DEBUG ] 06:48:31 DEBUG } 06:48:31 DEBUG ] 06:48:31 DEBUG } 06:48:31 DEBUG } 06:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728112" 06:48:32 INFO [http_cmd]: http status code OK 06:48:32 DEBUG --- status code --- 06:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:32 DEBUG --- http response --- 06:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:48:32 DEBUG ----- output ----- 06:48:32 DEBUG { 06:48:32 DEBUG "status": "success", 06:48:32 DEBUG "data": { 06:48:32 DEBUG "resultType": "vector", 06:48:32 DEBUG "result": [ 06:48:32 DEBUG { 06:48:32 DEBUG "metric": { 06:48:32 DEBUG "instance": "10.142.0.25:9100" 06:48:32 DEBUG }, 06:48:32 DEBUG "value": [ 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG [...] 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG "instance": "10.142.0.77:9100" 06:48:32 DEBUG }, 06:48:32 DEBUG "value": [ 06:48:32 DEBUG 1712728112, 06:48:32 DEBUG "437853221888" 06:48:32 DEBUG ] 06:48:32 DEBUG } 06:48:32 DEBUG ] 06:48:32 DEBUG } 06:48:32 DEBUG } 06:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728112" 06:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728112" 06:48:32 INFO [http_cmd]: http status code OK 06:48:32 DEBUG --- status code --- 06:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:32 DEBUG --- http response --- 06:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:48:32 DEBUG ----- output ----- 06:48:32 DEBUG { 06:48:32 DEBUG "status": "success", 06:48:32 DEBUG "data": { 06:48:32 DEBUG "resultType": "vector", 06:48:32 DEBUG "result": [ 06:48:32 DEBUG { 06:48:32 DEBUG "metric": { 06:48:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:48:32 DEBUG }, 06:48:32 DEBUG "value": [ 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG [...] 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:48:32 DEBUG }, 06:48:32 DEBUG "value": [ 06:48:32 DEBUG 1712728112, 06:48:32 DEBUG "111858" 06:48:32 DEBUG ] 06:48:32 DEBUG } 06:48:32 DEBUG ] 06:48:32 DEBUG } 06:48:32 DEBUG } 06:48:32 INFO [http_cmd]: http status code OK 06:48:32 DEBUG --- status code --- 06:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:32 DEBUG --- http response --- 06:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:48:32 DEBUG ----- output ----- 06:48:32 DEBUG { 06:48:32 DEBUG "status": "success", 06:48:32 DEBUG "data": { 06:48:32 DEBUG "resultType": "vector", 06:48:32 DEBUG "result": [ 06:48:32 DEBUG { 06:48:32 DEBUG "metric": { 06:48:32 DEBUG "instance": "10.142.0.25:9100" 06:48:32 DEBUG }, 06:48:32 DEBUG "value": [ 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG [...] 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG "instance": "10.142.0.77:9100" 06:48:32 DEBUG }, 06:48:32 DEBUG "value": [ 06:48:32 DEBUG 1712728112, 06:48:32 DEBUG "0.014499999999998417" 06:48:32 DEBUG ] 06:48:32 DEBUG } 06:48:32 DEBUG ] 06:48:32 DEBUG } 06:48:32 DEBUG } 06:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728112" 06:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728112" 06:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728112" 06:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728112" 06:48:32 INFO [http_cmd]: http status code OK 06:48:32 DEBUG --- status code --- 06:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:32 DEBUG --- http response --- 06:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:48:32 DEBUG ----- output ----- 06:48:32 DEBUG { 06:48:32 DEBUG "status": "success", 06:48:32 DEBUG "data": { 06:48:32 DEBUG "resultType": "vector", 06:48:32 DEBUG "result": [ 06:48:32 DEBUG { 06:48:32 DEBUG "metric": { 06:48:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:48:32 DEBUG }, 06:48:32 DEBUG "value": [ 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG [...] 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG "pod": "login-ui-5f964fb894-5zpvq" 06:48:32 DEBUG }, 06:48:32 DEBUG "value": [ 06:48:32 DEBUG 1712728112, 06:48:32 DEBUG "0" 06:48:32 DEBUG ] 06:48:32 DEBUG } 06:48:32 DEBUG ] 06:48:32 DEBUG } 06:48:32 DEBUG } 06:48:32 INFO [http_cmd]: http status code OK 06:48:32 DEBUG --- status code --- 06:48:32 INFO [http_cmd]: http status code OK 06:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:32 DEBUG --- status code --- 06:48:32 DEBUG --- http response --- 06:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:48:32 DEBUG --- http response --- 06:48:32 DEBUG ----- output ----- 06:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:48:32 DEBUG { 06:48:32 DEBUG ----- output ----- 06:48:32 DEBUG "status": "success", 06:48:32 DEBUG { 06:48:32 INFO [http_cmd]: http status code OK 06:48:32 DEBUG "data": { 06:48:32 DEBUG "status": "success", 06:48:32 DEBUG --- status code --- 06:48:32 DEBUG "resultType": "vector", 06:48:32 DEBUG "data": { 06:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:32 DEBUG "result": [ 06:48:32 DEBUG "resultType": "vector", 06:48:32 DEBUG --- http response --- 06:48:32 DEBUG { 06:48:32 DEBUG "result": [ 06:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:48:32 DEBUG "metric": { 06:48:32 DEBUG { 06:48:32 DEBUG ----- output ----- 06:48:32 DEBUG "pod": "ds-idrepo-0" 06:48:32 DEBUG "metric": { 06:48:32 DEBUG { 06:48:32 DEBUG }, 06:48:32 DEBUG "pod": "ds-idrepo-0" 06:48:32 DEBUG "status": "success", 06:48:32 DEBUG "value": [ 06:48:32 DEBUG }, 06:48:32 DEBUG "data": { 06:48:32 DEBUG 06:48:32 DEBUG "value": [ 06:48:32 DEBUG "resultType": "vector", 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG "result": [ 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG { 06:48:32 DEBUG [...] 06:48:32 DEBUG 06:48:32 DEBUG "metric": { 06:48:32 DEBUG 06:48:32 DEBUG [...] 06:48:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG }, 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG "value": [ 06:48:32 DEBUG "pod": "ds-cts-2" 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG }, 06:48:32 DEBUG "pod": "ds-cts-2" 06:48:32 DEBUG 06:48:32 DEBUG "value": [ 06:48:32 DEBUG }, 06:48:32 DEBUG 06:48:32 DEBUG 1712728112, 06:48:32 DEBUG "value": [ 06:48:32 DEBUG [...] 06:48:32 DEBUG "0" 06:48:32 DEBUG 1712728112, 06:48:32 DEBUG 06:48:32 DEBUG ] 06:48:32 DEBUG "29.766666666666666" 06:48:32 DEBUG 06:48:32 DEBUG } 06:48:32 DEBUG ] 06:48:32 DEBUG 06:48:32 DEBUG ] 06:48:32 DEBUG } 06:48:32 DEBUG "pod": "login-ui-5f964fb894-5zpvq" 06:48:32 DEBUG } 06:48:32 DEBUG ] 06:48:32 DEBUG }, 06:48:32 DEBUG } 06:48:32 DEBUG } 06:48:32 DEBUG "value": [ 06:48:32 DEBUG } 06:48:32 DEBUG 1712728112, 06:48:32 DEBUG "0" 06:48:32 DEBUG ] 06:48:32 DEBUG } 06:48:32 DEBUG ] 06:48:32 DEBUG } 06:48:32 DEBUG } 06:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712728112" 06:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712728112" 06:48:32 INFO [http_cmd]: http status code OK 06:48:32 DEBUG --- status code --- 06:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:32 DEBUG --- http response --- 06:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:48:32 DEBUG ----- output ----- 06:48:32 DEBUG { 06:48:32 DEBUG "status": "success", 06:48:32 DEBUG "data": { 06:48:32 DEBUG "resultType": "vector", 06:48:32 DEBUG "result": [ 06:48:32 DEBUG { 06:48:32 DEBUG "metric": { 06:48:32 DEBUG "pod": "ds-idrepo-0" 06:48:32 DEBUG }, 06:48:32 DEBUG "value": [ 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG [...] 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG "pod": "ds-cts-2" 06:48:32 DEBUG }, 06:48:32 DEBUG "value": [ 06:48:32 DEBUG 1712728112, 06:48:32 DEBUG "0" 06:48:32 DEBUG ] 06:48:32 DEBUG } 06:48:32 DEBUG ] 06:48:32 DEBUG } 06:48:32 DEBUG } 06:48:32 INFO [http_cmd]: http status code OK 06:48:32 DEBUG --- status code --- 06:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:32 DEBUG --- http response --- 06:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:48:32 DEBUG ----- output ----- 06:48:32 DEBUG { 06:48:32 DEBUG "status": "success", 06:48:32 DEBUG "data": { 06:48:32 DEBUG "resultType": "vector", 06:48:32 DEBUG "result": [ 06:48:32 DEBUG { 06:48:32 DEBUG "metric": { 06:48:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:48:32 DEBUG }, 06:48:32 DEBUG "value": [ 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG [...] 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:48:32 DEBUG }, 06:48:32 DEBUG "value": [ 06:48:32 DEBUG 1712728112, 06:48:32 DEBUG "0.03048945864774621" 06:48:32 DEBUG ] 06:48:32 DEBUG } 06:48:32 DEBUG ] 06:48:32 DEBUG } 06:48:32 DEBUG } 06:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728112" 06:48:32 INFO [http_cmd]: http status code OK 06:48:32 DEBUG --- status code --- 06:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:32 DEBUG --- http response --- 06:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:48:32 DEBUG ----- output ----- 06:48:32 DEBUG { 06:48:32 DEBUG "status": "success", 06:48:32 DEBUG "data": { 06:48:32 DEBUG "resultType": "vector", 06:48:32 DEBUG "result": [ 06:48:32 DEBUG { 06:48:32 DEBUG "metric": { 06:48:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:48:32 DEBUG }, 06:48:32 DEBUG "value": [ 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG [...] 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:48:32 DEBUG }, 06:48:32 DEBUG "value": [ 06:48:32 DEBUG 1712728112, 06:48:32 DEBUG "0" 06:48:32 DEBUG ] 06:48:32 DEBUG } 06:48:32 DEBUG ] 06:48:32 DEBUG } 06:48:32 DEBUG } 06:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712728112" 06:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728112" 06:48:32 INFO [http_cmd]: http status code OK 06:48:32 INFO [http_cmd]: http status code OK 06:48:32 DEBUG --- status code --- 06:48:32 DEBUG --- status code --- 06:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:32 DEBUG --- http response --- 06:48:32 DEBUG --- http response --- 06:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:48:32 DEBUG ----- output ----- 06:48:32 DEBUG ----- output ----- 06:48:32 DEBUG { 06:48:32 DEBUG { 06:48:32 DEBUG "status": "success", 06:48:32 DEBUG "status": "success", 06:48:32 DEBUG "data": { 06:48:32 DEBUG "data": { 06:48:32 DEBUG "resultType": "vector", 06:48:32 DEBUG "resultType": "vector", 06:48:32 DEBUG "result": [ 06:48:32 DEBUG "result": [ 06:48:32 DEBUG { 06:48:32 DEBUG { 06:48:32 DEBUG "metric": { 06:48:32 DEBUG "metric": { 06:48:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:48:32 DEBUG "pod": "ds-idrepo-2" 06:48:32 DEBUG }, 06:48:32 DEBUG }, 06:48:32 DEBUG "value": [ 06:48:32 DEBUG "value": [ 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG [...] 06:48:32 DEBUG [...] 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:48:32 DEBUG "pod": "ds-cts-2" 06:48:32 DEBUG }, 06:48:32 DEBUG }, 06:48:32 DEBUG "value": [ 06:48:32 DEBUG "value": [ 06:48:32 DEBUG 1712728112, 06:48:32 DEBUG 1712728112, 06:48:32 DEBUG "0.052" 06:48:32 DEBUG "0.00018464333333333333" 06:48:32 DEBUG ] 06:48:32 DEBUG ] 06:48:32 DEBUG } 06:48:32 DEBUG } 06:48:32 DEBUG ] 06:48:32 DEBUG ] 06:48:32 DEBUG } 06:48:32 DEBUG } 06:48:32 DEBUG } 06:48:32 DEBUG } 06:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728112" 06:48:32 INFO [http_cmd]: http status code OK 06:48:32 DEBUG --- status code --- 06:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:32 DEBUG --- http response --- 06:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:48:32 DEBUG ----- output ----- 06:48:32 DEBUG { 06:48:32 DEBUG "status": "success", 06:48:32 DEBUG "data": { 06:48:32 DEBUG "resultType": "vector", 06:48:32 DEBUG "result": [ 06:48:32 DEBUG { 06:48:32 DEBUG "metric": { 06:48:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:48:32 DEBUG }, 06:48:32 DEBUG "value": [ 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG [...] 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG 06:48:32 DEBUG "pod": "login-ui-5f964fb894-5zpvq" 06:48:32 DEBUG }, 06:48:32 DEBUG "value": [ 06:48:32 DEBUG 1712728112, 06:48:32 DEBUG "0.000030661555458838996" 06:48:32 DEBUG ] 06:48:32 DEBUG } 06:48:32 DEBUG ] 06:48:32 DEBUG } 06:48:32 DEBUG } 06:48:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728113" 06:48:33 INFO [http_cmd]: http status code OK 06:48:33 DEBUG --- status code --- 06:48:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:33 DEBUG --- http response --- 06:48:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:48:33 DEBUG ----- output ----- 06:48:33 DEBUG { 06:48:33 DEBUG "status": "success", 06:48:33 DEBUG "data": { 06:48:33 DEBUG "resultType": "vector", 06:48:33 DEBUG "result": [ 06:48:33 DEBUG { 06:48:33 DEBUG "metric": { 06:48:33 DEBUG "pod": "ds-idrepo-0" 06:48:33 DEBUG }, 06:48:33 DEBUG "value": [ 06:48:33 DEBUG 06:48:33 DEBUG 06:48:33 DEBUG 06:48:33 DEBUG [...] 06:48:33 DEBUG 06:48:33 DEBUG 06:48:33 DEBUG 06:48:33 DEBUG "pod": "ds-idrepo-1" 06:48:33 DEBUG }, 06:48:33 DEBUG "value": [ 06:48:33 DEBUG 1712728113, 06:48:33 DEBUG "0" 06:48:33 DEBUG ] 06:48:33 DEBUG } 06:48:33 DEBUG ] 06:48:33 DEBUG } 06:48:33 DEBUG } 06:48:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728113" 06:48:33 INFO [http_cmd]: http status code OK 06:48:33 DEBUG --- status code --- 06:48:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:33 DEBUG --- http response --- 06:48:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:48:33 DEBUG ----- output ----- 06:48:33 DEBUG { 06:48:33 DEBUG "status": "success", 06:48:33 DEBUG "data": { 06:48:33 DEBUG "resultType": "vector", 06:48:33 DEBUG "result": [ 06:48:33 DEBUG { 06:48:33 DEBUG "metric": { 06:48:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:48:33 DEBUG }, 06:48:33 DEBUG "value": [ 06:48:33 DEBUG 06:48:33 DEBUG 06:48:33 DEBUG 06:48:33 DEBUG [...] 06:48:33 DEBUG 06:48:33 DEBUG 06:48:33 DEBUG 06:48:33 DEBUG "pod": "am-654c4fcbcf-487cv" 06:48:33 DEBUG }, 06:48:33 DEBUG "value": [ 06:48:33 DEBUG 1712728113, 06:48:33 DEBUG "1424.9333333333334" 06:48:33 DEBUG ] 06:48:33 DEBUG } 06:48:33 DEBUG ] 06:48:33 DEBUG } 06:48:33 DEBUG } 06:48:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728113" 06:48:33 INFO [http_cmd]: http status code OK 06:48:33 DEBUG --- status code --- 06:48:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:33 DEBUG --- http response --- 06:48:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:48:33 DEBUG ----- output ----- 06:48:33 DEBUG { 06:48:33 DEBUG "status": "success", 06:48:33 DEBUG "data": { 06:48:33 DEBUG "resultType": "vector", 06:48:33 DEBUG "result": [ 06:48:33 DEBUG { 06:48:33 DEBUG "metric": { 06:48:33 DEBUG "pod": "ds-idrepo-0" 06:48:33 DEBUG }, 06:48:33 DEBUG "value": [ 06:48:33 DEBUG 06:48:33 DEBUG 06:48:33 DEBUG 06:48:33 DEBUG [...] 06:48:33 DEBUG 06:48:33 DEBUG 06:48:33 DEBUG 06:48:33 DEBUG "pod": "ds-idrepo-1" 06:48:33 DEBUG }, 06:48:33 DEBUG "value": [ 06:48:33 DEBUG 1712728113, 06:48:33 DEBUG "0" 06:48:33 DEBUG ] 06:48:33 DEBUG } 06:48:33 DEBUG ] 06:48:33 DEBUG } 06:48:33 DEBUG } 06:48:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728113" 06:48:33 INFO [http_cmd]: http status code OK 06:48:33 DEBUG --- status code --- 06:48:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:33 DEBUG --- http response --- 06:48:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:48:33 DEBUG ----- output ----- 06:48:33 DEBUG { 06:48:33 DEBUG "status": "success", 06:48:33 DEBUG "data": { 06:48:33 DEBUG "resultType": "vector", 06:48:33 DEBUG "result": [ 06:48:33 DEBUG { 06:48:33 DEBUG "metric": { 06:48:33 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:48:33 DEBUG }, 06:48:33 DEBUG "value": [ 06:48:33 DEBUG 06:48:33 DEBUG 06:48:33 DEBUG 06:48:33 DEBUG [...] 06:48:33 DEBUG 06:48:33 DEBUG 06:48:33 DEBUG 06:48:33 DEBUG "pod": "login-ui-5f964fb894-5zpvq" 06:48:33 DEBUG }, 06:48:33 DEBUG "value": [ 06:48:33 DEBUG 1712728113, 06:48:33 DEBUG "0" 06:48:33 DEBUG ] 06:48:33 DEBUG } 06:48:33 DEBUG ] 06:48:33 DEBUG } 06:48:33 DEBUG } 06:48:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712728113" 06:48:33 INFO [http_cmd]: http status code OK 06:48:33 DEBUG --- status code --- 06:48:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:33 DEBUG --- http response --- 06:48:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:48:33 DEBUG ----- output ----- 06:48:33 DEBUG { 06:48:33 DEBUG "status": "success", 06:48:33 DEBUG "data": { 06:48:33 DEBUG "resultType": "vector", 06:48:33 DEBUG "result": [ 06:48:33 DEBUG { 06:48:33 DEBUG "metric": { 06:48:33 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:48:33 DEBUG }, 06:48:33 DEBUG "value": [ 06:48:33 DEBUG 06:48:33 DEBUG 06:48:33 DEBUG 06:48:33 DEBUG [...] 06:48:33 DEBUG 06:48:33 DEBUG 06:48:33 DEBUG 06:48:33 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:48:33 DEBUG }, 06:48:33 DEBUG "value": [ 06:48:33 DEBUG 1712728113, 06:48:33 DEBUG "2487713792" 06:48:33 DEBUG ] 06:48:33 DEBUG } 06:48:33 DEBUG ] 06:48:33 DEBUG } 06:48:33 DEBUG } 06:48:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728113" 06:48:33 INFO [http_cmd]: http status code OK 06:48:33 DEBUG --- status code --- 06:48:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:33 DEBUG --- http response --- 06:48:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:48:33 DEBUG ----- output ----- 06:48:33 DEBUG { 06:48:33 DEBUG "status": "success", 06:48:33 DEBUG "data": { 06:48:33 DEBUG "resultType": "vector", 06:48:33 DEBUG "result": [ 06:48:33 DEBUG { 06:48:33 DEBUG "metric": { 06:48:33 DEBUG "instance": "10.142.0.25:9100" 06:48:33 DEBUG }, 06:48:33 DEBUG "value": [ 06:48:33 DEBUG 06:48:33 DEBUG 06:48:33 DEBUG 06:48:33 DEBUG [...] 06:48:33 DEBUG 06:48:33 DEBUG 06:48:33 DEBUG 06:48:33 DEBUG "instance": "10.142.0.77:9100" 06:48:33 DEBUG }, 06:48:33 DEBUG "value": [ 06:48:33 DEBUG 1712728113, 06:48:33 DEBUG "9589042.366666667" 06:48:33 DEBUG ] 06:48:33 DEBUG } 06:48:33 DEBUG ] 06:48:33 DEBUG } 06:48:33 DEBUG } 06:48:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728113" 06:48:33 INFO [http_cmd]: http status code OK 06:48:33 DEBUG --- status code --- 06:48:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:33 DEBUG --- http response --- 06:48:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:48:33 DEBUG ----- output ----- 06:48:33 DEBUG { 06:48:33 DEBUG "status": "success", 06:48:33 DEBUG "data": { 06:48:33 DEBUG "resultType": "vector", 06:48:33 DEBUG "result": [ 06:48:33 DEBUG { 06:48:33 DEBUG "metric": { 06:48:33 DEBUG "instance": "10.142.0.25:9100" 06:48:33 DEBUG }, 06:48:33 DEBUG "value": [ 06:48:33 DEBUG 06:48:33 DEBUG 06:48:33 DEBUG 06:48:33 DEBUG [...] 06:48:33 DEBUG 06:48:33 DEBUG 06:48:33 DEBUG 06:48:33 DEBUG "instance": "10.142.0.77:9100" 06:48:33 DEBUG }, 06:48:33 DEBUG "value": [ 06:48:33 DEBUG 1712728113, 06:48:33 DEBUG "0.0016666666666663352" 06:48:33 DEBUG ] 06:48:33 DEBUG } 06:48:33 DEBUG ] 06:48:33 DEBUG } 06:48:33 DEBUG } 06:48:54 INFO 06:48:54 INFO [loop_until]: kubectl --namespace=xlou top pods 06:48:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:48:54 INFO [loop_until]: OK (rc = 0) 06:48:54 DEBUG --- stdout --- 06:48:54 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 7Mi am-654c4fcbcf-487cv 1706m 8881Mi am-654c4fcbcf-9scrj 1658m 8406Mi am-654c4fcbcf-v5qb5 1412m 8130Mi ds-cts-0 3786m 12701Mi ds-cts-1 3983m 12890Mi ds-cts-2 3355m 11429Mi ds-idrepo-0 1359m 5634Mi ds-idrepo-1 11m 5673Mi ds-idrepo-2 12m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 4Mi idm-6bfb7c96cc-qklk9 4m 1191Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 10m 73Mi login-ui-5f964fb894-5zpvq 1m 3Mi overseer-0-6f8784665d-zq5hb 3m 1189Mi 06:48:54 DEBUG --- stderr --- 06:48:54 DEBUG 06:49:03 INFO 06:49:03 INFO [loop_until]: kubectl --namespace=xlou top node 06:49:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:49:03 INFO [loop_until]: OK (rc = 0) 06:49:03 DEBUG --- stdout --- 06:49:03 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 102m 0% 2309Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 1580m 9% 8999Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 85m 0% 3258Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 205m 1% 6526Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 1886m 11% 9286Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 1926m 12% 9736Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 84m 0% 852Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 1362m 8% 6321Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 3813m 23% 13437Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 3438m 21% 12178Mi 20% gke-xlou-cdm-ds-32e4dcb1-gxzj 69m 0% 6399Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 3857m 24% 13631Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 64m 0% 6364Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 82m 0% 4815Mi 8% 06:49:03 DEBUG --- stderr --- 06:49:03 DEBUG 06:49:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728170" 06:49:30 INFO [http_cmd]: http status code OK 06:49:30 DEBUG --- status code --- 06:49:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:30 DEBUG --- http response --- 06:49:30 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:49:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712728170" 06:49:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728170" 06:49:30 ERROR [http_cmd]: ERROR 06:49:30 ERROR --- status code --- 06:49:30 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:49:30 ERROR --- http response --- 06:49:30 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:49:30 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:49:30 INFO [http_cmd]: http status code OK 06:49:30 DEBUG --- status code --- 06:49:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:30 DEBUG --- http response --- 06:49:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:49:30 DEBUG ----- output ----- 06:49:30 DEBUG { 06:49:30 DEBUG "status": "success", 06:49:30 DEBUG "data": { 06:49:30 DEBUG "resultType": "vector", 06:49:30 DEBUG "result": [ 06:49:30 DEBUG { 06:49:30 DEBUG "metric": { 06:49:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:49:30 DEBUG }, 06:49:30 DEBUG "value": [ 06:49:30 DEBUG 06:49:30 DEBUG 06:49:30 DEBUG 06:49:30 DEBUG [...] 06:49:30 DEBUG 06:49:30 DEBUG 06:49:30 DEBUG 06:49:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:49:30 DEBUG }, 06:49:30 DEBUG "value": [ 06:49:30 DEBUG 1712728170, 06:49:30 DEBUG "0" 06:49:30 DEBUG ] 06:49:30 DEBUG } 06:49:30 DEBUG ] 06:49:30 DEBUG } 06:49:30 DEBUG } 06:49:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728171" 06:49:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728171" 06:49:31 INFO [http_cmd]: http status code OK 06:49:31 DEBUG --- status code --- 06:49:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:31 DEBUG --- http response --- 06:49:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:49:31 DEBUG ----- output ----- 06:49:31 DEBUG { 06:49:31 DEBUG "status": "success", 06:49:31 DEBUG "data": { 06:49:31 DEBUG "resultType": "vector", 06:49:31 DEBUG "result": [ 06:49:31 DEBUG { 06:49:31 DEBUG "metric": { 06:49:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:49:31 DEBUG }, 06:49:31 DEBUG "value": [ 06:49:31 DEBUG 06:49:31 DEBUG 06:49:31 DEBUG 06:49:31 DEBUG [...] 06:49:31 DEBUG 06:49:31 DEBUG 06:49:31 DEBUG 06:49:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:49:31 DEBUG }, 06:49:31 DEBUG "value": [ 06:49:31 DEBUG 1712728171, 06:49:31 DEBUG "0" 06:49:31 DEBUG ] 06:49:31 DEBUG } 06:49:31 DEBUG ] 06:49:31 DEBUG } 06:49:31 DEBUG } 06:49:31 INFO [http_cmd]: http status code OK 06:49:31 DEBUG --- status code --- 06:49:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:31 DEBUG --- http response --- 06:49:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:49:31 DEBUG ----- output ----- 06:49:31 DEBUG { 06:49:31 DEBUG "status": "success", 06:49:31 DEBUG "data": { 06:49:31 DEBUG "resultType": "vector", 06:49:31 DEBUG "result": [ 06:49:31 DEBUG { 06:49:31 DEBUG "metric": { 06:49:31 DEBUG "instance": "10.142.0.25:9100" 06:49:31 DEBUG }, 06:49:31 DEBUG "value": [ 06:49:31 DEBUG 06:49:31 DEBUG 06:49:31 DEBUG 06:49:31 DEBUG [...] 06:49:31 DEBUG 06:49:31 DEBUG 06:49:31 DEBUG 06:49:31 DEBUG "instance": "10.142.0.77:9100" 06:49:31 DEBUG }, 06:49:31 DEBUG "value": [ 06:49:31 DEBUG 1712728171, 06:49:31 DEBUG "8115188.933333334" 06:49:31 DEBUG ] 06:49:31 DEBUG } 06:49:31 DEBUG ] 06:49:31 DEBUG } 06:49:31 DEBUG } 06:49:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728171" 06:49:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728171" 06:49:32 INFO [http_cmd]: http status code OK 06:49:32 DEBUG --- status code --- 06:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:32 DEBUG --- http response --- 06:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:49:32 DEBUG ----- output ----- 06:49:32 DEBUG { 06:49:32 DEBUG "status": "success", 06:49:32 DEBUG "data": { 06:49:32 DEBUG "resultType": "vector", 06:49:32 DEBUG "result": [ 06:49:32 DEBUG { 06:49:32 DEBUG "metric": { 06:49:32 DEBUG "pod": "ds-idrepo-0" 06:49:32 DEBUG }, 06:49:32 DEBUG "value": [ 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG [...] 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG "pod": "ds-idrepo-1" 06:49:32 DEBUG }, 06:49:32 INFO [http_cmd]: http status code OK 06:49:32 DEBUG "value": [ 06:49:32 DEBUG --- status code --- 06:49:32 DEBUG 1712728171, 06:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:32 DEBUG "0" 06:49:32 DEBUG --- http response --- 06:49:32 DEBUG ] 06:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:49:32 DEBUG } 06:49:32 DEBUG ----- output ----- 06:49:32 DEBUG ] 06:49:32 DEBUG { 06:49:32 DEBUG } 06:49:32 DEBUG "status": "success", 06:49:32 DEBUG } 06:49:32 DEBUG "data": { 06:49:32 DEBUG "resultType": "vector", 06:49:32 DEBUG "result": [ 06:49:32 DEBUG { 06:49:32 DEBUG "metric": { 06:49:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:49:32 DEBUG }, 06:49:32 DEBUG "value": [ 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG [...] 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:49:32 DEBUG }, 06:49:32 DEBUG "value": [ 06:49:32 DEBUG 1712728171, 06:49:32 DEBUG "0" 06:49:32 DEBUG ] 06:49:32 DEBUG } 06:49:32 DEBUG ] 06:49:32 DEBUG } 06:49:32 DEBUG } 06:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728172" 06:49:32 INFO [http_cmd]: http status code OK 06:49:32 DEBUG --- status code --- 06:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:32 DEBUG --- http response --- 06:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:49:32 DEBUG ----- output ----- 06:49:32 DEBUG { 06:49:32 DEBUG "status": "success", 06:49:32 DEBUG "data": { 06:49:32 DEBUG "resultType": "vector", 06:49:32 DEBUG "result": [ 06:49:32 DEBUG { 06:49:32 DEBUG "metric": { 06:49:32 DEBUG "instance": "10.142.0.25:9100" 06:49:32 DEBUG }, 06:49:32 DEBUG "value": [ 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG [...] 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG "instance": "10.142.0.77:9100" 06:49:32 DEBUG }, 06:49:32 DEBUG "value": [ 06:49:32 DEBUG 1712728172, 06:49:32 DEBUG "438032409600" 06:49:32 DEBUG ] 06:49:32 DEBUG } 06:49:32 DEBUG ] 06:49:32 DEBUG } 06:49:32 DEBUG } 06:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728172" 06:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728172" 06:49:32 INFO [http_cmd]: http status code OK 06:49:32 DEBUG --- status code --- 06:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:32 DEBUG --- http response --- 06:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:49:32 DEBUG ----- output ----- 06:49:32 DEBUG { 06:49:32 DEBUG "status": "success", 06:49:32 DEBUG "data": { 06:49:32 DEBUG "resultType": "vector", 06:49:32 DEBUG "result": [ 06:49:32 DEBUG { 06:49:32 DEBUG "metric": { 06:49:32 DEBUG "instance": "10.142.0.25:9100" 06:49:32 DEBUG }, 06:49:32 DEBUG "value": [ 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG [...] 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG "instance": "10.142.0.77:9100" 06:49:32 DEBUG }, 06:49:32 DEBUG "value": [ 06:49:32 DEBUG 1712728172, 06:49:32 DEBUG "0.014333333333334034" 06:49:32 DEBUG ] 06:49:32 DEBUG } 06:49:32 DEBUG ] 06:49:32 DEBUG } 06:49:32 DEBUG } 06:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728172" 06:49:32 INFO [http_cmd]: http status code OK 06:49:32 DEBUG --- status code --- 06:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:32 DEBUG --- http response --- 06:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:49:32 DEBUG ----- output ----- 06:49:32 DEBUG { 06:49:32 DEBUG "status": "success", 06:49:32 DEBUG "data": { 06:49:32 DEBUG "resultType": "vector", 06:49:32 DEBUG "result": [ 06:49:32 DEBUG { 06:49:32 DEBUG "metric": { 06:49:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:49:32 DEBUG }, 06:49:32 DEBUG "value": [ 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG [...] 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:49:32 DEBUG }, 06:49:32 DEBUG "value": [ 06:49:32 DEBUG 1712728172, 06:49:32 DEBUG "25425" 06:49:32 DEBUG ] 06:49:32 DEBUG } 06:49:32 DEBUG ] 06:49:32 DEBUG } 06:49:32 DEBUG } 06:49:32 INFO [http_cmd]: http status code OK 06:49:32 DEBUG --- status code --- 06:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:32 DEBUG --- http response --- 06:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:49:32 DEBUG ----- output ----- 06:49:32 DEBUG { 06:49:32 DEBUG "status": "success", 06:49:32 DEBUG "data": { 06:49:32 DEBUG "resultType": "vector", 06:49:32 DEBUG "result": [ 06:49:32 DEBUG { 06:49:32 DEBUG "metric": { 06:49:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:49:32 DEBUG }, 06:49:32 DEBUG "value": [ 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG [...] 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:49:32 DEBUG }, 06:49:32 DEBUG "value": [ 06:49:32 DEBUG 1712728172, 06:49:32 DEBUG "307.73155357804967" 06:49:32 DEBUG ] 06:49:32 DEBUG } 06:49:32 DEBUG ] 06:49:32 DEBUG } 06:49:32 DEBUG } 06:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728172" 06:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728172" 06:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728172" 06:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712728172" 06:49:32 INFO [http_cmd]: http status code OK 06:49:32 DEBUG --- status code --- 06:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:32 DEBUG --- http response --- 06:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:49:32 DEBUG ----- output ----- 06:49:32 DEBUG { 06:49:32 DEBUG "status": "success", 06:49:32 DEBUG "data": { 06:49:32 DEBUG "resultType": "vector", 06:49:32 DEBUG "result": [ 06:49:32 DEBUG { 06:49:32 DEBUG "metric": { 06:49:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:49:32 DEBUG }, 06:49:32 DEBUG "value": [ 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG [...] 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:49:32 DEBUG }, 06:49:32 DEBUG "value": [ 06:49:32 DEBUG 1712728172, 06:49:32 DEBUG "0" 06:49:32 DEBUG ] 06:49:32 DEBUG } 06:49:32 DEBUG ] 06:49:32 DEBUG } 06:49:32 DEBUG } 06:49:32 INFO [http_cmd]: http status code OK 06:49:32 INFO [http_cmd]: http status code OK 06:49:32 DEBUG --- status code --- 06:49:32 DEBUG --- status code --- 06:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:32 DEBUG --- http response --- 06:49:32 DEBUG --- http response --- 06:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:49:32 DEBUG ----- output ----- 06:49:32 DEBUG ----- output ----- 06:49:32 DEBUG { 06:49:32 DEBUG { 06:49:32 DEBUG "status": "success", 06:49:32 DEBUG "status": "success", 06:49:32 DEBUG "data": { 06:49:32 DEBUG "data": { 06:49:32 DEBUG "resultType": "vector", 06:49:32 DEBUG "resultType": "vector", 06:49:32 DEBUG "result": [ 06:49:32 DEBUG "result": [ 06:49:32 DEBUG { 06:49:32 DEBUG { 06:49:32 DEBUG "metric": { 06:49:32 DEBUG "metric": { 06:49:32 DEBUG "pod": "ds-idrepo-0" 06:49:32 DEBUG "pod": "ds-idrepo-0" 06:49:32 DEBUG }, 06:49:32 DEBUG }, 06:49:32 DEBUG "value": [ 06:49:32 DEBUG "value": [ 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG [...] 06:49:32 DEBUG [...] 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG "pod": "ds-cts-2" 06:49:32 DEBUG "pod": "ds-cts-2" 06:49:32 DEBUG }, 06:49:32 DEBUG }, 06:49:32 DEBUG "value": [ 06:49:32 DEBUG "value": [ 06:49:32 DEBUG 1712728172, 06:49:32 DEBUG 1712728172, 06:49:32 DEBUG "0" 06:49:32 DEBUG "29.9" 06:49:32 DEBUG ] 06:49:32 DEBUG ] 06:49:32 INFO [http_cmd]: http status code OK 06:49:32 DEBUG } 06:49:32 DEBUG } 06:49:32 DEBUG --- status code --- 06:49:32 DEBUG ] 06:49:32 DEBUG ] 06:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:32 DEBUG } 06:49:32 DEBUG } 06:49:32 DEBUG --- http response --- 06:49:32 DEBUG } 06:49:32 DEBUG } 06:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:49:32 DEBUG ----- output ----- 06:49:32 DEBUG { 06:49:32 DEBUG "status": "success", 06:49:32 DEBUG "data": { 06:49:32 DEBUG "resultType": "vector", 06:49:32 DEBUG "result": [ 06:49:32 DEBUG { 06:49:32 DEBUG "metric": { 06:49:32 DEBUG "pod": "ds-idrepo-0" 06:49:32 DEBUG }, 06:49:32 DEBUG "value": [ 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG [...] 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG "pod": "ds-cts-2" 06:49:32 DEBUG }, 06:49:32 DEBUG "value": [ 06:49:32 DEBUG 1712728172, 06:49:32 DEBUG "0" 06:49:32 DEBUG ] 06:49:32 DEBUG } 06:49:32 DEBUG ] 06:49:32 DEBUG } 06:49:32 DEBUG } 06:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712728172" 06:49:32 INFO [http_cmd]: http status code OK 06:49:32 DEBUG --- status code --- 06:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:32 DEBUG --- http response --- 06:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:49:32 DEBUG ----- output ----- 06:49:32 DEBUG { 06:49:32 DEBUG "status": "success", 06:49:32 DEBUG "data": { 06:49:32 DEBUG "resultType": "vector", 06:49:32 DEBUG "result": [ 06:49:32 DEBUG { 06:49:32 DEBUG "metric": { 06:49:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:49:32 DEBUG }, 06:49:32 DEBUG "value": [ 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG [...] 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:49:32 DEBUG }, 06:49:32 DEBUG "value": [ 06:49:32 DEBUG 1712728172, 06:49:32 DEBUG "0.43933482390750056" 06:49:32 DEBUG ] 06:49:32 DEBUG } 06:49:32 DEBUG ] 06:49:32 DEBUG } 06:49:32 DEBUG } 06:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728172" 06:49:32 INFO [http_cmd]: http status code OK 06:49:32 DEBUG --- status code --- 06:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:32 DEBUG --- http response --- 06:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:49:32 DEBUG ----- output ----- 06:49:32 DEBUG { 06:49:32 DEBUG "status": "success", 06:49:32 DEBUG "data": { 06:49:32 DEBUG "resultType": "vector", 06:49:32 DEBUG "result": [ 06:49:32 DEBUG { 06:49:32 DEBUG "metric": { 06:49:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:49:32 DEBUG }, 06:49:32 DEBUG "value": [ 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG [...] 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:49:32 DEBUG }, 06:49:32 DEBUG "value": [ 06:49:32 DEBUG 1712728172, 06:49:32 DEBUG "0" 06:49:32 DEBUG ] 06:49:32 DEBUG } 06:49:32 DEBUG ] 06:49:32 DEBUG } 06:49:32 DEBUG } 06:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728172" 06:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712728172" 06:49:32 INFO [http_cmd]: http status code OK 06:49:32 DEBUG --- status code --- 06:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:32 DEBUG --- http response --- 06:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:49:32 DEBUG ----- output ----- 06:49:32 DEBUG { 06:49:32 DEBUG "status": "success", 06:49:32 DEBUG "data": { 06:49:32 DEBUG "resultType": "vector", 06:49:32 DEBUG "result": [ 06:49:32 DEBUG { 06:49:32 DEBUG "metric": { 06:49:32 DEBUG "pod": "ds-idrepo-2" 06:49:32 DEBUG }, 06:49:32 DEBUG "value": [ 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG [...] 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG "pod": "ds-cts-2" 06:49:32 DEBUG }, 06:49:32 DEBUG "value": [ 06:49:32 DEBUG 1712728172, 06:49:32 DEBUG "0" 06:49:32 DEBUG ] 06:49:32 DEBUG } 06:49:32 DEBUG ] 06:49:32 DEBUG } 06:49:32 DEBUG } 06:49:32 INFO [http_cmd]: http status code OK 06:49:32 DEBUG --- status code --- 06:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:32 DEBUG --- http response --- 06:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:49:32 DEBUG ----- output ----- 06:49:32 DEBUG { 06:49:32 DEBUG "status": "success", 06:49:32 DEBUG "data": { 06:49:32 DEBUG "resultType": "vector", 06:49:32 DEBUG "result": [ 06:49:32 DEBUG { 06:49:32 DEBUG "metric": { 06:49:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:49:32 DEBUG }, 06:49:32 DEBUG "value": [ 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG [...] 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:49:32 DEBUG }, 06:49:32 DEBUG "value": [ 06:49:32 DEBUG 1712728172, 06:49:32 DEBUG "0.052" 06:49:32 DEBUG ] 06:49:32 DEBUG } 06:49:32 DEBUG ] 06:49:32 DEBUG } 06:49:32 DEBUG } 06:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728172" 06:49:32 INFO [http_cmd]: http status code OK 06:49:32 DEBUG --- status code --- 06:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:32 DEBUG --- http response --- 06:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:49:32 DEBUG ----- output ----- 06:49:32 DEBUG { 06:49:32 DEBUG "status": "success", 06:49:32 DEBUG "data": { 06:49:32 DEBUG "resultType": "vector", 06:49:32 DEBUG "result": [ 06:49:32 DEBUG { 06:49:32 DEBUG "metric": { 06:49:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:49:32 DEBUG }, 06:49:32 DEBUG "value": [ 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG [...] 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG 06:49:32 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:49:32 DEBUG }, 06:49:32 DEBUG "value": [ 06:49:32 DEBUG 1712728172, 06:49:32 DEBUG "0.01036658357346152" 06:49:32 DEBUG ] 06:49:32 DEBUG } 06:49:32 DEBUG ] 06:49:32 DEBUG } 06:49:32 DEBUG } 06:49:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728173" 06:49:33 INFO [http_cmd]: http status code OK 06:49:33 DEBUG --- status code --- 06:49:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:33 DEBUG --- http response --- 06:49:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:49:33 DEBUG ----- output ----- 06:49:33 DEBUG { 06:49:33 DEBUG "status": "success", 06:49:33 DEBUG "data": { 06:49:33 DEBUG "resultType": "vector", 06:49:33 DEBUG "result": [ 06:49:33 DEBUG { 06:49:33 DEBUG "metric": { 06:49:33 DEBUG "pod": "ds-idrepo-0" 06:49:33 DEBUG }, 06:49:33 DEBUG "value": [ 06:49:33 DEBUG 06:49:33 DEBUG 06:49:33 DEBUG 06:49:33 DEBUG [...] 06:49:33 DEBUG 06:49:33 DEBUG 06:49:33 DEBUG 06:49:33 DEBUG "pod": "ds-idrepo-1" 06:49:33 DEBUG }, 06:49:33 DEBUG "value": [ 06:49:33 DEBUG 1712728173, 06:49:33 DEBUG "0" 06:49:33 DEBUG ] 06:49:33 DEBUG } 06:49:33 DEBUG ] 06:49:33 DEBUG } 06:49:33 DEBUG } 06:49:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728173" 06:49:33 INFO [http_cmd]: http status code OK 06:49:33 DEBUG --- status code --- 06:49:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:33 DEBUG --- http response --- 06:49:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:49:33 DEBUG ----- output ----- 06:49:33 DEBUG { 06:49:33 DEBUG "status": "success", 06:49:33 DEBUG "data": { 06:49:33 DEBUG "resultType": "vector", 06:49:33 DEBUG "result": [ 06:49:33 DEBUG { 06:49:33 DEBUG "metric": { 06:49:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:49:33 DEBUG }, 06:49:33 DEBUG "value": [ 06:49:33 DEBUG 06:49:33 DEBUG 06:49:33 DEBUG 06:49:33 DEBUG [...] 06:49:33 DEBUG 06:49:33 DEBUG 06:49:33 DEBUG 06:49:33 DEBUG "pod": "am-654c4fcbcf-487cv" 06:49:33 DEBUG }, 06:49:33 DEBUG "value": [ 06:49:33 DEBUG 1712728173, 06:49:33 DEBUG "1459.4666666666667" 06:49:33 DEBUG ] 06:49:33 DEBUG } 06:49:33 DEBUG ] 06:49:33 DEBUG } 06:49:33 DEBUG } 06:49:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728173" 06:49:33 INFO [http_cmd]: http status code OK 06:49:33 DEBUG --- status code --- 06:49:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:33 DEBUG --- http response --- 06:49:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:49:33 DEBUG ----- output ----- 06:49:33 DEBUG { 06:49:33 DEBUG "status": "success", 06:49:33 DEBUG "data": { 06:49:33 DEBUG "resultType": "vector", 06:49:33 DEBUG "result": [ 06:49:33 DEBUG { 06:49:33 DEBUG "metric": { 06:49:33 DEBUG "pod": "ds-idrepo-0" 06:49:33 DEBUG }, 06:49:33 DEBUG "value": [ 06:49:33 DEBUG 06:49:33 DEBUG 06:49:33 DEBUG 06:49:33 DEBUG [...] 06:49:33 DEBUG 06:49:33 DEBUG 06:49:33 DEBUG 06:49:33 DEBUG "pod": "ds-idrepo-1" 06:49:33 DEBUG }, 06:49:33 DEBUG "value": [ 06:49:33 DEBUG 1712728173, 06:49:33 DEBUG "0" 06:49:33 DEBUG ] 06:49:33 DEBUG } 06:49:33 DEBUG ] 06:49:33 DEBUG } 06:49:33 DEBUG } 06:49:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728173" 06:49:33 INFO [http_cmd]: http status code OK 06:49:33 DEBUG --- status code --- 06:49:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:33 DEBUG --- http response --- 06:49:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:49:33 DEBUG ----- output ----- 06:49:33 DEBUG { 06:49:33 DEBUG "status": "success", 06:49:33 DEBUG "data": { 06:49:33 DEBUG "resultType": "vector", 06:49:33 DEBUG "result": [ 06:49:33 DEBUG { 06:49:33 DEBUG "metric": { 06:49:33 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:49:33 DEBUG }, 06:49:33 DEBUG "value": [ 06:49:33 DEBUG 06:49:33 DEBUG 06:49:33 DEBUG 06:49:33 DEBUG [...] 06:49:33 DEBUG 06:49:33 DEBUG 06:49:33 DEBUG 06:49:33 DEBUG "pod": "login-ui-5f964fb894-5zpvq" 06:49:33 DEBUG }, 06:49:33 DEBUG "value": [ 06:49:33 DEBUG 1712728173, 06:49:33 DEBUG "0" 06:49:33 DEBUG ] 06:49:33 DEBUG } 06:49:33 DEBUG ] 06:49:33 DEBUG } 06:49:33 DEBUG } 06:49:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712728173" 06:49:33 INFO [http_cmd]: http status code OK 06:49:33 DEBUG --- status code --- 06:49:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:33 DEBUG --- http response --- 06:49:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:49:33 DEBUG ----- output ----- 06:49:33 DEBUG { 06:49:33 DEBUG "status": "success", 06:49:33 DEBUG "data": { 06:49:33 DEBUG "resultType": "vector", 06:49:33 DEBUG "result": [ 06:49:33 DEBUG { 06:49:33 DEBUG "metric": { 06:49:33 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:49:33 DEBUG }, 06:49:33 DEBUG "value": [ 06:49:33 DEBUG 06:49:33 DEBUG 06:49:33 DEBUG 06:49:33 DEBUG [...] 06:49:33 DEBUG 06:49:33 DEBUG 06:49:33 DEBUG 06:49:33 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:49:33 DEBUG }, 06:49:33 DEBUG "value": [ 06:49:33 DEBUG 1712728173, 06:49:33 DEBUG "2488573952" 06:49:33 DEBUG ] 06:49:33 DEBUG } 06:49:33 DEBUG ] 06:49:33 DEBUG } 06:49:33 DEBUG } 06:49:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728173" 06:49:33 INFO [http_cmd]: http status code OK 06:49:33 DEBUG --- status code --- 06:49:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:33 DEBUG --- http response --- 06:49:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:49:33 DEBUG ----- output ----- 06:49:33 DEBUG { 06:49:33 DEBUG "status": "success", 06:49:33 DEBUG "data": { 06:49:33 DEBUG "resultType": "vector", 06:49:33 DEBUG "result": [ 06:49:33 DEBUG { 06:49:33 DEBUG "metric": { 06:49:33 DEBUG "instance": "10.142.0.25:9100" 06:49:33 DEBUG }, 06:49:33 DEBUG "value": [ 06:49:33 DEBUG 06:49:33 DEBUG 06:49:33 DEBUG 06:49:33 DEBUG [...] 06:49:33 DEBUG 06:49:33 DEBUG 06:49:33 DEBUG 06:49:33 DEBUG "instance": "10.142.0.77:9100" 06:49:33 DEBUG }, 06:49:33 DEBUG "value": [ 06:49:33 DEBUG 1712728173, 06:49:33 DEBUG "8052567.966666667" 06:49:33 DEBUG ] 06:49:33 DEBUG } 06:49:33 DEBUG ] 06:49:33 DEBUG } 06:49:33 DEBUG } 06:49:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728173" 06:49:33 INFO [http_cmd]: http status code OK 06:49:33 DEBUG --- status code --- 06:49:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:33 DEBUG --- http response --- 06:49:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:49:33 DEBUG ----- output ----- 06:49:33 DEBUG { 06:49:33 DEBUG "status": "success", 06:49:33 DEBUG "data": { 06:49:33 DEBUG "resultType": "vector", 06:49:33 DEBUG "result": [ 06:49:33 DEBUG { 06:49:33 DEBUG "metric": { 06:49:33 DEBUG "instance": "10.142.0.25:9100" 06:49:33 DEBUG }, 06:49:33 DEBUG "value": [ 06:49:33 DEBUG 06:49:33 DEBUG 06:49:33 DEBUG 06:49:33 DEBUG [...] 06:49:33 DEBUG 06:49:33 DEBUG 06:49:33 DEBUG 06:49:33 DEBUG "instance": "10.142.0.77:9100" 06:49:33 DEBUG }, 06:49:33 DEBUG "value": [ 06:49:33 DEBUG 1712728173, 06:49:33 DEBUG "0.004333333333333419" 06:49:33 DEBUG ] 06:49:33 DEBUG } 06:49:33 DEBUG ] 06:49:33 DEBUG } 06:49:33 DEBUG } 06:49:54 INFO 06:49:54 INFO [loop_until]: kubectl --namespace=xlou top pods 06:49:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:49:54 INFO [loop_until]: OK (rc = 0) 06:49:54 DEBUG --- stdout --- 06:49:54 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 9Mi am-654c4fcbcf-487cv 688m 8896Mi am-654c4fcbcf-9scrj 446m 8397Mi am-654c4fcbcf-v5qb5 247m 8114Mi ds-cts-0 984m 12723Mi ds-cts-1 886m 12921Mi ds-cts-2 844m 11473Mi ds-idrepo-0 675m 5615Mi ds-idrepo-1 344m 5798Mi ds-idrepo-2 356m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 5Mi idm-6bfb7c96cc-qklk9 5m 1199Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 5m 73Mi login-ui-5f964fb894-5zpvq 1m 4Mi overseer-0-6f8784665d-zq5hb 612m 1715Mi 06:49:54 DEBUG --- stderr --- 06:49:54 DEBUG 06:50:03 INFO 06:50:03 INFO [loop_until]: kubectl --namespace=xlou top node 06:50:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:50:03 INFO [loop_until]: OK (rc = 0) 06:50:03 DEBUG --- stdout --- 06:50:03 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 100m 0% 2317Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 338m 2% 9004Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 80m 0% 3255Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 199m 1% 6510Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 81m 0% 9294Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 650m 4% 9742Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 76m 0% 858Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 671m 4% 6307Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 1140m 7% 13458Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 235m 1% 12208Mi 20% gke-xlou-cdm-ds-32e4dcb1-gxzj 395m 2% 6400Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 852m 5% 13652Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 359m 2% 6369Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 589m 3% 5345Mi 9% 06:50:03 DEBUG --- stderr --- 06:50:03 DEBUG 06:50:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728230" 06:50:30 INFO [http_cmd]: http status code OK 06:50:30 DEBUG --- status code --- 06:50:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:30 DEBUG --- http response --- 06:50:30 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:50:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712728230" 06:50:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728230" 06:50:30 ERROR [http_cmd]: ERROR 06:50:30 ERROR --- status code --- 06:50:30 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:50:30 ERROR --- http response --- 06:50:30 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:50:30 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:50:30 INFO [http_cmd]: http status code OK 06:50:30 DEBUG --- status code --- 06:50:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:30 DEBUG --- http response --- 06:50:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:50:30 DEBUG ----- output ----- 06:50:30 DEBUG { 06:50:30 DEBUG "status": "success", 06:50:30 DEBUG "data": { 06:50:30 DEBUG "resultType": "vector", 06:50:30 DEBUG "result": [ 06:50:30 DEBUG { 06:50:30 DEBUG "metric": { 06:50:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:50:30 DEBUG }, 06:50:30 DEBUG "value": [ 06:50:30 DEBUG 06:50:30 DEBUG 06:50:30 DEBUG 06:50:30 DEBUG [...] 06:50:30 DEBUG 06:50:30 DEBUG 06:50:30 DEBUG 06:50:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:50:30 DEBUG }, 06:50:30 DEBUG "value": [ 06:50:30 DEBUG 1712728230, 06:50:30 DEBUG "0" 06:50:30 DEBUG ] 06:50:30 DEBUG } 06:50:30 DEBUG ] 06:50:30 DEBUG } 06:50:30 DEBUG } 06:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728232" 06:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728232" 06:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728232" 06:50:32 INFO [http_cmd]: http status code OK 06:50:32 INFO [http_cmd]: http status code OK 06:50:32 INFO [http_cmd]: http status code OK 06:50:32 DEBUG --- status code --- 06:50:32 DEBUG --- status code --- 06:50:32 DEBUG --- status code --- 06:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:32 DEBUG --- http response --- 06:50:32 DEBUG --- http response --- 06:50:32 DEBUG --- http response --- 06:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:50:32 DEBUG ----- output ----- 06:50:32 DEBUG ----- output ----- 06:50:32 DEBUG ----- output ----- 06:50:32 DEBUG { 06:50:32 DEBUG { 06:50:32 DEBUG { 06:50:32 DEBUG "status": "success", 06:50:32 DEBUG "status": "success", 06:50:32 DEBUG "status": "success", 06:50:32 DEBUG "data": { 06:50:32 DEBUG "data": { 06:50:32 DEBUG "data": { 06:50:32 DEBUG "resultType": "vector", 06:50:32 DEBUG "resultType": "vector", 06:50:32 DEBUG "resultType": "vector", 06:50:32 DEBUG "result": [ 06:50:32 DEBUG "result": [ 06:50:32 DEBUG "result": [ 06:50:32 DEBUG { 06:50:32 DEBUG { 06:50:32 DEBUG { 06:50:32 DEBUG "metric": { 06:50:32 DEBUG "metric": { 06:50:32 DEBUG "metric": { 06:50:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:50:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:50:32 DEBUG "instance": "10.142.0.25:9100" 06:50:32 DEBUG }, 06:50:32 DEBUG }, 06:50:32 DEBUG }, 06:50:32 DEBUG "value": [ 06:50:32 DEBUG "value": [ 06:50:32 DEBUG "value": [ 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG [...] 06:50:32 DEBUG [...] 06:50:32 DEBUG [...] 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728232" 06:50:32 DEBUG 06:50:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:50:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:50:32 DEBUG "instance": "10.142.0.77:9100" 06:50:32 DEBUG }, 06:50:32 DEBUG }, 06:50:32 DEBUG }, 06:50:32 DEBUG "value": [ 06:50:32 DEBUG "value": [ 06:50:32 DEBUG "value": [ 06:50:32 DEBUG 1712728232, 06:50:32 DEBUG 1712728232, 06:50:32 DEBUG 1712728232, 06:50:32 DEBUG "0" 06:50:32 DEBUG "0" 06:50:32 DEBUG "24973.699999999997" 06:50:32 DEBUG ] 06:50:32 DEBUG ] 06:50:32 DEBUG ] 06:50:32 DEBUG } 06:50:32 DEBUG } 06:50:32 DEBUG } 06:50:32 DEBUG ] 06:50:32 DEBUG ] 06:50:32 DEBUG ] 06:50:32 DEBUG } 06:50:32 DEBUG } 06:50:32 DEBUG } 06:50:32 DEBUG } 06:50:32 DEBUG } 06:50:32 DEBUG } 06:50:32 INFO [http_cmd]: http status code OK 06:50:32 DEBUG --- status code --- 06:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:32 DEBUG --- http response --- 06:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:50:32 DEBUG ----- output ----- 06:50:32 DEBUG { 06:50:32 DEBUG "status": "success", 06:50:32 DEBUG "data": { 06:50:32 DEBUG "resultType": "vector", 06:50:32 DEBUG "result": [ 06:50:32 DEBUG { 06:50:32 DEBUG "metric": { 06:50:32 DEBUG "pod": "ds-idrepo-0" 06:50:32 DEBUG }, 06:50:32 DEBUG "value": [ 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG [...] 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG "pod": "ds-idrepo-1" 06:50:32 DEBUG }, 06:50:32 DEBUG "value": [ 06:50:32 DEBUG 1712728232, 06:50:32 DEBUG "0" 06:50:32 DEBUG ] 06:50:32 DEBUG } 06:50:32 DEBUG ] 06:50:32 DEBUG } 06:50:32 DEBUG } 06:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728232" 06:50:32 INFO [http_cmd]: http status code OK 06:50:32 DEBUG --- status code --- 06:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:32 DEBUG --- http response --- 06:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:50:32 DEBUG ----- output ----- 06:50:32 DEBUG { 06:50:32 DEBUG "status": "success", 06:50:32 DEBUG "data": { 06:50:32 DEBUG "resultType": "vector", 06:50:32 DEBUG "result": [ 06:50:32 DEBUG { 06:50:32 DEBUG "metric": { 06:50:32 DEBUG "instance": "10.142.0.25:9100" 06:50:32 DEBUG }, 06:50:32 DEBUG "value": [ 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG [...] 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG "instance": "10.142.0.77:9100" 06:50:32 DEBUG }, 06:50:32 DEBUG "value": [ 06:50:32 DEBUG 1712728232, 06:50:32 DEBUG "438078118912" 06:50:32 DEBUG ] 06:50:32 DEBUG } 06:50:32 DEBUG ] 06:50:32 DEBUG } 06:50:32 DEBUG } 06:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728232" 06:50:32 INFO [http_cmd]: http status code OK 06:50:32 DEBUG --- status code --- 06:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:32 DEBUG --- http response --- 06:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:50:32 DEBUG ----- output ----- 06:50:32 DEBUG { 06:50:32 DEBUG "status": "success", 06:50:32 DEBUG "data": { 06:50:32 DEBUG "resultType": "vector", 06:50:32 DEBUG "result": [ 06:50:32 DEBUG { 06:50:32 DEBUG "metric": { 06:50:32 DEBUG "instance": "10.142.0.25:9100" 06:50:32 DEBUG }, 06:50:32 DEBUG "value": [ 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG [...] 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG "instance": "10.142.0.77:9100" 06:50:32 DEBUG }, 06:50:32 DEBUG "value": [ 06:50:32 DEBUG 1712728232, 06:50:32 DEBUG "0.0005333333333323272" 06:50:32 DEBUG ] 06:50:32 DEBUG } 06:50:32 DEBUG ] 06:50:32 DEBUG } 06:50:32 DEBUG } 06:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728232" 06:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728232" 06:50:32 INFO [http_cmd]: http status code OK 06:50:32 DEBUG --- status code --- 06:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:32 DEBUG --- http response --- 06:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:50:32 DEBUG ----- output ----- 06:50:32 DEBUG { 06:50:32 DEBUG "status": "success", 06:50:32 DEBUG "data": { 06:50:32 DEBUG "resultType": "vector", 06:50:32 DEBUG "result": [ 06:50:32 DEBUG { 06:50:32 DEBUG "metric": { 06:50:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:50:32 DEBUG }, 06:50:32 DEBUG "value": [ 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG [...] 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:50:32 DEBUG }, 06:50:32 DEBUG "value": [ 06:50:32 DEBUG 1712728232, 06:50:32 DEBUG "0" 06:50:32 DEBUG ] 06:50:32 DEBUG } 06:50:32 DEBUG ] 06:50:32 DEBUG } 06:50:32 DEBUG } 06:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728232" 06:50:32 INFO [http_cmd]: http status code OK 06:50:32 DEBUG --- status code --- 06:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:32 DEBUG --- http response --- 06:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:50:32 DEBUG ----- output ----- 06:50:32 DEBUG { 06:50:32 DEBUG "status": "success", 06:50:32 DEBUG "data": { 06:50:32 DEBUG "resultType": "vector", 06:50:32 DEBUG "result": [ 06:50:32 DEBUG { 06:50:32 DEBUG "metric": { 06:50:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:50:32 DEBUG }, 06:50:32 DEBUG "value": [ 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG [...] 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:50:32 DEBUG }, 06:50:32 DEBUG "value": [ 06:50:32 DEBUG 1712728232, 06:50:32 DEBUG "0" 06:50:32 DEBUG ] 06:50:32 DEBUG } 06:50:32 DEBUG ] 06:50:32 DEBUG } 06:50:32 DEBUG } 06:50:32 INFO [http_cmd]: http status code OK 06:50:32 DEBUG --- status code --- 06:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:32 DEBUG --- http response --- 06:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:50:32 DEBUG ----- output ----- 06:50:32 DEBUG { 06:50:32 DEBUG "status": "success", 06:50:32 DEBUG "data": { 06:50:32 DEBUG "resultType": "vector", 06:50:32 DEBUG "result": [ 06:50:32 DEBUG { 06:50:32 DEBUG "metric": { 06:50:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:50:32 DEBUG }, 06:50:32 DEBUG "value": [ 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG [...] 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:50:32 DEBUG }, 06:50:32 DEBUG "value": [ 06:50:32 DEBUG 1712728232, 06:50:32 DEBUG "83825810.04693873" 06:50:32 DEBUG ] 06:50:32 DEBUG } 06:50:32 DEBUG ] 06:50:32 DEBUG } 06:50:32 DEBUG } 06:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728232" 06:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728232" 06:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712728232" 06:50:32 INFO [http_cmd]: http status code OK 06:50:32 DEBUG --- status code --- 06:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:32 DEBUG --- http response --- 06:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:50:32 INFO [http_cmd]: http status code OK 06:50:32 INFO [http_cmd]: http status code OK 06:50:32 DEBUG ----- output ----- 06:50:32 DEBUG --- status code --- 06:50:32 DEBUG --- status code --- 06:50:32 DEBUG { 06:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:32 DEBUG "status": "success", 06:50:32 DEBUG --- http response --- 06:50:32 DEBUG --- http response --- 06:50:32 DEBUG "data": { 06:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:50:32 DEBUG "resultType": "vector", 06:50:32 DEBUG ----- output ----- 06:50:32 DEBUG ----- output ----- 06:50:32 DEBUG "result": [ 06:50:32 DEBUG { 06:50:32 DEBUG { 06:50:32 DEBUG { 06:50:32 DEBUG "status": "success", 06:50:32 DEBUG "status": "success", 06:50:32 DEBUG "metric": { 06:50:32 DEBUG "data": { 06:50:32 DEBUG "data": { 06:50:32 DEBUG "pod": "ds-idrepo-0" 06:50:32 DEBUG "resultType": "vector", 06:50:32 DEBUG "resultType": "vector", 06:50:32 DEBUG }, 06:50:32 DEBUG "result": [ 06:50:32 DEBUG "result": [ 06:50:32 DEBUG "value": [ 06:50:32 DEBUG { 06:50:32 DEBUG { 06:50:32 DEBUG 06:50:32 DEBUG "metric": { 06:50:32 DEBUG "metric": { 06:50:32 DEBUG 06:50:32 DEBUG "pod": "ds-idrepo-0" 06:50:32 DEBUG "pod": "ds-idrepo-0" 06:50:32 DEBUG 06:50:32 DEBUG }, 06:50:32 DEBUG }, 06:50:32 DEBUG [...] 06:50:32 DEBUG "value": [ 06:50:32 DEBUG "value": [ 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG "pod": "ds-cts-2" 06:50:32 DEBUG [...] 06:50:32 DEBUG [...] 06:50:32 DEBUG }, 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG "value": [ 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 1712728232, 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG "29.599999999999998" 06:50:32 DEBUG "pod": "ds-cts-2" 06:50:32 DEBUG "pod": "ds-cts-2" 06:50:32 DEBUG ] 06:50:32 DEBUG }, 06:50:32 DEBUG }, 06:50:32 DEBUG } 06:50:32 DEBUG "value": [ 06:50:32 DEBUG "value": [ 06:50:32 DEBUG ] 06:50:32 DEBUG 1712728232, 06:50:32 DEBUG 1712728232, 06:50:32 DEBUG } 06:50:32 DEBUG "0" 06:50:32 DEBUG "0" 06:50:32 DEBUG } 06:50:32 DEBUG ] 06:50:32 DEBUG ] 06:50:32 DEBUG } 06:50:32 DEBUG } 06:50:32 DEBUG ] 06:50:32 DEBUG ] 06:50:32 DEBUG } 06:50:32 DEBUG } 06:50:32 DEBUG } 06:50:32 DEBUG } 06:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712728232" 06:50:32 INFO [http_cmd]: http status code OK 06:50:32 DEBUG --- status code --- 06:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:32 DEBUG --- http response --- 06:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:50:32 DEBUG ----- output ----- 06:50:32 DEBUG { 06:50:32 DEBUG "status": "success", 06:50:32 DEBUG "data": { 06:50:32 DEBUG "resultType": "vector", 06:50:32 DEBUG "result": [ 06:50:32 DEBUG { 06:50:32 DEBUG "metric": { 06:50:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:50:32 DEBUG }, 06:50:32 DEBUG "value": [ 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG [...] 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:50:32 DEBUG }, 06:50:32 DEBUG "value": [ 06:50:32 DEBUG 1712728232, 06:50:32 DEBUG "1.1312395006190648" 06:50:32 DEBUG ] 06:50:32 DEBUG } 06:50:32 DEBUG ] 06:50:32 DEBUG } 06:50:32 DEBUG } 06:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728232" 06:50:32 INFO [http_cmd]: http status code OK 06:50:32 DEBUG --- status code --- 06:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:32 DEBUG --- http response --- 06:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:50:32 DEBUG ----- output ----- 06:50:32 DEBUG { 06:50:32 DEBUG "status": "success", 06:50:32 DEBUG "data": { 06:50:32 DEBUG "resultType": "vector", 06:50:32 DEBUG "result": [ 06:50:32 DEBUG { 06:50:32 DEBUG "metric": { 06:50:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:50:32 DEBUG }, 06:50:32 DEBUG "value": [ 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG [...] 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:50:32 DEBUG }, 06:50:32 DEBUG "value": [ 06:50:32 DEBUG 1712728232, 06:50:32 DEBUG "0" 06:50:32 DEBUG ] 06:50:32 DEBUG } 06:50:32 DEBUG ] 06:50:32 DEBUG } 06:50:32 DEBUG } 06:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712728232" 06:50:32 INFO [http_cmd]: http status code OK 06:50:32 DEBUG --- status code --- 06:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:32 DEBUG --- http response --- 06:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:50:32 DEBUG ----- output ----- 06:50:32 DEBUG { 06:50:32 DEBUG "status": "success", 06:50:32 DEBUG "data": { 06:50:32 DEBUG "resultType": "vector", 06:50:32 DEBUG "result": [ 06:50:32 DEBUG { 06:50:32 DEBUG "metric": { 06:50:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:50:32 DEBUG }, 06:50:32 DEBUG "value": [ 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG [...] 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:50:32 DEBUG }, 06:50:32 DEBUG "value": [ 06:50:32 DEBUG 1712728232, 06:50:32 DEBUG "0.052" 06:50:32 DEBUG ] 06:50:32 DEBUG } 06:50:32 DEBUG ] 06:50:32 DEBUG } 06:50:32 DEBUG } 06:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728232" 06:50:32 INFO [http_cmd]: http status code OK 06:50:32 DEBUG --- status code --- 06:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:32 DEBUG --- http response --- 06:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:50:32 DEBUG ----- output ----- 06:50:32 DEBUG { 06:50:32 DEBUG "status": "success", 06:50:32 DEBUG "data": { 06:50:32 DEBUG "resultType": "vector", 06:50:32 DEBUG "result": [ 06:50:32 DEBUG { 06:50:32 DEBUG "metric": { 06:50:32 DEBUG "pod": "ds-idrepo-2" 06:50:32 DEBUG }, 06:50:32 DEBUG "value": [ 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG [...] 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG "pod": "ds-cts-2" 06:50:32 DEBUG }, 06:50:32 DEBUG "value": [ 06:50:32 DEBUG 1712728232, 06:50:32 DEBUG "0" 06:50:32 DEBUG ] 06:50:32 DEBUG } 06:50:32 DEBUG ] 06:50:32 DEBUG } 06:50:32 DEBUG } 06:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728232" 06:50:32 INFO [http_cmd]: http status code OK 06:50:32 DEBUG --- status code --- 06:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:32 DEBUG --- http response --- 06:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:50:32 DEBUG ----- output ----- 06:50:32 DEBUG { 06:50:32 DEBUG "status": "success", 06:50:32 DEBUG "data": { 06:50:32 DEBUG "resultType": "vector", 06:50:32 DEBUG "result": [ 06:50:32 DEBUG { 06:50:32 DEBUG "metric": { 06:50:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:50:32 DEBUG }, 06:50:32 DEBUG "value": [ 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG [...] 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG 06:50:32 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:50:32 DEBUG }, 06:50:32 DEBUG "value": [ 06:50:32 DEBUG 1712728232, 06:50:32 DEBUG "1.0391438375872395" 06:50:32 DEBUG ] 06:50:32 DEBUG } 06:50:32 DEBUG ] 06:50:32 DEBUG } 06:50:32 DEBUG } 06:50:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728233" 06:50:33 INFO [http_cmd]: http status code OK 06:50:33 DEBUG --- status code --- 06:50:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:33 DEBUG --- http response --- 06:50:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:50:33 DEBUG ----- output ----- 06:50:33 DEBUG { 06:50:33 DEBUG "status": "success", 06:50:33 DEBUG "data": { 06:50:33 DEBUG "resultType": "vector", 06:50:33 DEBUG "result": [ 06:50:33 DEBUG { 06:50:33 DEBUG "metric": { 06:50:33 DEBUG "pod": "ds-idrepo-0" 06:50:33 DEBUG }, 06:50:33 DEBUG "value": [ 06:50:33 DEBUG 06:50:33 DEBUG 06:50:33 DEBUG 06:50:33 DEBUG [...] 06:50:33 DEBUG 06:50:33 DEBUG 06:50:33 DEBUG 06:50:33 DEBUG "pod": "ds-idrepo-1" 06:50:33 DEBUG }, 06:50:33 DEBUG "value": [ 06:50:33 DEBUG 1712728233, 06:50:33 DEBUG "0" 06:50:33 DEBUG ] 06:50:33 DEBUG } 06:50:33 DEBUG ] 06:50:33 DEBUG } 06:50:33 DEBUG } 06:50:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728233" 06:50:33 INFO [http_cmd]: http status code OK 06:50:33 DEBUG --- status code --- 06:50:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:33 DEBUG --- http response --- 06:50:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:50:33 DEBUG ----- output ----- 06:50:33 DEBUG { 06:50:33 DEBUG "status": "success", 06:50:33 DEBUG "data": { 06:50:33 DEBUG "resultType": "vector", 06:50:33 DEBUG "result": [ 06:50:33 DEBUG { 06:50:33 DEBUG "metric": { 06:50:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:50:33 DEBUG }, 06:50:33 DEBUG "value": [ 06:50:33 DEBUG 06:50:33 DEBUG 06:50:33 DEBUG 06:50:33 DEBUG [...] 06:50:33 DEBUG 06:50:33 DEBUG 06:50:33 DEBUG 06:50:33 DEBUG "pod": "am-654c4fcbcf-487cv" 06:50:33 DEBUG }, 06:50:33 DEBUG "value": [ 06:50:33 DEBUG 1712728233, 06:50:33 DEBUG "0" 06:50:33 DEBUG ] 06:50:33 DEBUG } 06:50:33 DEBUG ] 06:50:33 DEBUG } 06:50:33 DEBUG } 06:50:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728233" 06:50:33 INFO [http_cmd]: http status code OK 06:50:33 DEBUG --- status code --- 06:50:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:33 DEBUG --- http response --- 06:50:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:50:33 DEBUG ----- output ----- 06:50:33 DEBUG { 06:50:33 DEBUG "status": "success", 06:50:33 DEBUG "data": { 06:50:33 DEBUG "resultType": "vector", 06:50:33 DEBUG "result": [ 06:50:33 DEBUG { 06:50:33 DEBUG "metric": { 06:50:33 DEBUG "pod": "ds-idrepo-0" 06:50:33 DEBUG }, 06:50:33 DEBUG "value": [ 06:50:33 DEBUG 06:50:33 DEBUG 06:50:33 DEBUG 06:50:33 DEBUG [...] 06:50:33 DEBUG 06:50:33 DEBUG 06:50:33 DEBUG 06:50:33 DEBUG "pod": "ds-idrepo-1" 06:50:33 DEBUG }, 06:50:33 DEBUG "value": [ 06:50:33 DEBUG 1712728233, 06:50:33 DEBUG "0" 06:50:33 DEBUG ] 06:50:33 DEBUG } 06:50:33 DEBUG ] 06:50:33 DEBUG } 06:50:33 DEBUG } 06:50:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728233" 06:50:33 INFO [http_cmd]: http status code OK 06:50:33 DEBUG --- status code --- 06:50:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:33 DEBUG --- http response --- 06:50:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:50:33 DEBUG ----- output ----- 06:50:33 DEBUG { 06:50:33 DEBUG "status": "success", 06:50:33 DEBUG "data": { 06:50:33 DEBUG "resultType": "vector", 06:50:33 DEBUG "result": [ 06:50:33 DEBUG { 06:50:33 DEBUG "metric": { 06:50:33 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:50:33 DEBUG }, 06:50:33 DEBUG "value": [ 06:50:33 DEBUG 06:50:33 DEBUG 06:50:33 DEBUG 06:50:33 DEBUG [...] 06:50:33 DEBUG 06:50:33 DEBUG 06:50:33 DEBUG 06:50:33 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:50:33 DEBUG }, 06:50:33 DEBUG "value": [ 06:50:33 DEBUG 1712728233, 06:50:33 DEBUG "757.0609481336571" 06:50:33 DEBUG ] 06:50:33 DEBUG } 06:50:33 DEBUG ] 06:50:33 DEBUG } 06:50:33 DEBUG } 06:50:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712728233" 06:50:33 INFO [http_cmd]: http status code OK 06:50:33 DEBUG --- status code --- 06:50:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:33 DEBUG --- http response --- 06:50:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:50:33 DEBUG ----- output ----- 06:50:33 DEBUG { 06:50:33 DEBUG "status": "success", 06:50:33 DEBUG "data": { 06:50:33 DEBUG "resultType": "vector", 06:50:33 DEBUG "result": [ 06:50:33 DEBUG { 06:50:33 DEBUG "metric": { 06:50:33 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:50:33 DEBUG }, 06:50:33 DEBUG "value": [ 06:50:33 DEBUG 06:50:33 DEBUG 06:50:33 DEBUG 06:50:33 DEBUG [...] 06:50:33 DEBUG 06:50:33 DEBUG 06:50:33 DEBUG 06:50:33 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:50:33 DEBUG }, 06:50:33 DEBUG "value": [ 06:50:33 DEBUG 1712728233, 06:50:33 DEBUG "2486075392" 06:50:33 DEBUG ] 06:50:33 DEBUG } 06:50:33 DEBUG ] 06:50:33 DEBUG } 06:50:33 DEBUG } 06:50:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728233" 06:50:33 INFO [http_cmd]: http status code OK 06:50:33 DEBUG --- status code --- 06:50:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:33 DEBUG --- http response --- 06:50:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:50:33 DEBUG ----- output ----- 06:50:33 DEBUG { 06:50:33 DEBUG "status": "success", 06:50:33 DEBUG "data": { 06:50:33 DEBUG "resultType": "vector", 06:50:33 DEBUG "result": [ 06:50:33 DEBUG { 06:50:33 DEBUG "metric": { 06:50:33 DEBUG "instance": "10.142.0.25:9100" 06:50:33 DEBUG }, 06:50:33 DEBUG "value": [ 06:50:33 DEBUG 06:50:33 DEBUG 06:50:33 DEBUG 06:50:33 DEBUG [...] 06:50:33 DEBUG 06:50:33 DEBUG 06:50:33 DEBUG 06:50:33 DEBUG "instance": "10.142.0.77:9100" 06:50:33 DEBUG }, 06:50:33 DEBUG "value": [ 06:50:33 DEBUG 1712728233, 06:50:33 DEBUG "26139.3" 06:50:33 DEBUG ] 06:50:33 DEBUG } 06:50:33 DEBUG ] 06:50:33 DEBUG } 06:50:33 DEBUG } 06:50:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728233" 06:50:33 INFO [http_cmd]: http status code OK 06:50:33 DEBUG --- status code --- 06:50:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:33 DEBUG --- http response --- 06:50:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:50:33 DEBUG ----- output ----- 06:50:33 DEBUG { 06:50:33 DEBUG "status": "success", 06:50:33 DEBUG "data": { 06:50:33 DEBUG "resultType": "vector", 06:50:33 DEBUG "result": [ 06:50:33 DEBUG { 06:50:33 DEBUG "metric": { 06:50:33 DEBUG "instance": "10.142.0.25:9100" 06:50:33 DEBUG }, 06:50:33 DEBUG "value": [ 06:50:33 DEBUG 06:50:33 DEBUG 06:50:33 DEBUG 06:50:33 DEBUG [...] 06:50:33 DEBUG 06:50:33 DEBUG 06:50:33 DEBUG 06:50:33 DEBUG "instance": "10.142.0.77:9100" 06:50:33 DEBUG }, 06:50:33 DEBUG "value": [ 06:50:33 DEBUG 1712728233, 06:50:33 DEBUG "0.0003333333333330302" 06:50:33 DEBUG ] 06:50:33 DEBUG } 06:50:33 DEBUG ] 06:50:33 DEBUG } 06:50:33 DEBUG } 06:50:54 INFO 06:50:54 INFO [loop_until]: kubectl --namespace=xlou top pods 06:50:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:50:54 INFO [loop_until]: OK (rc = 0) 06:50:54 DEBUG --- stdout --- 06:50:54 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 9Mi am-654c4fcbcf-487cv 10m 8870Mi am-654c4fcbcf-9scrj 8m 8378Mi am-654c4fcbcf-v5qb5 8m 8096Mi ds-cts-0 15m 12723Mi ds-cts-1 18m 12921Mi ds-cts-2 18m 11473Mi ds-idrepo-0 15m 5615Mi ds-idrepo-1 12m 5677Mi ds-idrepo-2 12m 5704Mi end-user-ui-6d9fd5487b-2s47b 1m 5Mi idm-6bfb7c96cc-qklk9 6m 1199Mi idm-6bfb7c96cc-vsb58 4m 2214Mi lodemon-b776b7d88-thrqf 4m 73Mi login-ui-5f964fb894-5zpvq 1m 4Mi overseer-0-6f8784665d-zq5hb 703m 1451Mi 06:50:54 DEBUG --- stderr --- 06:50:54 DEBUG 06:51:03 INFO 06:51:03 INFO [loop_until]: kubectl --namespace=xlou top node 06:51:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:51:03 INFO [loop_until]: OK (rc = 0) 06:51:03 DEBUG --- stdout --- 06:51:03 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 100m 0% 2314Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 70m 0% 8987Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 77m 0% 3256Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 216m 1% 6505Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 68m 0% 9279Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 66m 0% 9731Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 75m 0% 861Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 71m 0% 6309Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 75m 0% 13456Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 70m 0% 12208Mi 20% gke-xlou-cdm-ds-32e4dcb1-gxzj 66m 0% 6399Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 70m 0% 13655Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 66m 0% 6368Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 511m 3% 5078Mi 8% 06:51:03 DEBUG --- stderr --- 06:51:03 DEBUG 06:51:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712728290" 06:51:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728290" 06:51:30 ERROR [http_cmd]: ERROR 06:51:30 ERROR --- status code --- 06:51:30 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:51:30 ERROR --- http response --- 06:51:30 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:51:30 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:51:30 INFO [http_cmd]: http status code OK 06:51:30 DEBUG --- status code --- 06:51:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:30 DEBUG --- http response --- 06:51:30 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:51:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728290" 06:51:30 INFO [http_cmd]: http status code OK 06:51:30 DEBUG --- status code --- 06:51:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:30 DEBUG --- http response --- 06:51:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:51:30 DEBUG ----- output ----- 06:51:30 DEBUG { 06:51:30 DEBUG "status": "success", 06:51:30 DEBUG "data": { 06:51:30 DEBUG "resultType": "vector", 06:51:30 DEBUG "result": [ 06:51:30 DEBUG { 06:51:30 DEBUG "metric": { 06:51:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:51:30 DEBUG }, 06:51:30 DEBUG "value": [ 06:51:30 DEBUG 06:51:30 DEBUG 06:51:30 DEBUG 06:51:30 DEBUG [...] 06:51:30 DEBUG 06:51:30 DEBUG 06:51:30 DEBUG 06:51:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:51:30 DEBUG }, 06:51:30 DEBUG "value": [ 06:51:30 DEBUG 1712728290, 06:51:30 DEBUG "0" 06:51:30 DEBUG ] 06:51:30 DEBUG } 06:51:30 DEBUG ] 06:51:30 DEBUG } 06:51:30 DEBUG } 06:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728292" 06:51:32 INFO [http_cmd]: http status code OK 06:51:32 DEBUG --- status code --- 06:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:32 DEBUG --- http response --- 06:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:51:32 DEBUG ----- output ----- 06:51:32 DEBUG { 06:51:32 DEBUG "status": "success", 06:51:32 DEBUG "data": { 06:51:32 DEBUG "resultType": "vector", 06:51:32 DEBUG "result": [ 06:51:32 DEBUG { 06:51:32 DEBUG "metric": { 06:51:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:51:32 DEBUG }, 06:51:32 DEBUG "value": [ 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG [...] 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:51:32 DEBUG }, 06:51:32 DEBUG "value": [ 06:51:32 DEBUG 1712728292, 06:51:32 DEBUG "0" 06:51:32 DEBUG ] 06:51:32 DEBUG } 06:51:32 DEBUG ] 06:51:32 DEBUG } 06:51:32 DEBUG } 06:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728292" 06:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728292" 06:51:32 INFO [http_cmd]: http status code OK 06:51:32 INFO [http_cmd]: http status code OK 06:51:32 DEBUG --- status code --- 06:51:32 DEBUG --- status code --- 06:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:32 DEBUG --- http response --- 06:51:32 DEBUG --- http response --- 06:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:51:32 DEBUG ----- output ----- 06:51:32 DEBUG ----- output ----- 06:51:32 DEBUG { 06:51:32 DEBUG { 06:51:32 DEBUG "status": "success", 06:51:32 DEBUG "status": "success", 06:51:32 DEBUG "data": { 06:51:32 DEBUG "data": { 06:51:32 DEBUG "resultType": "vector", 06:51:32 DEBUG "resultType": "vector", 06:51:32 DEBUG "result": [ 06:51:32 DEBUG "result": [ 06:51:32 DEBUG { 06:51:32 DEBUG { 06:51:32 DEBUG "metric": { 06:51:32 DEBUG "metric": { 06:51:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:51:32 DEBUG "instance": "10.142.0.25:9100" 06:51:32 DEBUG }, 06:51:32 DEBUG }, 06:51:32 DEBUG "value": [ 06:51:32 DEBUG "value": [ 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG [...] 06:51:32 DEBUG [...] 06:51:32 DEBUG 06:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728292" 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:51:32 DEBUG "instance": "10.142.0.77:9100" 06:51:32 DEBUG }, 06:51:32 DEBUG }, 06:51:32 DEBUG "value": [ 06:51:32 DEBUG "value": [ 06:51:32 DEBUG 1712728292, 06:51:32 DEBUG 1712728292, 06:51:32 DEBUG "0" 06:51:32 DEBUG "24912.966666666667" 06:51:32 DEBUG ] 06:51:32 DEBUG ] 06:51:32 DEBUG } 06:51:32 DEBUG } 06:51:32 DEBUG ] 06:51:32 DEBUG ] 06:51:32 DEBUG } 06:51:32 DEBUG } 06:51:32 DEBUG } 06:51:32 DEBUG } 06:51:32 INFO [http_cmd]: http status code OK 06:51:32 DEBUG --- status code --- 06:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:32 DEBUG --- http response --- 06:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:51:32 DEBUG ----- output ----- 06:51:32 DEBUG { 06:51:32 DEBUG "status": "success", 06:51:32 DEBUG "data": { 06:51:32 DEBUG "resultType": "vector", 06:51:32 DEBUG "result": [ 06:51:32 DEBUG { 06:51:32 DEBUG "metric": { 06:51:32 DEBUG "pod": "ds-idrepo-0" 06:51:32 DEBUG }, 06:51:32 DEBUG "value": [ 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG [...] 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG "pod": "ds-idrepo-1" 06:51:32 DEBUG }, 06:51:32 DEBUG "value": [ 06:51:32 DEBUG 1712728292, 06:51:32 DEBUG "0" 06:51:32 DEBUG ] 06:51:32 DEBUG } 06:51:32 DEBUG ] 06:51:32 DEBUG } 06:51:32 DEBUG } 06:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728292" 06:51:32 INFO [http_cmd]: http status code OK 06:51:32 DEBUG --- status code --- 06:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:32 DEBUG --- http response --- 06:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:51:32 DEBUG ----- output ----- 06:51:32 DEBUG { 06:51:32 DEBUG "status": "success", 06:51:32 DEBUG "data": { 06:51:32 DEBUG "resultType": "vector", 06:51:32 DEBUG "result": [ 06:51:32 DEBUG { 06:51:32 DEBUG "metric": { 06:51:32 DEBUG "instance": "10.142.0.25:9100" 06:51:32 DEBUG }, 06:51:32 DEBUG "value": [ 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG [...] 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG "instance": "10.142.0.77:9100" 06:51:32 DEBUG }, 06:51:32 DEBUG "value": [ 06:51:32 DEBUG 1712728292, 06:51:32 DEBUG "438079663104" 06:51:32 DEBUG ] 06:51:32 DEBUG } 06:51:32 DEBUG ] 06:51:32 DEBUG } 06:51:32 DEBUG } 06:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728292" 06:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728292" 06:51:32 INFO [http_cmd]: http status code OK 06:51:32 INFO [http_cmd]: http status code OK 06:51:32 DEBUG --- status code --- 06:51:32 DEBUG --- status code --- 06:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:32 DEBUG --- http response --- 06:51:32 DEBUG --- http response --- 06:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:51:32 DEBUG ----- output ----- 06:51:32 DEBUG ----- output ----- 06:51:32 DEBUG { 06:51:32 DEBUG { 06:51:32 DEBUG "status": "success", 06:51:32 DEBUG "status": "success", 06:51:32 DEBUG "data": { 06:51:32 DEBUG "data": { 06:51:32 DEBUG "resultType": "vector", 06:51:32 DEBUG "resultType": "vector", 06:51:32 DEBUG "result": [ 06:51:32 DEBUG "result": [ 06:51:32 DEBUG { 06:51:32 DEBUG { 06:51:32 DEBUG "metric": { 06:51:32 DEBUG "metric": { 06:51:32 DEBUG "instance": "10.142.0.25:9100" 06:51:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:51:32 DEBUG }, 06:51:32 DEBUG }, 06:51:32 DEBUG "value": [ 06:51:32 DEBUG "value": [ 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG [...] 06:51:32 DEBUG [...] 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG "instance": "10.142.0.77:9100" 06:51:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:51:32 DEBUG }, 06:51:32 DEBUG }, 06:51:32 DEBUG "value": [ 06:51:32 DEBUG "value": [ 06:51:32 DEBUG 1712728292, 06:51:32 DEBUG 1712728292, 06:51:32 DEBUG "0.0006666666666717447" 06:51:32 DEBUG "0" 06:51:32 DEBUG ] 06:51:32 DEBUG ] 06:51:32 DEBUG } 06:51:32 DEBUG } 06:51:32 DEBUG ] 06:51:32 DEBUG ] 06:51:32 DEBUG } 06:51:32 DEBUG } 06:51:32 DEBUG } 06:51:32 DEBUG } 06:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728292" 06:51:32 INFO [http_cmd]: http status code OK 06:51:32 DEBUG --- status code --- 06:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:32 DEBUG --- http response --- 06:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 06:51:32 DEBUG ----- output ----- 06:51:32 DEBUG { 06:51:32 DEBUG "status": "success", 06:51:32 DEBUG "data": { 06:51:32 DEBUG "resultType": "vector", 06:51:32 DEBUG "result": [ 06:51:32 DEBUG { 06:51:32 DEBUG "metric": { 06:51:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:51:32 DEBUG }, 06:51:32 DEBUG "value": [ 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG [...] 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG "pod": "idm-6bfb7c96cc-qklk9" 06:51:32 DEBUG }, 06:51:32 DEBUG "value": [ 06:51:32 DEBUG 1712728292, 06:51:32 DEBUG "0" 06:51:32 DEBUG ] 06:51:32 DEBUG } 06:51:32 DEBUG ] 06:51:32 DEBUG } 06:51:32 DEBUG } 06:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728292" 06:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712728292" 06:51:32 INFO [http_cmd]: http status code OK 06:51:32 DEBUG --- status code --- 06:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:32 DEBUG --- http response --- 06:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:51:32 DEBUG ----- output ----- 06:51:32 DEBUG { 06:51:32 DEBUG "status": "success", 06:51:32 DEBUG "data": { 06:51:32 DEBUG "resultType": "vector", 06:51:32 DEBUG "result": [ 06:51:32 DEBUG { 06:51:32 DEBUG "metric": { 06:51:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:51:32 DEBUG }, 06:51:32 DEBUG "value": [ 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG [...] 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:51:32 DEBUG }, 06:51:32 DEBUG "value": [ 06:51:32 DEBUG 1712728292, 06:51:32 DEBUG "0" 06:51:32 DEBUG ] 06:51:32 DEBUG } 06:51:32 DEBUG ] 06:51:32 DEBUG } 06:51:32 DEBUG } 06:51:32 INFO [http_cmd]: http status code OK 06:51:32 DEBUG --- status code --- 06:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:32 DEBUG --- http response --- 06:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:51:32 DEBUG ----- output ----- 06:51:32 DEBUG { 06:51:32 DEBUG "status": "success", 06:51:32 DEBUG "data": { 06:51:32 DEBUG "resultType": "vector", 06:51:32 DEBUG "result": [ 06:51:32 DEBUG { 06:51:32 DEBUG "metric": { 06:51:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:51:32 DEBUG }, 06:51:32 DEBUG "value": [ 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG [...] 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:51:32 DEBUG }, 06:51:32 DEBUG "value": [ 06:51:32 DEBUG 1712728292, 06:51:32 DEBUG "0.844316672620531" 06:51:32 DEBUG ] 06:51:32 DEBUG } 06:51:32 DEBUG ] 06:51:32 DEBUG } 06:51:32 DEBUG } 06:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728292" 06:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728292" 06:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712728292" 06:51:32 INFO [http_cmd]: http status code OK 06:51:32 DEBUG --- status code --- 06:51:32 INFO [http_cmd]: http status code OK 06:51:32 INFO [http_cmd]: http status code OK 06:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:32 DEBUG --- status code --- 06:51:32 DEBUG --- status code --- 06:51:32 DEBUG --- http response --- 06:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:51:32 DEBUG --- http response --- 06:51:32 DEBUG --- http response --- 06:51:32 DEBUG ----- output ----- 06:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:51:32 DEBUG { 06:51:32 DEBUG ----- output ----- 06:51:32 DEBUG ----- output ----- 06:51:32 DEBUG "status": "success", 06:51:32 DEBUG { 06:51:32 DEBUG { 06:51:32 DEBUG "data": { 06:51:32 DEBUG "status": "success", 06:51:32 DEBUG "status": "success", 06:51:32 DEBUG "resultType": "vector", 06:51:32 DEBUG "data": { 06:51:32 DEBUG "data": { 06:51:32 DEBUG "result": [ 06:51:32 DEBUG "resultType": "vector", 06:51:32 DEBUG "resultType": "vector", 06:51:32 DEBUG { 06:51:32 DEBUG "result": [ 06:51:32 DEBUG "result": [ 06:51:32 DEBUG "metric": { 06:51:32 DEBUG { 06:51:32 DEBUG { 06:51:32 DEBUG "pod": "ds-idrepo-0" 06:51:32 DEBUG "metric": { 06:51:32 DEBUG "metric": { 06:51:32 DEBUG }, 06:51:32 DEBUG "pod": "ds-idrepo-0" 06:51:32 DEBUG "pod": "ds-idrepo-0" 06:51:32 DEBUG "value": [ 06:51:32 DEBUG }, 06:51:32 DEBUG }, 06:51:32 DEBUG 06:51:32 DEBUG "value": [ 06:51:32 DEBUG "value": [ 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG [...] 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG [...] 06:51:32 DEBUG [...] 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG "pod": "ds-cts-2" 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG }, 06:51:32 DEBUG "pod": "ds-cts-2" 06:51:32 DEBUG "pod": "ds-cts-2" 06:51:32 DEBUG "value": [ 06:51:32 DEBUG }, 06:51:32 DEBUG }, 06:51:32 DEBUG 1712728292, 06:51:32 DEBUG "value": [ 06:51:32 DEBUG "value": [ 06:51:32 DEBUG "0" 06:51:32 DEBUG 1712728292, 06:51:32 DEBUG 1712728292, 06:51:32 DEBUG ] 06:51:32 DEBUG "0" 06:51:32 DEBUG "29.933333333333334" 06:51:32 DEBUG } 06:51:32 DEBUG ] 06:51:32 DEBUG ] 06:51:32 DEBUG ] 06:51:32 DEBUG } 06:51:32 DEBUG } 06:51:32 DEBUG } 06:51:32 DEBUG ] 06:51:32 DEBUG ] 06:51:32 DEBUG } 06:51:32 DEBUG } 06:51:32 DEBUG } 06:51:32 DEBUG } 06:51:32 DEBUG } 06:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728292" 06:51:32 INFO [http_cmd]: http status code OK 06:51:32 DEBUG --- status code --- 06:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:32 DEBUG --- http response --- 06:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:51:32 DEBUG ----- output ----- 06:51:32 DEBUG { 06:51:32 DEBUG "status": "success", 06:51:32 DEBUG "data": { 06:51:32 DEBUG "resultType": "vector", 06:51:32 DEBUG "result": [ 06:51:32 DEBUG { 06:51:32 DEBUG "metric": { 06:51:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:51:32 DEBUG }, 06:51:32 DEBUG "value": [ 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG [...] 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:51:32 DEBUG }, 06:51:32 DEBUG "value": [ 06:51:32 DEBUG 1712728292, 06:51:32 DEBUG "0" 06:51:32 DEBUG ] 06:51:32 DEBUG } 06:51:32 DEBUG ] 06:51:32 DEBUG } 06:51:32 DEBUG } 06:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712728292" 06:51:32 INFO [http_cmd]: http status code OK 06:51:32 DEBUG --- status code --- 06:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:32 DEBUG --- http response --- 06:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:51:32 DEBUG ----- output ----- 06:51:32 DEBUG { 06:51:32 DEBUG "status": "success", 06:51:32 DEBUG "data": { 06:51:32 DEBUG "resultType": "vector", 06:51:32 DEBUG "result": [ 06:51:32 DEBUG { 06:51:32 DEBUG "metric": { 06:51:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:51:32 DEBUG }, 06:51:32 DEBUG "value": [ 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG [...] 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG 06:51:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:51:32 DEBUG }, 06:51:32 DEBUG "value": [ 06:51:32 DEBUG 1712728292, 06:51:32 DEBUG "0.052" 06:51:32 DEBUG ] 06:51:32 DEBUG } 06:51:32 DEBUG ] 06:51:32 DEBUG } 06:51:32 DEBUG } 06:51:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728293" 06:51:33 INFO [http_cmd]: http status code OK 06:51:33 DEBUG --- status code --- 06:51:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:33 DEBUG --- http response --- 06:51:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:51:33 DEBUG ----- output ----- 06:51:33 DEBUG { 06:51:33 DEBUG "status": "success", 06:51:33 DEBUG "data": { 06:51:33 DEBUG "resultType": "vector", 06:51:33 DEBUG "result": [ 06:51:33 DEBUG { 06:51:33 DEBUG "metric": { 06:51:33 DEBUG "pod": "ds-idrepo-2" 06:51:33 DEBUG }, 06:51:33 DEBUG "value": [ 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG [...] 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG "pod": "ds-cts-2" 06:51:33 DEBUG }, 06:51:33 DEBUG "value": [ 06:51:33 DEBUG 1712728293, 06:51:33 DEBUG "0" 06:51:33 DEBUG ] 06:51:33 DEBUG } 06:51:33 DEBUG ] 06:51:33 DEBUG } 06:51:33 DEBUG } 06:51:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728293" 06:51:33 INFO [http_cmd]: http status code OK 06:51:33 DEBUG --- status code --- 06:51:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:33 DEBUG --- http response --- 06:51:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:51:33 DEBUG ----- output ----- 06:51:33 DEBUG { 06:51:33 DEBUG "status": "success", 06:51:33 DEBUG "data": { 06:51:33 DEBUG "resultType": "vector", 06:51:33 DEBUG "result": [ 06:51:33 DEBUG { 06:51:33 DEBUG "metric": { 06:51:33 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:51:33 DEBUG }, 06:51:33 DEBUG "value": [ 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG [...] 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:51:33 DEBUG }, 06:51:33 DEBUG "value": [ 06:51:33 DEBUG 1712728293, 06:51:33 DEBUG "0" 06:51:33 DEBUG ] 06:51:33 DEBUG } 06:51:33 DEBUG ] 06:51:33 DEBUG } 06:51:33 DEBUG } 06:51:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728293" 06:51:33 INFO [http_cmd]: http status code OK 06:51:33 DEBUG --- status code --- 06:51:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:33 DEBUG --- http response --- 06:51:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:51:33 DEBUG ----- output ----- 06:51:33 DEBUG { 06:51:33 DEBUG "status": "success", 06:51:33 DEBUG "data": { 06:51:33 DEBUG "resultType": "vector", 06:51:33 DEBUG "result": [ 06:51:33 DEBUG { 06:51:33 DEBUG "metric": { 06:51:33 DEBUG "pod": "ds-idrepo-0" 06:51:33 DEBUG }, 06:51:33 DEBUG "value": [ 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG [...] 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG "pod": "ds-idrepo-1" 06:51:33 DEBUG }, 06:51:33 DEBUG "value": [ 06:51:33 DEBUG 1712728293, 06:51:33 DEBUG "0" 06:51:33 DEBUG ] 06:51:33 DEBUG } 06:51:33 DEBUG ] 06:51:33 DEBUG } 06:51:33 DEBUG } 06:51:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728293" 06:51:33 INFO [http_cmd]: http status code OK 06:51:33 DEBUG --- status code --- 06:51:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:33 DEBUG --- http response --- 06:51:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:51:33 DEBUG ----- output ----- 06:51:33 DEBUG { 06:51:33 DEBUG "status": "success", 06:51:33 DEBUG "data": { 06:51:33 DEBUG "resultType": "vector", 06:51:33 DEBUG "result": [ 06:51:33 DEBUG { 06:51:33 DEBUG "metric": { 06:51:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:51:33 DEBUG }, 06:51:33 DEBUG "value": [ 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG [...] 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG "pod": "am-654c4fcbcf-487cv" 06:51:33 DEBUG }, 06:51:33 DEBUG "value": [ 06:51:33 DEBUG 1712728293, 06:51:33 DEBUG "0" 06:51:33 DEBUG ] 06:51:33 DEBUG } 06:51:33 DEBUG ] 06:51:33 DEBUG } 06:51:33 DEBUG } 06:51:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728293" 06:51:33 INFO [http_cmd]: http status code OK 06:51:33 DEBUG --- status code --- 06:51:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:33 DEBUG --- http response --- 06:51:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:51:33 DEBUG ----- output ----- 06:51:33 DEBUG { 06:51:33 DEBUG "status": "success", 06:51:33 DEBUG "data": { 06:51:33 DEBUG "resultType": "vector", 06:51:33 DEBUG "result": [ 06:51:33 DEBUG { 06:51:33 DEBUG "metric": { 06:51:33 DEBUG "pod": "ds-idrepo-0" 06:51:33 DEBUG }, 06:51:33 DEBUG "value": [ 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG [...] 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG "pod": "ds-idrepo-1" 06:51:33 DEBUG }, 06:51:33 DEBUG "value": [ 06:51:33 DEBUG 1712728293, 06:51:33 DEBUG "0" 06:51:33 DEBUG ] 06:51:33 DEBUG } 06:51:33 DEBUG ] 06:51:33 DEBUG } 06:51:33 DEBUG } 06:51:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728293" 06:51:33 INFO [http_cmd]: http status code OK 06:51:33 DEBUG --- status code --- 06:51:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:33 DEBUG --- http response --- 06:51:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:51:33 DEBUG ----- output ----- 06:51:33 DEBUG { 06:51:33 DEBUG "status": "success", 06:51:33 DEBUG "data": { 06:51:33 DEBUG "resultType": "vector", 06:51:33 DEBUG "result": [ 06:51:33 DEBUG { 06:51:33 DEBUG "metric": { 06:51:33 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:51:33 DEBUG }, 06:51:33 DEBUG "value": [ 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG [...] 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:51:33 DEBUG }, 06:51:33 DEBUG "value": [ 06:51:33 DEBUG 1712728293, 06:51:33 DEBUG "0" 06:51:33 DEBUG ] 06:51:33 DEBUG } 06:51:33 DEBUG ] 06:51:33 DEBUG } 06:51:33 DEBUG } 06:51:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712728293" 06:51:33 INFO [http_cmd]: http status code OK 06:51:33 DEBUG --- status code --- 06:51:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:33 DEBUG --- http response --- 06:51:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:51:33 DEBUG ----- output ----- 06:51:33 DEBUG { 06:51:33 DEBUG "status": "success", 06:51:33 DEBUG "data": { 06:51:33 DEBUG "resultType": "vector", 06:51:33 DEBUG "result": [ 06:51:33 DEBUG { 06:51:33 DEBUG "metric": { 06:51:33 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:51:33 DEBUG }, 06:51:33 DEBUG "value": [ 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG [...] 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:51:33 DEBUG }, 06:51:33 DEBUG "value": [ 06:51:33 DEBUG 1712728293, 06:51:33 DEBUG "2486419456" 06:51:33 DEBUG ] 06:51:33 DEBUG } 06:51:33 DEBUG ] 06:51:33 DEBUG } 06:51:33 DEBUG } 06:51:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728293" 06:51:33 INFO [http_cmd]: http status code OK 06:51:33 DEBUG --- status code --- 06:51:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:33 DEBUG --- http response --- 06:51:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:51:33 DEBUG ----- output ----- 06:51:33 DEBUG { 06:51:33 DEBUG "status": "success", 06:51:33 DEBUG "data": { 06:51:33 DEBUG "resultType": "vector", 06:51:33 DEBUG "result": [ 06:51:33 DEBUG { 06:51:33 DEBUG "metric": { 06:51:33 DEBUG "instance": "10.142.0.25:9100" 06:51:33 DEBUG }, 06:51:33 DEBUG "value": [ 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG [...] 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG "instance": "10.142.0.77:9100" 06:51:33 DEBUG }, 06:51:33 DEBUG "value": [ 06:51:33 DEBUG 1712728293, 06:51:33 DEBUG "26085.733333333334" 06:51:33 DEBUG ] 06:51:33 DEBUG } 06:51:33 DEBUG ] 06:51:33 DEBUG } 06:51:33 DEBUG } 06:51:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728293" 06:51:33 INFO [http_cmd]: http status code OK 06:51:33 DEBUG --- status code --- 06:51:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:33 DEBUG --- http response --- 06:51:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:51:33 DEBUG ----- output ----- 06:51:33 DEBUG { 06:51:33 DEBUG "status": "success", 06:51:33 DEBUG "data": { 06:51:33 DEBUG "resultType": "vector", 06:51:33 DEBUG "result": [ 06:51:33 DEBUG { 06:51:33 DEBUG "metric": { 06:51:33 DEBUG "instance": "10.142.0.25:9100" 06:51:33 DEBUG }, 06:51:33 DEBUG "value": [ 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG [...] 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG 06:51:33 DEBUG "instance": "10.142.0.77:9100" 06:51:33 DEBUG }, 06:51:33 DEBUG "value": [ 06:51:33 DEBUG 1712728293, 06:51:33 DEBUG "0" 06:51:33 DEBUG ] 06:51:33 DEBUG } 06:51:33 DEBUG ] 06:51:33 DEBUG } 06:51:33 DEBUG } 06:51:54 INFO 06:51:54 INFO [loop_until]: kubectl --namespace=xlou top pods 06:51:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:51:54 INFO [loop_until]: OK (rc = 0) 06:51:54 DEBUG --- stdout --- 06:51:54 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 9Mi am-654c4fcbcf-487cv 7m 8870Mi am-654c4fcbcf-9scrj 7m 8378Mi am-654c4fcbcf-v5qb5 8m 8096Mi ds-cts-0 16m 12723Mi ds-cts-1 19m 12921Mi ds-cts-2 18m 11473Mi ds-idrepo-0 16m 5615Mi ds-idrepo-1 10m 5677Mi ds-idrepo-2 13m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 5Mi idm-6bfb7c96cc-qklk9 4m 1199Mi idm-6bfb7c96cc-vsb58 3m 2213Mi lodemon-b776b7d88-thrqf 6m 73Mi login-ui-5f964fb894-5zpvq 1m 4Mi overseer-0-6f8784665d-zq5hb 1020m 2097Mi 06:51:54 DEBUG --- stderr --- 06:51:54 DEBUG 06:52:03 INFO 06:52:03 INFO [loop_until]: kubectl --namespace=xlou top node 06:52:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:52:03 INFO [loop_until]: OK (rc = 0) 06:52:03 DEBUG --- stdout --- 06:52:03 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 91m 0% 2316Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 63m 0% 8989Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 77m 0% 3258Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 201m 1% 6508Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 68m 0% 9282Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 68m 0% 9731Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 77m 0% 874Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 78m 0% 6308Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 73m 0% 13459Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 68m 0% 12210Mi 20% gke-xlou-cdm-ds-32e4dcb1-gxzj 67m 0% 6398Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 73m 0% 13655Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 65m 0% 6369Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 1073m 6% 5719Mi 9% 06:52:03 DEBUG --- stderr --- 06:52:03 DEBUG 06:52:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712728350" 06:52:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728350" 06:52:30 INFO [http_cmd]: http status code OK 06:52:30 DEBUG --- status code --- 06:52:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:30 DEBUG --- http response --- 06:52:30 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:52:30 ERROR [http_cmd]: ERROR 06:52:30 ERROR --- status code --- 06:52:30 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:52:30 ERROR --- http response --- 06:52:30 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:52:30 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:52:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728350" 06:52:30 INFO [http_cmd]: http status code OK 06:52:30 DEBUG --- status code --- 06:52:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:30 DEBUG --- http response --- 06:52:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:52:30 DEBUG ----- output ----- 06:52:30 DEBUG { 06:52:30 DEBUG "status": "success", 06:52:30 DEBUG "data": { 06:52:30 DEBUG "resultType": "vector", 06:52:30 DEBUG "result": [ 06:52:30 DEBUG { 06:52:30 DEBUG "metric": { 06:52:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:52:30 DEBUG }, 06:52:30 DEBUG "value": [ 06:52:30 DEBUG 06:52:30 DEBUG 06:52:30 DEBUG 06:52:30 DEBUG [...] 06:52:30 DEBUG 06:52:30 DEBUG 06:52:30 DEBUG 06:52:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:52:30 DEBUG }, 06:52:30 DEBUG "value": [ 06:52:30 DEBUG 1712728350, 06:52:30 DEBUG "0" 06:52:30 DEBUG ] 06:52:30 DEBUG } 06:52:30 DEBUG ] 06:52:30 DEBUG } 06:52:30 DEBUG } 06:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728352" 06:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728352" 06:52:32 INFO [http_cmd]: http status code OK 06:52:32 DEBUG --- status code --- 06:52:32 INFO [http_cmd]: http status code OK 06:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:32 DEBUG --- status code --- 06:52:32 DEBUG --- http response --- 06:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:52:32 DEBUG --- http response --- 06:52:32 DEBUG ----- output ----- 06:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:52:32 DEBUG { 06:52:32 DEBUG ----- output ----- 06:52:32 DEBUG "status": "success", 06:52:32 DEBUG { 06:52:32 DEBUG "data": { 06:52:32 DEBUG "status": "success", 06:52:32 DEBUG "resultType": "vector", 06:52:32 DEBUG "data": { 06:52:32 DEBUG "result": [ 06:52:32 DEBUG "resultType": "vector", 06:52:32 DEBUG { 06:52:32 DEBUG "result": [ 06:52:32 DEBUG "metric": { 06:52:32 DEBUG { 06:52:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:52:32 DEBUG "metric": { 06:52:32 DEBUG }, 06:52:32 DEBUG "instance": "10.142.0.25:9100" 06:52:32 DEBUG "value": [ 06:52:32 DEBUG }, 06:52:32 DEBUG 06:52:32 DEBUG "value": [ 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG [...] 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG [...] 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:52:32 DEBUG 06:52:32 DEBUG }, 06:52:32 DEBUG "instance": "10.142.0.77:9100" 06:52:32 DEBUG "value": [ 06:52:32 DEBUG }, 06:52:32 DEBUG 1712728352, 06:52:32 DEBUG "value": [ 06:52:32 DEBUG "0" 06:52:32 DEBUG 1712728352, 06:52:32 DEBUG ] 06:52:32 DEBUG "25383.833333333332" 06:52:32 DEBUG } 06:52:32 DEBUG ] 06:52:32 DEBUG ] 06:52:32 DEBUG } 06:52:32 DEBUG } 06:52:32 DEBUG ] 06:52:32 DEBUG } 06:52:32 DEBUG } 06:52:32 DEBUG } 06:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728352" 06:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728352" 06:52:32 INFO [http_cmd]: http status code OK 06:52:32 DEBUG --- status code --- 06:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:32 DEBUG --- http response --- 06:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:52:32 DEBUG ----- output ----- 06:52:32 DEBUG { 06:52:32 DEBUG "status": "success", 06:52:32 DEBUG "data": { 06:52:32 DEBUG "resultType": "vector", 06:52:32 DEBUG "result": [ 06:52:32 DEBUG { 06:52:32 DEBUG "metric": { 06:52:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:52:32 DEBUG }, 06:52:32 DEBUG "value": [ 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG [...] 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:52:32 DEBUG }, 06:52:32 DEBUG "value": [ 06:52:32 DEBUG 1712728352, 06:52:32 DEBUG "0" 06:52:32 DEBUG ] 06:52:32 DEBUG } 06:52:32 DEBUG ] 06:52:32 DEBUG } 06:52:32 DEBUG } 06:52:32 INFO [http_cmd]: http status code OK 06:52:32 DEBUG --- status code --- 06:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:32 DEBUG --- http response --- 06:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:52:32 DEBUG ----- output ----- 06:52:32 DEBUG { 06:52:32 DEBUG "status": "success", 06:52:32 DEBUG "data": { 06:52:32 DEBUG "resultType": "vector", 06:52:32 DEBUG "result": [ 06:52:32 DEBUG { 06:52:32 DEBUG "metric": { 06:52:32 DEBUG "pod": "ds-idrepo-0" 06:52:32 DEBUG }, 06:52:32 DEBUG "value": [ 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG [...] 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG "pod": "ds-idrepo-1" 06:52:32 DEBUG }, 06:52:32 DEBUG "value": [ 06:52:32 DEBUG 1712728352, 06:52:32 DEBUG "0" 06:52:32 DEBUG ] 06:52:32 DEBUG } 06:52:32 DEBUG ] 06:52:32 DEBUG } 06:52:32 DEBUG } 06:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728352" 06:52:32 INFO [http_cmd]: http status code OK 06:52:32 DEBUG --- status code --- 06:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:32 DEBUG --- http response --- 06:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:52:32 DEBUG ----- output ----- 06:52:32 DEBUG { 06:52:32 DEBUG "status": "success", 06:52:32 DEBUG "data": { 06:52:32 DEBUG "resultType": "vector", 06:52:32 DEBUG "result": [ 06:52:32 DEBUG { 06:52:32 DEBUG "metric": { 06:52:32 DEBUG "instance": "10.142.0.25:9100" 06:52:32 DEBUG }, 06:52:32 DEBUG "value": [ 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG [...] 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG "instance": "10.142.0.77:9100" 06:52:32 DEBUG }, 06:52:32 DEBUG "value": [ 06:52:32 DEBUG 1712728352, 06:52:32 DEBUG "438081184768" 06:52:32 DEBUG ] 06:52:32 DEBUG } 06:52:32 DEBUG ] 06:52:32 DEBUG } 06:52:32 DEBUG } 06:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728352" 06:52:32 INFO [http_cmd]: http status code OK 06:52:32 DEBUG --- status code --- 06:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:32 DEBUG --- http response --- 06:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:52:32 DEBUG ----- output ----- 06:52:32 DEBUG { 06:52:32 DEBUG "status": "success", 06:52:32 DEBUG "data": { 06:52:32 DEBUG "resultType": "vector", 06:52:32 DEBUG "result": [ 06:52:32 DEBUG { 06:52:32 DEBUG "metric": { 06:52:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:52:32 DEBUG }, 06:52:32 DEBUG "value": [ 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG [...] 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:52:32 DEBUG }, 06:52:32 DEBUG "value": [ 06:52:32 DEBUG 1712728352, 06:52:32 DEBUG "0" 06:52:32 DEBUG ] 06:52:32 DEBUG } 06:52:32 DEBUG ] 06:52:32 DEBUG } 06:52:32 DEBUG } 06:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728352" 06:52:32 INFO [http_cmd]: http status code OK 06:52:32 DEBUG --- status code --- 06:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:32 DEBUG --- http response --- 06:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:52:32 DEBUG ----- output ----- 06:52:32 DEBUG { 06:52:32 DEBUG "status": "success", 06:52:32 DEBUG "data": { 06:52:32 DEBUG "resultType": "vector", 06:52:32 DEBUG "result": [ 06:52:32 DEBUG { 06:52:32 DEBUG "metric": { 06:52:32 DEBUG "instance": "10.142.0.25:9100" 06:52:32 DEBUG }, 06:52:32 DEBUG "value": [ 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG [...] 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG "instance": "10.142.0.77:9100" 06:52:32 DEBUG }, 06:52:32 DEBUG "value": [ 06:52:32 DEBUG 1712728352, 06:52:32 DEBUG "0.0008999999999969551" 06:52:32 DEBUG ] 06:52:32 DEBUG } 06:52:32 DEBUG ] 06:52:32 DEBUG } 06:52:32 DEBUG } 06:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728352" 06:52:32 INFO [http_cmd]: http status code OK 06:52:32 DEBUG --- status code --- 06:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:32 DEBUG --- http response --- 06:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:52:32 DEBUG ----- output ----- 06:52:32 DEBUG { 06:52:32 DEBUG "status": "success", 06:52:32 DEBUG "data": { 06:52:32 DEBUG "resultType": "vector", 06:52:32 DEBUG "result": [ 06:52:32 DEBUG { 06:52:32 DEBUG "metric": { 06:52:32 DEBUG "pod": "ds-cts-2" 06:52:32 DEBUG }, 06:52:32 DEBUG "value": [ 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG [...] 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:52:32 DEBUG }, 06:52:32 DEBUG "value": [ 06:52:32 DEBUG 1712728352, 06:52:32 DEBUG "0" 06:52:32 DEBUG ] 06:52:32 DEBUG } 06:52:32 DEBUG ] 06:52:32 DEBUG } 06:52:32 DEBUG } 06:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712728352" 06:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728352" 06:52:32 INFO [http_cmd]: http status code OK 06:52:32 DEBUG --- status code --- 06:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:32 DEBUG --- http response --- 06:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:52:32 DEBUG ----- output ----- 06:52:32 DEBUG { 06:52:32 DEBUG "status": "success", 06:52:32 DEBUG "data": { 06:52:32 DEBUG "resultType": "vector", 06:52:32 INFO [http_cmd]: http status code OK 06:52:32 DEBUG "result": [ 06:52:32 DEBUG --- status code --- 06:52:32 DEBUG { 06:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:32 DEBUG "metric": { 06:52:32 DEBUG --- http response --- 06:52:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:52:32 DEBUG }, 06:52:32 DEBUG ----- output ----- 06:52:32 DEBUG "value": [ 06:52:32 DEBUG { 06:52:32 DEBUG 06:52:32 DEBUG "status": "success", 06:52:32 DEBUG 06:52:32 DEBUG "data": { 06:52:32 DEBUG 06:52:32 DEBUG "resultType": "vector", 06:52:32 DEBUG [...] 06:52:32 DEBUG "result": [ 06:52:32 DEBUG 06:52:32 DEBUG { 06:52:32 DEBUG 06:52:32 DEBUG "metric": { 06:52:32 DEBUG 06:52:32 DEBUG "pod": "ds-cts-2" 06:52:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:52:32 DEBUG }, 06:52:32 DEBUG }, 06:52:32 DEBUG "value": [ 06:52:32 DEBUG "value": [ 06:52:32 DEBUG 06:52:32 DEBUG 1712728352, 06:52:32 DEBUG 06:52:32 DEBUG "1.036547540885319" 06:52:32 DEBUG 06:52:32 DEBUG ] 06:52:32 DEBUG [...] 06:52:32 DEBUG } 06:52:32 DEBUG 06:52:32 DEBUG ] 06:52:32 DEBUG 06:52:32 DEBUG } 06:52:32 DEBUG 06:52:32 DEBUG } 06:52:32 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:52:32 DEBUG }, 06:52:32 DEBUG "value": [ 06:52:32 DEBUG 1712728352, 06:52:32 DEBUG "60068318.293583475" 06:52:32 DEBUG ] 06:52:32 DEBUG } 06:52:32 DEBUG ] 06:52:32 DEBUG } 06:52:32 DEBUG } 06:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728352" 06:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712728352" 06:52:32 INFO [http_cmd]: http status code OK 06:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728352" 06:52:32 INFO [http_cmd]: http status code OK 06:52:32 DEBUG --- status code --- 06:52:32 DEBUG --- status code --- 06:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:32 DEBUG --- http response --- 06:52:32 DEBUG --- http response --- 06:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:52:32 DEBUG ----- output ----- 06:52:32 DEBUG ----- output ----- 06:52:32 DEBUG { 06:52:32 DEBUG { 06:52:32 DEBUG "status": "success", 06:52:32 DEBUG "status": "success", 06:52:32 DEBUG "data": { 06:52:32 DEBUG "data": { 06:52:32 DEBUG "resultType": "vector", 06:52:32 DEBUG "resultType": "vector", 06:52:32 DEBUG "result": [ 06:52:32 DEBUG "result": [ 06:52:32 DEBUG { 06:52:32 DEBUG { 06:52:32 DEBUG "metric": { 06:52:32 DEBUG "metric": { 06:52:32 DEBUG "pod": "ds-idrepo-0" 06:52:32 DEBUG "pod": "ds-idrepo-0" 06:52:32 DEBUG }, 06:52:32 DEBUG }, 06:52:32 DEBUG "value": [ 06:52:32 DEBUG "value": [ 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG [...] 06:52:32 DEBUG [...] 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG "pod": "ds-cts-2" 06:52:32 DEBUG "pod": "ds-cts-2" 06:52:32 DEBUG }, 06:52:32 DEBUG }, 06:52:32 DEBUG "value": [ 06:52:32 DEBUG "value": [ 06:52:32 DEBUG 1712728352, 06:52:32 DEBUG 1712728352, 06:52:32 DEBUG "29.866666666666667" 06:52:32 DEBUG "0" 06:52:32 DEBUG ] 06:52:32 DEBUG ] 06:52:32 DEBUG } 06:52:32 DEBUG } 06:52:32 DEBUG ] 06:52:32 DEBUG ] 06:52:32 DEBUG } 06:52:32 DEBUG } 06:52:32 DEBUG } 06:52:32 DEBUG } 06:52:32 INFO [http_cmd]: http status code OK 06:52:32 DEBUG --- status code --- 06:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:32 DEBUG --- http response --- 06:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:52:32 DEBUG ----- output ----- 06:52:32 DEBUG { 06:52:32 DEBUG "status": "success", 06:52:32 DEBUG "data": { 06:52:32 DEBUG "resultType": "vector", 06:52:32 DEBUG "result": [ 06:52:32 DEBUG { 06:52:32 DEBUG "metric": { 06:52:32 DEBUG "pod": "ds-idrepo-0" 06:52:32 DEBUG }, 06:52:32 DEBUG "value": [ 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG [...] 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG "pod": "ds-cts-2" 06:52:32 DEBUG }, 06:52:32 DEBUG "value": [ 06:52:32 DEBUG 1712728352, 06:52:32 DEBUG "0" 06:52:32 DEBUG ] 06:52:32 DEBUG } 06:52:32 DEBUG ] 06:52:32 DEBUG } 06:52:32 DEBUG } 06:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728352" 06:52:32 INFO [http_cmd]: http status code OK 06:52:32 DEBUG --- status code --- 06:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:32 DEBUG --- http response --- 06:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:52:32 DEBUG ----- output ----- 06:52:32 DEBUG { 06:52:32 DEBUG "status": "success", 06:52:32 DEBUG "data": { 06:52:32 DEBUG "resultType": "vector", 06:52:32 DEBUG "result": [ 06:52:32 DEBUG { 06:52:32 DEBUG "metric": { 06:52:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:52:32 DEBUG }, 06:52:32 DEBUG "value": [ 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG [...] 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG 06:52:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:52:32 DEBUG }, 06:52:32 DEBUG "value": [ 06:52:32 DEBUG 1712728352, 06:52:32 DEBUG "0" 06:52:32 DEBUG ] 06:52:32 DEBUG } 06:52:32 DEBUG ] 06:52:32 DEBUG } 06:52:32 DEBUG } 06:52:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712728353" 06:52:33 INFO [http_cmd]: http status code OK 06:52:33 DEBUG --- status code --- 06:52:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:33 DEBUG --- http response --- 06:52:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:52:33 DEBUG ----- output ----- 06:52:33 DEBUG { 06:52:33 DEBUG "status": "success", 06:52:33 DEBUG "data": { 06:52:33 DEBUG "resultType": "vector", 06:52:33 DEBUG "result": [ 06:52:33 DEBUG { 06:52:33 DEBUG "metric": { 06:52:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:52:33 DEBUG }, 06:52:33 DEBUG "value": [ 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG [...] 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG "pod": "am-654c4fcbcf-487cv" 06:52:33 DEBUG }, 06:52:33 DEBUG "value": [ 06:52:33 DEBUG 1712728353, 06:52:33 DEBUG "0.052" 06:52:33 DEBUG ] 06:52:33 DEBUG } 06:52:33 DEBUG ] 06:52:33 DEBUG } 06:52:33 DEBUG } 06:52:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728353" 06:52:33 INFO [http_cmd]: http status code OK 06:52:33 DEBUG --- status code --- 06:52:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:33 DEBUG --- http response --- 06:52:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:52:33 DEBUG ----- output ----- 06:52:33 DEBUG { 06:52:33 DEBUG "status": "success", 06:52:33 DEBUG "data": { 06:52:33 DEBUG "resultType": "vector", 06:52:33 DEBUG "result": [ 06:52:33 DEBUG { 06:52:33 DEBUG "metric": { 06:52:33 DEBUG "pod": "ds-idrepo-2" 06:52:33 DEBUG }, 06:52:33 DEBUG "value": [ 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG [...] 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG "pod": "ds-cts-2" 06:52:33 DEBUG }, 06:52:33 DEBUG "value": [ 06:52:33 DEBUG 1712728353, 06:52:33 DEBUG "0" 06:52:33 DEBUG ] 06:52:33 DEBUG } 06:52:33 DEBUG ] 06:52:33 DEBUG } 06:52:33 DEBUG } 06:52:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728353" 06:52:33 INFO [http_cmd]: http status code OK 06:52:33 DEBUG --- status code --- 06:52:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:33 DEBUG --- http response --- 06:52:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:52:33 DEBUG ----- output ----- 06:52:33 DEBUG { 06:52:33 DEBUG "status": "success", 06:52:33 DEBUG "data": { 06:52:33 DEBUG "resultType": "vector", 06:52:33 DEBUG "result": [ 06:52:33 DEBUG { 06:52:33 DEBUG "metric": { 06:52:33 DEBUG "pod": "ds-cts-2" 06:52:33 DEBUG }, 06:52:33 DEBUG "value": [ 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG [...] 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:52:33 DEBUG }, 06:52:33 DEBUG "value": [ 06:52:33 DEBUG 1712728353, 06:52:33 DEBUG "0.002229565945995069" 06:52:33 DEBUG ] 06:52:33 DEBUG } 06:52:33 DEBUG ] 06:52:33 DEBUG } 06:52:33 DEBUG } 06:52:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728353" 06:52:33 INFO [http_cmd]: http status code OK 06:52:33 DEBUG --- status code --- 06:52:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:33 DEBUG --- http response --- 06:52:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:52:33 DEBUG ----- output ----- 06:52:33 DEBUG { 06:52:33 DEBUG "status": "success", 06:52:33 DEBUG "data": { 06:52:33 DEBUG "resultType": "vector", 06:52:33 DEBUG "result": [ 06:52:33 DEBUG { 06:52:33 DEBUG "metric": { 06:52:33 DEBUG "pod": "ds-idrepo-0" 06:52:33 DEBUG }, 06:52:33 DEBUG "value": [ 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG [...] 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG "pod": "ds-idrepo-1" 06:52:33 DEBUG }, 06:52:33 DEBUG "value": [ 06:52:33 DEBUG 1712728353, 06:52:33 DEBUG "0" 06:52:33 DEBUG ] 06:52:33 DEBUG } 06:52:33 DEBUG ] 06:52:33 DEBUG } 06:52:33 DEBUG } 06:52:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728353" 06:52:33 INFO [http_cmd]: http status code OK 06:52:33 DEBUG --- status code --- 06:52:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:33 DEBUG --- http response --- 06:52:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:52:33 DEBUG ----- output ----- 06:52:33 DEBUG { 06:52:33 DEBUG "status": "success", 06:52:33 DEBUG "data": { 06:52:33 DEBUG "resultType": "vector", 06:52:33 DEBUG "result": [ 06:52:33 DEBUG { 06:52:33 DEBUG "metric": { 06:52:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:52:33 DEBUG }, 06:52:33 DEBUG "value": [ 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG [...] 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG "pod": "am-654c4fcbcf-487cv" 06:52:33 DEBUG }, 06:52:33 DEBUG "value": [ 06:52:33 DEBUG 1712728353, 06:52:33 DEBUG "0" 06:52:33 DEBUG ] 06:52:33 DEBUG } 06:52:33 DEBUG ] 06:52:33 DEBUG } 06:52:33 DEBUG } 06:52:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728353" 06:52:33 INFO [http_cmd]: http status code OK 06:52:33 DEBUG --- status code --- 06:52:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:33 DEBUG --- http response --- 06:52:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:52:33 DEBUG ----- output ----- 06:52:33 DEBUG { 06:52:33 DEBUG "status": "success", 06:52:33 DEBUG "data": { 06:52:33 DEBUG "resultType": "vector", 06:52:33 DEBUG "result": [ 06:52:33 DEBUG { 06:52:33 DEBUG "metric": { 06:52:33 DEBUG "pod": "ds-idrepo-0" 06:52:33 DEBUG }, 06:52:33 DEBUG "value": [ 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG [...] 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG "pod": "ds-idrepo-1" 06:52:33 DEBUG }, 06:52:33 DEBUG "value": [ 06:52:33 DEBUG 1712728353, 06:52:33 DEBUG "0" 06:52:33 DEBUG ] 06:52:33 DEBUG } 06:52:33 DEBUG ] 06:52:33 DEBUG } 06:52:33 DEBUG } 06:52:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728353" 06:52:33 INFO [http_cmd]: http status code OK 06:52:33 DEBUG --- status code --- 06:52:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:33 DEBUG --- http response --- 06:52:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:52:33 DEBUG ----- output ----- 06:52:33 DEBUG { 06:52:33 DEBUG "status": "success", 06:52:33 DEBUG "data": { 06:52:33 DEBUG "resultType": "vector", 06:52:33 DEBUG "result": [ 06:52:33 DEBUG { 06:52:33 DEBUG "metric": { 06:52:33 DEBUG "pod": "ds-cts-2" 06:52:33 DEBUG }, 06:52:33 DEBUG "value": [ 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG [...] 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:52:33 DEBUG }, 06:52:33 DEBUG "value": [ 06:52:33 DEBUG 1712728353, 06:52:33 DEBUG "0" 06:52:33 DEBUG ] 06:52:33 DEBUG } 06:52:33 DEBUG ] 06:52:33 DEBUG } 06:52:33 DEBUG } 06:52:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712728353" 06:52:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728353" 06:52:33 INFO [http_cmd]: http status code OK 06:52:33 DEBUG --- status code --- 06:52:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:33 DEBUG --- http response --- 06:52:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:52:33 DEBUG ----- output ----- 06:52:33 DEBUG { 06:52:33 DEBUG "status": "success", 06:52:33 DEBUG "data": { 06:52:33 DEBUG "resultType": "vector", 06:52:33 DEBUG "result": [ 06:52:33 DEBUG { 06:52:33 DEBUG "metric": { 06:52:33 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:52:33 DEBUG }, 06:52:33 DEBUG "value": [ 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG [...] 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:52:33 DEBUG }, 06:52:33 DEBUG "value": [ 06:52:33 DEBUG 1712728353, 06:52:33 DEBUG "2488090624" 06:52:33 DEBUG ] 06:52:33 DEBUG } 06:52:33 DEBUG ] 06:52:33 DEBUG } 06:52:33 DEBUG } 06:52:33 INFO [http_cmd]: http status code OK 06:52:33 DEBUG --- status code --- 06:52:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:33 DEBUG --- http response --- 06:52:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:52:33 DEBUG ----- output ----- 06:52:33 DEBUG { 06:52:33 DEBUG "status": "success", 06:52:33 DEBUG "data": { 06:52:33 DEBUG "resultType": "vector", 06:52:33 DEBUG "result": [ 06:52:33 DEBUG { 06:52:33 DEBUG "metric": { 06:52:33 DEBUG "instance": "10.142.0.25:9100" 06:52:33 DEBUG }, 06:52:33 DEBUG "value": [ 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG [...] 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG 06:52:33 DEBUG "instance": "10.142.0.77:9100" 06:52:33 DEBUG }, 06:52:33 DEBUG "value": [ 06:52:33 DEBUG 1712728353, 06:52:33 DEBUG "28314.966666666667" 06:52:33 DEBUG ] 06:52:33 DEBUG } 06:52:33 DEBUG ] 06:52:33 DEBUG } 06:52:33 DEBUG } 06:52:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728353" 06:52:34 INFO [http_cmd]: http status code OK 06:52:34 DEBUG --- status code --- 06:52:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:34 DEBUG --- http response --- 06:52:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:52:34 DEBUG ----- output ----- 06:52:34 DEBUG { 06:52:34 DEBUG "status": "success", 06:52:34 DEBUG "data": { 06:52:34 DEBUG "resultType": "vector", 06:52:34 DEBUG "result": [ 06:52:34 DEBUG { 06:52:34 DEBUG "metric": { 06:52:34 DEBUG "instance": "10.142.0.25:9100" 06:52:34 DEBUG }, 06:52:34 DEBUG "value": [ 06:52:34 DEBUG 06:52:34 DEBUG 06:52:34 DEBUG 06:52:34 DEBUG [...] 06:52:34 DEBUG 06:52:34 DEBUG 06:52:34 DEBUG 06:52:34 DEBUG "instance": "10.142.0.77:9100" 06:52:34 DEBUG }, 06:52:34 DEBUG "value": [ 06:52:34 DEBUG 1712728353, 06:52:34 DEBUG "0.000333333333333267" 06:52:34 DEBUG ] 06:52:34 DEBUG } 06:52:34 DEBUG ] 06:52:34 DEBUG } 06:52:34 DEBUG } 06:52:54 INFO 06:52:54 INFO [loop_until]: kubectl --namespace=xlou top pods 06:52:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:52:54 INFO [loop_until]: OK (rc = 0) 06:52:54 DEBUG --- stdout --- 06:52:54 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 9Mi am-654c4fcbcf-487cv 8m 8870Mi am-654c4fcbcf-9scrj 7m 8378Mi am-654c4fcbcf-v5qb5 7m 8096Mi ds-cts-0 18m 12723Mi ds-cts-1 18m 12921Mi ds-cts-2 17m 11473Mi ds-idrepo-0 14m 5615Mi ds-idrepo-1 12m 5677Mi ds-idrepo-2 56m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 5Mi idm-6bfb7c96cc-qklk9 3m 1199Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 5m 73Mi login-ui-5f964fb894-5zpvq 1m 4Mi overseer-0-6f8784665d-zq5hb 273m 1886Mi 06:52:54 DEBUG --- stderr --- 06:52:54 DEBUG 06:53:03 INFO 06:53:03 INFO [loop_until]: kubectl --namespace=xlou top node 06:53:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:53:03 INFO [loop_until]: OK (rc = 0) 06:53:03 DEBUG --- stdout --- 06:53:03 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 93m 0% 2315Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 67m 0% 8987Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 75m 0% 3256Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 195m 1% 6513Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 67m 0% 9284Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 67m 0% 9731Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 74m 0% 858Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 69m 0% 6307Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 72m 0% 13457Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 68m 0% 12208Mi 20% gke-xlou-cdm-ds-32e4dcb1-gxzj 110m 0% 6400Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 73m 0% 13652Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 64m 0% 6369Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 546m 3% 5637Mi 9% 06:53:03 DEBUG --- stderr --- 06:53:03 DEBUG 06:53:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712728410" 06:53:30 ERROR [http_cmd]: ERROR 06:53:30 ERROR --- status code --- 06:53:30 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:53:30 ERROR --- http response --- 06:53:30 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:53:30 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:53:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728410" 06:53:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728410" 06:53:30 INFO [http_cmd]: http status code OK 06:53:30 DEBUG --- status code --- 06:53:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:30 DEBUG --- http response --- 06:53:30 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:53:30 INFO [http_cmd]: http status code OK 06:53:30 DEBUG --- status code --- 06:53:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:30 DEBUG --- http response --- 06:53:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:53:30 DEBUG ----- output ----- 06:53:30 DEBUG { 06:53:30 DEBUG "status": "success", 06:53:30 DEBUG "data": { 06:53:30 DEBUG "resultType": "vector", 06:53:30 DEBUG "result": [ 06:53:30 DEBUG { 06:53:30 DEBUG "metric": { 06:53:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:53:30 DEBUG }, 06:53:30 DEBUG "value": [ 06:53:30 DEBUG 06:53:30 DEBUG 06:53:30 DEBUG 06:53:30 DEBUG [...] 06:53:30 DEBUG 06:53:30 DEBUG 06:53:30 DEBUG 06:53:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:53:30 DEBUG }, 06:53:30 DEBUG "value": [ 06:53:30 DEBUG 1712728410, 06:53:30 DEBUG "0" 06:53:30 DEBUG ] 06:53:30 DEBUG } 06:53:30 DEBUG ] 06:53:30 DEBUG } 06:53:30 DEBUG } 06:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728412" 06:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728412" 06:53:32 INFO [http_cmd]: http status code OK 06:53:32 DEBUG --- status code --- 06:53:32 INFO [http_cmd]: http status code OK 06:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:32 DEBUG --- status code --- 06:53:32 DEBUG --- http response --- 06:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:53:32 DEBUG --- http response --- 06:53:32 DEBUG ----- output ----- 06:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:53:32 DEBUG { 06:53:32 DEBUG ----- output ----- 06:53:32 DEBUG "status": "success", 06:53:32 DEBUG { 06:53:32 DEBUG "data": { 06:53:32 DEBUG "status": "success", 06:53:32 DEBUG "resultType": "vector", 06:53:32 DEBUG "data": { 06:53:32 DEBUG "result": [ 06:53:32 DEBUG "resultType": "vector", 06:53:32 DEBUG { 06:53:32 DEBUG "result": [ 06:53:32 DEBUG "metric": { 06:53:32 DEBUG { 06:53:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:53:32 DEBUG "metric": { 06:53:32 DEBUG }, 06:53:32 DEBUG "instance": "10.142.0.25:9100" 06:53:32 DEBUG "value": [ 06:53:32 DEBUG }, 06:53:32 DEBUG 06:53:32 DEBUG "value": [ 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG [...] 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG [...] 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:53:32 DEBUG 06:53:32 DEBUG }, 06:53:32 DEBUG "instance": "10.142.0.77:9100" 06:53:32 DEBUG "value": [ 06:53:32 DEBUG }, 06:53:32 DEBUG 1712728412, 06:53:32 DEBUG "value": [ 06:53:32 DEBUG "0" 06:53:32 DEBUG 1712728412, 06:53:32 DEBUG ] 06:53:32 DEBUG "25070.3" 06:53:32 DEBUG } 06:53:32 DEBUG ] 06:53:32 DEBUG ] 06:53:32 DEBUG } 06:53:32 DEBUG } 06:53:32 DEBUG ] 06:53:32 DEBUG } 06:53:32 DEBUG } 06:53:32 DEBUG } 06:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728412" 06:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728412" 06:53:32 INFO [http_cmd]: http status code OK 06:53:32 DEBUG --- status code --- 06:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:32 DEBUG --- http response --- 06:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:53:32 DEBUG ----- output ----- 06:53:32 DEBUG { 06:53:32 DEBUG "status": "success", 06:53:32 DEBUG "data": { 06:53:32 DEBUG "resultType": "vector", 06:53:32 DEBUG "result": [ 06:53:32 DEBUG { 06:53:32 DEBUG "metric": { 06:53:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:53:32 DEBUG }, 06:53:32 DEBUG "value": [ 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG [...] 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:53:32 DEBUG }, 06:53:32 DEBUG "value": [ 06:53:32 DEBUG 1712728412, 06:53:32 DEBUG "0" 06:53:32 DEBUG ] 06:53:32 DEBUG } 06:53:32 DEBUG ] 06:53:32 DEBUG } 06:53:32 DEBUG } 06:53:32 INFO [http_cmd]: http status code OK 06:53:32 DEBUG --- status code --- 06:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:32 DEBUG --- http response --- 06:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:53:32 DEBUG ----- output ----- 06:53:32 DEBUG { 06:53:32 DEBUG "status": "success", 06:53:32 DEBUG "data": { 06:53:32 DEBUG "resultType": "vector", 06:53:32 DEBUG "result": [ 06:53:32 DEBUG { 06:53:32 DEBUG "metric": { 06:53:32 DEBUG "pod": "ds-idrepo-0" 06:53:32 DEBUG }, 06:53:32 DEBUG "value": [ 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG [...] 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG "pod": "ds-idrepo-1" 06:53:32 DEBUG }, 06:53:32 DEBUG "value": [ 06:53:32 DEBUG 1712728412, 06:53:32 DEBUG "0" 06:53:32 DEBUG ] 06:53:32 DEBUG } 06:53:32 DEBUG ] 06:53:32 DEBUG } 06:53:32 DEBUG } 06:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728412" 06:53:32 INFO [http_cmd]: http status code OK 06:53:32 DEBUG --- status code --- 06:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:32 DEBUG --- http response --- 06:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:53:32 DEBUG ----- output ----- 06:53:32 DEBUG { 06:53:32 DEBUG "status": "success", 06:53:32 DEBUG "data": { 06:53:32 DEBUG "resultType": "vector", 06:53:32 DEBUG "result": [ 06:53:32 DEBUG { 06:53:32 DEBUG "metric": { 06:53:32 DEBUG "instance": "10.142.0.25:9100" 06:53:32 DEBUG }, 06:53:32 DEBUG "value": [ 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG [...] 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG "instance": "10.142.0.77:9100" 06:53:32 DEBUG }, 06:53:32 DEBUG "value": [ 06:53:32 DEBUG 1712728412, 06:53:32 DEBUG "438082679808" 06:53:32 DEBUG ] 06:53:32 DEBUG } 06:53:32 DEBUG ] 06:53:32 DEBUG } 06:53:32 DEBUG } 06:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728412" 06:53:32 INFO [http_cmd]: http status code OK 06:53:32 DEBUG --- status code --- 06:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:32 DEBUG --- http response --- 06:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:53:32 DEBUG ----- output ----- 06:53:32 DEBUG { 06:53:32 DEBUG "status": "success", 06:53:32 DEBUG "data": { 06:53:32 DEBUG "resultType": "vector", 06:53:32 DEBUG "result": [ 06:53:32 DEBUG { 06:53:32 DEBUG "metric": { 06:53:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:53:32 DEBUG }, 06:53:32 DEBUG "value": [ 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG [...] 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:53:32 DEBUG }, 06:53:32 DEBUG "value": [ 06:53:32 DEBUG 1712728412, 06:53:32 DEBUG "0" 06:53:32 DEBUG ] 06:53:32 DEBUG } 06:53:32 DEBUG ] 06:53:32 DEBUG } 06:53:32 DEBUG } 06:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728412" 06:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728412" 06:53:32 INFO [http_cmd]: http status code OK 06:53:32 DEBUG --- status code --- 06:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:32 DEBUG --- http response --- 06:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:53:32 DEBUG ----- output ----- 06:53:32 DEBUG { 06:53:32 DEBUG "status": "success", 06:53:32 DEBUG "data": { 06:53:32 DEBUG "resultType": "vector", 06:53:32 DEBUG "result": [ 06:53:32 DEBUG { 06:53:32 DEBUG "metric": { 06:53:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:53:32 DEBUG }, 06:53:32 DEBUG "value": [ 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG [...] 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:53:32 DEBUG }, 06:53:32 DEBUG "value": [ 06:53:32 DEBUG 1712728412, 06:53:32 DEBUG "0" 06:53:32 DEBUG ] 06:53:32 DEBUG } 06:53:32 DEBUG ] 06:53:32 DEBUG } 06:53:32 DEBUG } 06:53:32 INFO [http_cmd]: http status code OK 06:53:32 DEBUG --- status code --- 06:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:32 DEBUG --- http response --- 06:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:53:32 DEBUG ----- output ----- 06:53:32 DEBUG { 06:53:32 DEBUG "status": "success", 06:53:32 DEBUG "data": { 06:53:32 DEBUG "resultType": "vector", 06:53:32 DEBUG "result": [ 06:53:32 DEBUG { 06:53:32 DEBUG "metric": { 06:53:32 DEBUG "instance": "10.142.0.25:9100" 06:53:32 DEBUG }, 06:53:32 DEBUG "value": [ 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG [...] 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG "instance": "10.142.0.77:9100" 06:53:32 DEBUG }, 06:53:32 DEBUG "value": [ 06:53:32 DEBUG 1712728412, 06:53:32 DEBUG "0.0006666666666641655" 06:53:32 DEBUG ] 06:53:32 DEBUG } 06:53:32 DEBUG ] 06:53:32 DEBUG } 06:53:32 DEBUG } 06:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712728412" 06:53:32 INFO [http_cmd]: http status code OK 06:53:32 DEBUG --- status code --- 06:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:32 DEBUG --- http response --- 06:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:53:32 DEBUG ----- output ----- 06:53:32 DEBUG { 06:53:32 DEBUG "status": "success", 06:53:32 DEBUG "data": { 06:53:32 DEBUG "resultType": "vector", 06:53:32 DEBUG "result": [ 06:53:32 DEBUG { 06:53:32 DEBUG "metric": { 06:53:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:53:32 DEBUG }, 06:53:32 DEBUG "value": [ 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG [...] 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:53:32 DEBUG }, 06:53:32 DEBUG "value": [ 06:53:32 DEBUG 1712728412, 06:53:32 DEBUG "1.2365197671803603" 06:53:32 DEBUG ] 06:53:32 DEBUG } 06:53:32 DEBUG ] 06:53:32 DEBUG } 06:53:32 DEBUG } 06:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728412" 06:53:32 INFO [http_cmd]: http status code OK 06:53:32 DEBUG --- status code --- 06:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:32 DEBUG --- http response --- 06:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:53:32 DEBUG ----- output ----- 06:53:32 DEBUG { 06:53:32 DEBUG "status": "success", 06:53:32 DEBUG "data": { 06:53:32 DEBUG "resultType": "vector", 06:53:32 DEBUG "result": [ 06:53:32 DEBUG { 06:53:32 DEBUG "metric": { 06:53:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:53:32 DEBUG }, 06:53:32 DEBUG "value": [ 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG [...] 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:53:32 DEBUG }, 06:53:32 DEBUG "value": [ 06:53:32 DEBUG 1712728412, 06:53:32 DEBUG "0" 06:53:32 DEBUG ] 06:53:32 DEBUG } 06:53:32 DEBUG ] 06:53:32 DEBUG } 06:53:32 DEBUG } 06:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728412" 06:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728412" 06:53:32 INFO [http_cmd]: http status code OK 06:53:32 DEBUG --- status code --- 06:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:32 DEBUG --- http response --- 06:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:53:32 DEBUG ----- output ----- 06:53:32 DEBUG { 06:53:32 DEBUG "status": "success", 06:53:32 DEBUG "data": { 06:53:32 DEBUG "resultType": "vector", 06:53:32 DEBUG "result": [ 06:53:32 DEBUG { 06:53:32 DEBUG "metric": { 06:53:32 DEBUG "pod": "ds-idrepo-0" 06:53:32 DEBUG }, 06:53:32 DEBUG "value": [ 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG [...] 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG "pod": "ds-cts-2" 06:53:32 DEBUG }, 06:53:32 DEBUG "value": [ 06:53:32 DEBUG 1712728412, 06:53:32 DEBUG "29.3" 06:53:32 DEBUG ] 06:53:32 DEBUG } 06:53:32 DEBUG ] 06:53:32 DEBUG } 06:53:32 DEBUG } 06:53:32 INFO [http_cmd]: http status code OK 06:53:32 DEBUG --- status code --- 06:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:32 DEBUG --- http response --- 06:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:53:32 DEBUG ----- output ----- 06:53:32 DEBUG { 06:53:32 DEBUG "status": "success", 06:53:32 DEBUG "data": { 06:53:32 DEBUG "resultType": "vector", 06:53:32 DEBUG "result": [ 06:53:32 DEBUG { 06:53:32 DEBUG "metric": { 06:53:32 DEBUG "pod": "ds-idrepo-0" 06:53:32 DEBUG }, 06:53:32 DEBUG "value": [ 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG [...] 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG "pod": "ds-cts-2" 06:53:32 DEBUG }, 06:53:32 DEBUG "value": [ 06:53:32 DEBUG 1712728412, 06:53:32 DEBUG "0" 06:53:32 DEBUG ] 06:53:32 DEBUG } 06:53:32 DEBUG ] 06:53:32 DEBUG } 06:53:32 DEBUG } 06:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712728412" 06:53:32 INFO [http_cmd]: http status code OK 06:53:32 DEBUG --- status code --- 06:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:32 DEBUG --- http response --- 06:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:53:32 DEBUG ----- output ----- 06:53:32 DEBUG { 06:53:32 DEBUG "status": "success", 06:53:32 DEBUG "data": { 06:53:32 DEBUG "resultType": "vector", 06:53:32 DEBUG "result": [ 06:53:32 DEBUG { 06:53:32 DEBUG "metric": { 06:53:32 DEBUG "pod": "ds-idrepo-0" 06:53:32 DEBUG }, 06:53:32 DEBUG "value": [ 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG [...] 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG "pod": "ds-cts-2" 06:53:32 DEBUG }, 06:53:32 DEBUG "value": [ 06:53:32 DEBUG 1712728412, 06:53:32 DEBUG "0" 06:53:32 DEBUG ] 06:53:32 DEBUG } 06:53:32 DEBUG ] 06:53:32 DEBUG } 06:53:32 DEBUG } 06:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728412" 06:53:32 INFO [http_cmd]: http status code OK 06:53:32 DEBUG --- status code --- 06:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:32 DEBUG --- http response --- 06:53:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:53:32 DEBUG ----- output ----- 06:53:32 DEBUG { 06:53:32 DEBUG "status": "success", 06:53:32 DEBUG "data": { 06:53:32 DEBUG "resultType": "vector", 06:53:32 DEBUG "result": [ 06:53:32 DEBUG { 06:53:32 DEBUG "metric": { 06:53:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:53:32 DEBUG }, 06:53:32 DEBUG "value": [ 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG [...] 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG 06:53:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:53:32 DEBUG }, 06:53:32 DEBUG "value": [ 06:53:32 DEBUG 1712728412, 06:53:32 DEBUG "0" 06:53:32 DEBUG ] 06:53:32 DEBUG } 06:53:32 DEBUG ] 06:53:32 DEBUG } 06:53:32 DEBUG } 06:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728413" 06:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712728413" 06:53:33 INFO [http_cmd]: http status code OK 06:53:33 DEBUG --- status code --- 06:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:33 DEBUG --- http response --- 06:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:53:33 DEBUG ----- output ----- 06:53:33 DEBUG { 06:53:33 DEBUG "status": "success", 06:53:33 DEBUG "data": { 06:53:33 DEBUG "resultType": "vector", 06:53:33 DEBUG "result": [ 06:53:33 DEBUG { 06:53:33 DEBUG "metric": { 06:53:33 DEBUG "pod": "ds-idrepo-2" 06:53:33 DEBUG }, 06:53:33 DEBUG "value": [ 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG [...] 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG "pod": "ds-cts-2" 06:53:33 DEBUG }, 06:53:33 DEBUG "value": [ 06:53:33 DEBUG 1712728413, 06:53:33 DEBUG "0" 06:53:33 DEBUG ] 06:53:33 DEBUG } 06:53:33 DEBUG ] 06:53:33 DEBUG } 06:53:33 DEBUG } 06:53:33 INFO [http_cmd]: http status code OK 06:53:33 DEBUG --- status code --- 06:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:33 DEBUG --- http response --- 06:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:53:33 DEBUG ----- output ----- 06:53:33 DEBUG { 06:53:33 DEBUG "status": "success", 06:53:33 DEBUG "data": { 06:53:33 DEBUG "resultType": "vector", 06:53:33 DEBUG "result": [ 06:53:33 DEBUG { 06:53:33 DEBUG "metric": { 06:53:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:53:33 DEBUG }, 06:53:33 DEBUG "value": [ 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG [...] 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG "pod": "am-654c4fcbcf-487cv" 06:53:33 DEBUG }, 06:53:33 DEBUG "value": [ 06:53:33 DEBUG 1712728413, 06:53:33 DEBUG "0.053" 06:53:33 DEBUG ] 06:53:33 DEBUG } 06:53:33 DEBUG ] 06:53:33 DEBUG } 06:53:33 DEBUG } 06:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728413" 06:53:33 INFO [http_cmd]: http status code OK 06:53:33 DEBUG --- status code --- 06:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:33 DEBUG --- http response --- 06:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:53:33 DEBUG ----- output ----- 06:53:33 DEBUG { 06:53:33 DEBUG "status": "success", 06:53:33 DEBUG "data": { 06:53:33 DEBUG "resultType": "vector", 06:53:33 DEBUG "result": [ 06:53:33 DEBUG { 06:53:33 DEBUG "metric": { 06:53:33 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:53:33 DEBUG }, 06:53:33 DEBUG "value": [ 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG [...] 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:53:33 DEBUG }, 06:53:33 DEBUG "value": [ 06:53:33 DEBUG 1712728413, 06:53:33 DEBUG "0" 06:53:33 DEBUG ] 06:53:33 DEBUG } 06:53:33 DEBUG ] 06:53:33 DEBUG } 06:53:33 DEBUG } 06:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728413" 06:53:33 INFO [http_cmd]: http status code OK 06:53:33 DEBUG --- status code --- 06:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:33 DEBUG --- http response --- 06:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:53:33 DEBUG ----- output ----- 06:53:33 DEBUG { 06:53:33 DEBUG "status": "success", 06:53:33 DEBUG "data": { 06:53:33 DEBUG "resultType": "vector", 06:53:33 DEBUG "result": [ 06:53:33 DEBUG { 06:53:33 DEBUG "metric": { 06:53:33 DEBUG "pod": "ds-idrepo-0" 06:53:33 DEBUG }, 06:53:33 DEBUG "value": [ 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG [...] 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG "pod": "ds-idrepo-1" 06:53:33 DEBUG }, 06:53:33 DEBUG "value": [ 06:53:33 DEBUG 1712728413, 06:53:33 DEBUG "0" 06:53:33 DEBUG ] 06:53:33 DEBUG } 06:53:33 DEBUG ] 06:53:33 DEBUG } 06:53:33 DEBUG } 06:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728413" 06:53:33 INFO [http_cmd]: http status code OK 06:53:33 DEBUG --- status code --- 06:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:33 DEBUG --- http response --- 06:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:53:33 DEBUG ----- output ----- 06:53:33 DEBUG { 06:53:33 DEBUG "status": "success", 06:53:33 DEBUG "data": { 06:53:33 DEBUG "resultType": "vector", 06:53:33 DEBUG "result": [ 06:53:33 DEBUG { 06:53:33 DEBUG "metric": { 06:53:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:53:33 DEBUG }, 06:53:33 DEBUG "value": [ 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG [...] 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG "pod": "am-654c4fcbcf-487cv" 06:53:33 DEBUG }, 06:53:33 DEBUG "value": [ 06:53:33 DEBUG 1712728413, 06:53:33 DEBUG "0" 06:53:33 DEBUG ] 06:53:33 DEBUG } 06:53:33 DEBUG ] 06:53:33 DEBUG } 06:53:33 DEBUG } 06:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728413" 06:53:33 INFO [http_cmd]: http status code OK 06:53:33 DEBUG --- status code --- 06:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:33 DEBUG --- http response --- 06:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:53:33 DEBUG ----- output ----- 06:53:33 DEBUG { 06:53:33 DEBUG "status": "success", 06:53:33 DEBUG "data": { 06:53:33 DEBUG "resultType": "vector", 06:53:33 DEBUG "result": [ 06:53:33 DEBUG { 06:53:33 DEBUG "metric": { 06:53:33 DEBUG "pod": "ds-idrepo-0" 06:53:33 DEBUG }, 06:53:33 DEBUG "value": [ 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG [...] 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG "pod": "ds-idrepo-1" 06:53:33 DEBUG }, 06:53:33 DEBUG "value": [ 06:53:33 DEBUG 1712728413, 06:53:33 DEBUG "0" 06:53:33 DEBUG ] 06:53:33 DEBUG } 06:53:33 DEBUG ] 06:53:33 DEBUG } 06:53:33 DEBUG } 06:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728413" 06:53:33 INFO [http_cmd]: http status code OK 06:53:33 DEBUG --- status code --- 06:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:33 DEBUG --- http response --- 06:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:53:33 DEBUG ----- output ----- 06:53:33 DEBUG { 06:53:33 DEBUG "status": "success", 06:53:33 DEBUG "data": { 06:53:33 DEBUG "resultType": "vector", 06:53:33 DEBUG "result": [ 06:53:33 DEBUG { 06:53:33 DEBUG "metric": { 06:53:33 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:53:33 DEBUG }, 06:53:33 DEBUG "value": [ 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG [...] 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:53:33 DEBUG }, 06:53:33 DEBUG "value": [ 06:53:33 DEBUG 1712728413, 06:53:33 DEBUG "0" 06:53:33 DEBUG ] 06:53:33 DEBUG } 06:53:33 DEBUG ] 06:53:33 DEBUG } 06:53:33 DEBUG } 06:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712728413" 06:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728413" 06:53:33 INFO [http_cmd]: http status code OK 06:53:33 DEBUG --- status code --- 06:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:33 DEBUG --- http response --- 06:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:53:33 INFO [http_cmd]: http status code OK 06:53:33 DEBUG ----- output ----- 06:53:33 DEBUG --- status code --- 06:53:33 DEBUG { 06:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:33 DEBUG "status": "success", 06:53:33 DEBUG --- http response --- 06:53:33 DEBUG "data": { 06:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:53:33 DEBUG "resultType": "vector", 06:53:33 DEBUG ----- output ----- 06:53:33 DEBUG "result": [ 06:53:33 DEBUG { 06:53:33 DEBUG { 06:53:33 DEBUG "status": "success", 06:53:33 DEBUG "metric": { 06:53:33 DEBUG "data": { 06:53:33 DEBUG "instance": "10.142.0.25:9100" 06:53:33 DEBUG "resultType": "vector", 06:53:33 DEBUG }, 06:53:33 DEBUG "result": [ 06:53:33 DEBUG "value": [ 06:53:33 DEBUG { 06:53:33 DEBUG 06:53:33 DEBUG "metric": { 06:53:33 DEBUG 06:53:33 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:53:33 DEBUG 06:53:33 DEBUG }, 06:53:33 DEBUG [...] 06:53:33 DEBUG "value": [ 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG 06:53:33 DEBUG "instance": "10.142.0.77:9100" 06:53:33 DEBUG [...] 06:53:33 DEBUG }, 06:53:33 DEBUG 06:53:33 DEBUG "value": [ 06:53:33 DEBUG 06:53:33 DEBUG 1712728413, 06:53:33 DEBUG 06:53:33 DEBUG "26157.13333333333" 06:53:33 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:53:33 DEBUG ] 06:53:33 DEBUG }, 06:53:33 DEBUG } 06:53:33 DEBUG "value": [ 06:53:33 DEBUG ] 06:53:33 DEBUG 1712728413, 06:53:33 DEBUG } 06:53:33 DEBUG "2488963072" 06:53:33 DEBUG } 06:53:33 DEBUG ] 06:53:33 DEBUG } 06:53:33 DEBUG ] 06:53:33 DEBUG } 06:53:33 DEBUG } 06:53:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728414" 06:53:34 INFO [http_cmd]: http status code OK 06:53:34 DEBUG --- status code --- 06:53:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:34 DEBUG --- http response --- 06:53:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:53:34 DEBUG ----- output ----- 06:53:34 DEBUG { 06:53:34 DEBUG "status": "success", 06:53:34 DEBUG "data": { 06:53:34 DEBUG "resultType": "vector", 06:53:34 DEBUG "result": [ 06:53:34 DEBUG { 06:53:34 DEBUG "metric": { 06:53:34 DEBUG "instance": "10.142.0.25:9100" 06:53:34 DEBUG }, 06:53:34 DEBUG "value": [ 06:53:34 DEBUG 06:53:34 DEBUG 06:53:34 DEBUG 06:53:34 DEBUG [...] 06:53:34 DEBUG 06:53:34 DEBUG 06:53:34 DEBUG 06:53:34 DEBUG "instance": "10.142.0.77:9100" 06:53:34 DEBUG }, 06:53:34 DEBUG "value": [ 06:53:34 DEBUG 1712728414, 06:53:34 DEBUG "0" 06:53:34 DEBUG ] 06:53:34 DEBUG } 06:53:34 DEBUG ] 06:53:34 DEBUG } 06:53:34 DEBUG } 06:53:54 INFO 06:53:54 INFO [loop_until]: kubectl --namespace=xlou top pods 06:53:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:53:54 INFO [loop_until]: OK (rc = 0) 06:53:54 DEBUG --- stdout --- 06:53:54 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 9Mi am-654c4fcbcf-487cv 7m 8870Mi am-654c4fcbcf-9scrj 8m 8378Mi am-654c4fcbcf-v5qb5 7m 8096Mi ds-cts-0 16m 12723Mi ds-cts-1 16m 12921Mi ds-cts-2 13m 11473Mi ds-idrepo-0 15m 5615Mi ds-idrepo-1 12m 5677Mi ds-idrepo-2 13m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 5Mi idm-6bfb7c96cc-qklk9 4m 1199Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 4m 73Mi login-ui-5f964fb894-5zpvq 1m 4Mi overseer-0-6f8784665d-zq5hb 978m 2524Mi 06:53:54 DEBUG --- stderr --- 06:53:54 DEBUG 06:54:03 INFO 06:54:03 INFO [loop_until]: kubectl --namespace=xlou top node 06:54:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:54:03 INFO [loop_until]: OK (rc = 0) 06:54:03 DEBUG --- stdout --- 06:54:03 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 94m 0% 2313Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 68m 0% 8990Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 75m 0% 3258Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 218m 1% 6532Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 69m 0% 9283Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 65m 0% 9736Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 75m 0% 862Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 72m 0% 6309Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 73m 0% 13458Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 69m 0% 12211Mi 20% gke-xlou-cdm-ds-32e4dcb1-gxzj 65m 0% 6402Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 71m 0% 13653Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 66m 0% 6367Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 1074m 6% 6168Mi 10% 06:54:03 DEBUG --- stderr --- 06:54:03 DEBUG 06:54:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712728470" 06:54:30 ERROR [http_cmd]: ERROR 06:54:30 ERROR --- status code --- 06:54:30 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:54:30 ERROR --- http response --- 06:54:30 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:54:30 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:54:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728470" 06:54:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728470" 06:54:30 INFO [http_cmd]: http status code OK 06:54:30 INFO [http_cmd]: http status code OK 06:54:30 DEBUG --- status code --- 06:54:30 DEBUG --- status code --- 06:54:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:30 DEBUG --- http response --- 06:54:30 DEBUG --- http response --- 06:54:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:54:30 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:54:30 DEBUG ----- output ----- 06:54:30 DEBUG { 06:54:30 DEBUG "status": "success", 06:54:30 DEBUG "data": { 06:54:30 DEBUG "resultType": "vector", 06:54:30 DEBUG "result": [ 06:54:30 DEBUG { 06:54:30 DEBUG "metric": { 06:54:30 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:54:30 DEBUG }, 06:54:30 DEBUG "value": [ 06:54:30 DEBUG 06:54:30 DEBUG 06:54:30 DEBUG 06:54:30 DEBUG [...] 06:54:30 DEBUG 06:54:30 DEBUG 06:54:30 DEBUG 06:54:30 DEBUG "pod": "am-654c4fcbcf-487cv" 06:54:30 DEBUG }, 06:54:30 DEBUG "value": [ 06:54:30 DEBUG 1712728470, 06:54:30 DEBUG "0" 06:54:30 DEBUG ] 06:54:30 DEBUG } 06:54:30 DEBUG ] 06:54:30 DEBUG } 06:54:30 DEBUG } 06:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728472" 06:54:32 INFO [http_cmd]: http status code OK 06:54:32 DEBUG --- status code --- 06:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:32 DEBUG --- http response --- 06:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:54:32 DEBUG ----- output ----- 06:54:32 DEBUG { 06:54:32 DEBUG "status": "success", 06:54:32 DEBUG "data": { 06:54:32 DEBUG "resultType": "vector", 06:54:32 DEBUG "result": [ 06:54:32 DEBUG { 06:54:32 DEBUG "metric": { 06:54:32 DEBUG "pod": "ds-idrepo-0" 06:54:32 DEBUG }, 06:54:32 DEBUG "value": [ 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG [...] 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG "pod": "ds-idrepo-1" 06:54:32 DEBUG }, 06:54:32 DEBUG "value": [ 06:54:32 DEBUG 1712728472, 06:54:32 DEBUG "0" 06:54:32 DEBUG ] 06:54:32 DEBUG } 06:54:32 DEBUG ] 06:54:32 DEBUG } 06:54:32 DEBUG } 06:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728472" 06:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728472" 06:54:32 INFO [http_cmd]: http status code OK 06:54:32 DEBUG --- status code --- 06:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:32 DEBUG --- http response --- 06:54:32 INFO [http_cmd]: http status code OK 06:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:54:32 DEBUG --- status code --- 06:54:32 DEBUG ----- output ----- 06:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:32 DEBUG { 06:54:32 DEBUG --- http response --- 06:54:32 DEBUG "status": "success", 06:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:54:32 DEBUG "data": { 06:54:32 DEBUG ----- output ----- 06:54:32 DEBUG "resultType": "vector", 06:54:32 DEBUG { 06:54:32 DEBUG "result": [ 06:54:32 DEBUG "status": "success", 06:54:32 DEBUG { 06:54:32 DEBUG "data": { 06:54:32 DEBUG "metric": { 06:54:32 DEBUG "resultType": "vector", 06:54:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:54:32 DEBUG "result": [ 06:54:32 DEBUG }, 06:54:32 DEBUG { 06:54:32 DEBUG "value": [ 06:54:32 DEBUG "metric": { 06:54:32 DEBUG 06:54:32 DEBUG "instance": "10.142.0.25:9100" 06:54:32 DEBUG 06:54:32 DEBUG }, 06:54:32 DEBUG 06:54:32 DEBUG "value": [ 06:54:32 DEBUG [...] 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG [...] 06:54:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:54:32 DEBUG 06:54:32 DEBUG }, 06:54:32 DEBUG 06:54:32 DEBUG "value": [ 06:54:32 DEBUG 06:54:32 DEBUG 1712728472, 06:54:32 DEBUG "instance": "10.142.0.77:9100" 06:54:32 DEBUG "0" 06:54:32 DEBUG }, 06:54:32 DEBUG ] 06:54:32 DEBUG "value": [ 06:54:32 DEBUG } 06:54:32 DEBUG 1712728472, 06:54:32 DEBUG ] 06:54:32 DEBUG "24969.899999999998" 06:54:32 DEBUG } 06:54:32 DEBUG ] 06:54:32 DEBUG } 06:54:32 DEBUG } 06:54:32 DEBUG ] 06:54:32 DEBUG } 06:54:32 DEBUG } 06:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728472" 06:54:32 INFO [http_cmd]: http status code OK 06:54:32 DEBUG --- status code --- 06:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:32 DEBUG --- http response --- 06:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:54:32 DEBUG ----- output ----- 06:54:32 DEBUG { 06:54:32 DEBUG "status": "success", 06:54:32 DEBUG "data": { 06:54:32 DEBUG "resultType": "vector", 06:54:32 DEBUG "result": [ 06:54:32 DEBUG { 06:54:32 DEBUG "metric": { 06:54:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:54:32 DEBUG }, 06:54:32 DEBUG "value": [ 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG [...] 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:54:32 DEBUG }, 06:54:32 DEBUG "value": [ 06:54:32 DEBUG 1712728472, 06:54:32 DEBUG "0" 06:54:32 DEBUG ] 06:54:32 DEBUG } 06:54:32 DEBUG ] 06:54:32 DEBUG } 06:54:32 DEBUG } 06:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728472" 06:54:32 INFO [http_cmd]: http status code OK 06:54:32 DEBUG --- status code --- 06:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:32 DEBUG --- http response --- 06:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:54:32 DEBUG ----- output ----- 06:54:32 DEBUG { 06:54:32 DEBUG "status": "success", 06:54:32 DEBUG "data": { 06:54:32 DEBUG "resultType": "vector", 06:54:32 DEBUG "result": [ 06:54:32 DEBUG { 06:54:32 DEBUG "metric": { 06:54:32 DEBUG "instance": "10.142.0.25:9100" 06:54:32 DEBUG }, 06:54:32 DEBUG "value": [ 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG [...] 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG "instance": "10.142.0.77:9100" 06:54:32 DEBUG }, 06:54:32 DEBUG "value": [ 06:54:32 DEBUG 1712728472, 06:54:32 DEBUG "438084191232" 06:54:32 DEBUG ] 06:54:32 DEBUG } 06:54:32 DEBUG ] 06:54:32 DEBUG } 06:54:32 DEBUG } 06:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728472" 06:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712728472" 06:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728472" 06:54:32 INFO [http_cmd]: http status code OK 06:54:32 DEBUG --- status code --- 06:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:32 DEBUG --- http response --- 06:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:54:32 DEBUG ----- output ----- 06:54:32 DEBUG { 06:54:32 DEBUG "status": "success", 06:54:32 DEBUG "data": { 06:54:32 DEBUG "resultType": "vector", 06:54:32 DEBUG "result": [ 06:54:32 DEBUG { 06:54:32 DEBUG "metric": { 06:54:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:54:32 DEBUG }, 06:54:32 DEBUG "value": [ 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG [...] 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:54:32 DEBUG }, 06:54:32 DEBUG "value": [ 06:54:32 DEBUG 1712728472, 06:54:32 DEBUG "0" 06:54:32 DEBUG ] 06:54:32 DEBUG } 06:54:32 DEBUG ] 06:54:32 DEBUG } 06:54:32 DEBUG } 06:54:32 INFO [http_cmd]: http status code OK 06:54:32 DEBUG --- status code --- 06:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:32 DEBUG --- http response --- 06:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:54:32 DEBUG ----- output ----- 06:54:32 DEBUG { 06:54:32 DEBUG "status": "success", 06:54:32 DEBUG "data": { 06:54:32 DEBUG "resultType": "vector", 06:54:32 DEBUG "result": [ 06:54:32 DEBUG { 06:54:32 DEBUG "metric": { 06:54:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:54:32 DEBUG }, 06:54:32 DEBUG "value": [ 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG [...] 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:54:32 DEBUG }, 06:54:32 DEBUG "value": [ 06:54:32 DEBUG 1712728472, 06:54:32 DEBUG "1.0307335145900922" 06:54:32 DEBUG ] 06:54:32 DEBUG } 06:54:32 DEBUG ] 06:54:32 DEBUG } 06:54:32 DEBUG } 06:54:32 INFO [http_cmd]: http status code OK 06:54:32 DEBUG --- status code --- 06:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:32 DEBUG --- http response --- 06:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:54:32 DEBUG ----- output ----- 06:54:32 DEBUG { 06:54:32 DEBUG "status": "success", 06:54:32 DEBUG "data": { 06:54:32 DEBUG "resultType": "vector", 06:54:32 DEBUG "result": [ 06:54:32 DEBUG { 06:54:32 DEBUG "metric": { 06:54:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:54:32 DEBUG }, 06:54:32 DEBUG "value": [ 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG [...] 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG "pod": "login-ui-5f964fb894-5zpvq" 06:54:32 DEBUG }, 06:54:32 DEBUG "value": [ 06:54:32 DEBUG 1712728472, 06:54:32 DEBUG "0" 06:54:32 DEBUG ] 06:54:32 DEBUG } 06:54:32 DEBUG ] 06:54:32 DEBUG } 06:54:32 DEBUG } 06:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728472" 06:54:32 INFO [http_cmd]: http status code OK 06:54:32 DEBUG --- status code --- 06:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:32 DEBUG --- http response --- 06:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:54:32 DEBUG ----- output ----- 06:54:32 DEBUG { 06:54:32 DEBUG "status": "success", 06:54:32 DEBUG "data": { 06:54:32 DEBUG "resultType": "vector", 06:54:32 DEBUG "result": [ 06:54:32 DEBUG { 06:54:32 DEBUG "metric": { 06:54:32 DEBUG "instance": "10.142.0.25:9100" 06:54:32 DEBUG }, 06:54:32 DEBUG "value": [ 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG [...] 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG "instance": "10.142.0.77:9100" 06:54:32 DEBUG }, 06:54:32 DEBUG "value": [ 06:54:32 DEBUG 1712728472, 06:54:32 DEBUG "0.0006333333333299625" 06:54:32 DEBUG ] 06:54:32 DEBUG } 06:54:32 DEBUG ] 06:54:32 DEBUG } 06:54:32 DEBUG } 06:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728472" 06:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728472" 06:54:32 INFO [http_cmd]: http status code OK 06:54:32 DEBUG --- status code --- 06:54:32 INFO [http_cmd]: http status code OK 06:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:32 DEBUG --- status code --- 06:54:32 DEBUG --- http response --- 06:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:54:32 DEBUG --- http response --- 06:54:32 DEBUG ----- output ----- 06:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:54:32 DEBUG { 06:54:32 DEBUG ----- output ----- 06:54:32 DEBUG "status": "success", 06:54:32 DEBUG { 06:54:32 DEBUG "data": { 06:54:32 DEBUG "status": "success", 06:54:32 DEBUG "resultType": "vector", 06:54:32 DEBUG "data": { 06:54:32 DEBUG "result": [ 06:54:32 DEBUG "resultType": "vector", 06:54:32 DEBUG { 06:54:32 DEBUG "result": [ 06:54:32 DEBUG "metric": { 06:54:32 DEBUG { 06:54:32 DEBUG "pod": "ds-idrepo-0" 06:54:32 DEBUG "metric": { 06:54:32 DEBUG }, 06:54:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:54:32 DEBUG "value": [ 06:54:32 DEBUG }, 06:54:32 DEBUG 06:54:32 DEBUG "value": [ 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG [...] 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG [...] 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG "pod": "ds-cts-2" 06:54:32 DEBUG 06:54:32 DEBUG }, 06:54:32 DEBUG "pod": "login-ui-5f964fb894-5zpvq" 06:54:32 DEBUG "value": [ 06:54:32 DEBUG }, 06:54:32 DEBUG 1712728472, 06:54:32 DEBUG "value": [ 06:54:32 DEBUG "29.599999999999998" 06:54:32 DEBUG 1712728472, 06:54:32 DEBUG ] 06:54:32 DEBUG "0" 06:54:32 DEBUG } 06:54:32 DEBUG ] 06:54:32 DEBUG ] 06:54:32 DEBUG } 06:54:32 DEBUG } 06:54:32 DEBUG ] 06:54:32 DEBUG } 06:54:32 DEBUG } 06:54:32 DEBUG } 06:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712728472" 06:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728472" 06:54:32 INFO [http_cmd]: http status code OK 06:54:32 DEBUG --- status code --- 06:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:32 DEBUG --- http response --- 06:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:54:32 DEBUG ----- output ----- 06:54:32 DEBUG { 06:54:32 DEBUG "status": "success", 06:54:32 DEBUG "data": { 06:54:32 DEBUG "resultType": "vector", 06:54:32 DEBUG "result": [ 06:54:32 DEBUG { 06:54:32 DEBUG "metric": { 06:54:32 DEBUG "pod": "ds-idrepo-0" 06:54:32 DEBUG }, 06:54:32 DEBUG "value": [ 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG [...] 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG "pod": "ds-cts-2" 06:54:32 DEBUG }, 06:54:32 DEBUG "value": [ 06:54:32 DEBUG 1712728472, 06:54:32 DEBUG "0" 06:54:32 DEBUG ] 06:54:32 DEBUG } 06:54:32 DEBUG ] 06:54:32 DEBUG } 06:54:32 DEBUG } 06:54:32 INFO [http_cmd]: http status code OK 06:54:32 DEBUG --- status code --- 06:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:32 DEBUG --- http response --- 06:54:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:54:32 DEBUG ----- output ----- 06:54:32 DEBUG { 06:54:32 DEBUG "status": "success", 06:54:32 DEBUG "data": { 06:54:32 DEBUG "resultType": "vector", 06:54:32 DEBUG "result": [ 06:54:32 DEBUG { 06:54:32 DEBUG "metric": { 06:54:32 DEBUG "pod": "ds-idrepo-0" 06:54:32 DEBUG }, 06:54:32 DEBUG "value": [ 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG [...] 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG 06:54:32 DEBUG "pod": "ds-cts-2" 06:54:32 DEBUG }, 06:54:32 DEBUG "value": [ 06:54:32 DEBUG 1712728472, 06:54:32 DEBUG "0" 06:54:32 DEBUG ] 06:54:32 DEBUG } 06:54:32 DEBUG ] 06:54:32 DEBUG } 06:54:32 DEBUG } 06:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728472" 06:54:33 INFO [http_cmd]: http status code OK 06:54:33 DEBUG --- status code --- 06:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:33 DEBUG --- http response --- 06:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:54:33 DEBUG ----- output ----- 06:54:33 DEBUG { 06:54:33 DEBUG "status": "success", 06:54:33 DEBUG "data": { 06:54:33 DEBUG "resultType": "vector", 06:54:33 DEBUG "result": [ 06:54:33 DEBUG { 06:54:33 DEBUG "metric": { 06:54:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:54:33 DEBUG }, 06:54:33 DEBUG "value": [ 06:54:33 DEBUG 06:54:33 DEBUG 06:54:33 DEBUG 06:54:33 DEBUG [...] 06:54:33 DEBUG 06:54:33 DEBUG 06:54:33 DEBUG 06:54:33 DEBUG "pod": "am-654c4fcbcf-487cv" 06:54:33 DEBUG }, 06:54:33 DEBUG "value": [ 06:54:33 DEBUG 1712728472, 06:54:33 DEBUG "0" 06:54:33 DEBUG ] 06:54:33 DEBUG } 06:54:33 DEBUG ] 06:54:33 DEBUG } 06:54:33 DEBUG } 06:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728473" 06:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712728473" 06:54:33 INFO [http_cmd]: http status code OK 06:54:33 DEBUG --- status code --- 06:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:33 DEBUG --- http response --- 06:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:54:33 DEBUG ----- output ----- 06:54:33 DEBUG { 06:54:33 DEBUG "status": "success", 06:54:33 DEBUG "data": { 06:54:33 DEBUG "resultType": "vector", 06:54:33 DEBUG "result": [ 06:54:33 DEBUG { 06:54:33 DEBUG "metric": { 06:54:33 DEBUG "pod": "ds-idrepo-2" 06:54:33 DEBUG }, 06:54:33 DEBUG "value": [ 06:54:33 DEBUG 06:54:33 DEBUG 06:54:33 DEBUG 06:54:33 DEBUG [...] 06:54:33 DEBUG 06:54:33 DEBUG 06:54:33 DEBUG 06:54:33 DEBUG "pod": "ds-cts-2" 06:54:33 DEBUG }, 06:54:33 DEBUG "value": [ 06:54:33 DEBUG 1712728473, 06:54:33 DEBUG "0" 06:54:33 DEBUG ] 06:54:33 DEBUG } 06:54:33 DEBUG ] 06:54:33 DEBUG } 06:54:33 DEBUG } 06:54:33 INFO [http_cmd]: http status code OK 06:54:33 DEBUG --- status code --- 06:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:33 DEBUG --- http response --- 06:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:54:33 DEBUG ----- output ----- 06:54:33 DEBUG { 06:54:33 DEBUG "status": "success", 06:54:33 DEBUG "data": { 06:54:33 DEBUG "resultType": "vector", 06:54:33 DEBUG "result": [ 06:54:33 DEBUG { 06:54:33 DEBUG "metric": { 06:54:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:54:33 DEBUG }, 06:54:33 DEBUG "value": [ 06:54:33 DEBUG 06:54:33 DEBUG 06:54:33 DEBUG 06:54:33 DEBUG [...] 06:54:33 DEBUG 06:54:33 DEBUG 06:54:33 DEBUG 06:54:33 DEBUG "pod": "am-654c4fcbcf-487cv" 06:54:33 DEBUG }, 06:54:33 DEBUG "value": [ 06:54:33 DEBUG 1712728473, 06:54:33 DEBUG "0.054" 06:54:33 DEBUG ] 06:54:33 DEBUG } 06:54:33 DEBUG ] 06:54:33 DEBUG } 06:54:33 DEBUG } 06:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728473" 06:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728473" 06:54:33 INFO [http_cmd]: http status code OK 06:54:33 DEBUG --- status code --- 06:54:33 INFO [http_cmd]: http status code OK 06:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:33 DEBUG --- status code --- 06:54:33 DEBUG --- http response --- 06:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:54:33 DEBUG --- http response --- 06:54:33 DEBUG ----- output ----- 06:54:33 DEBUG { 06:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:54:33 DEBUG "status": "success", 06:54:33 DEBUG ----- output ----- 06:54:33 DEBUG "data": { 06:54:33 DEBUG { 06:54:33 DEBUG "resultType": "vector", 06:54:33 DEBUG "status": "success", 06:54:33 DEBUG "result": [ 06:54:33 DEBUG "data": { 06:54:33 DEBUG { 06:54:33 DEBUG "resultType": "vector", 06:54:33 DEBUG "metric": { 06:54:33 DEBUG "result": [ 06:54:33 DEBUG "pod": "ds-idrepo-0" 06:54:33 DEBUG { 06:54:33 DEBUG }, 06:54:33 DEBUG "metric": { 06:54:33 DEBUG "value": [ 06:54:33 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:54:33 DEBUG 06:54:33 DEBUG }, 06:54:33 DEBUG 06:54:33 DEBUG "value": [ 06:54:33 DEBUG 06:54:33 DEBUG 06:54:33 DEBUG [...] 06:54:33 DEBUG 06:54:33 DEBUG 06:54:33 DEBUG 06:54:33 DEBUG 06:54:33 DEBUG [...] 06:54:33 DEBUG 06:54:33 DEBUG 06:54:33 DEBUG "pod": "ds-idrepo-1" 06:54:33 DEBUG 06:54:33 DEBUG }, 06:54:33 DEBUG 06:54:33 DEBUG "value": [ 06:54:33 DEBUG "pod": "login-ui-5f964fb894-5zpvq" 06:54:33 DEBUG 1712728473, 06:54:33 DEBUG }, 06:54:33 DEBUG "0" 06:54:33 DEBUG "value": [ 06:54:33 DEBUG ] 06:54:33 DEBUG 1712728473, 06:54:33 DEBUG } 06:54:33 DEBUG "0.000034584468875812796" 06:54:33 DEBUG ] 06:54:33 DEBUG ] 06:54:33 DEBUG } 06:54:33 DEBUG } 06:54:33 DEBUG } 06:54:33 DEBUG ] 06:54:33 DEBUG } 06:54:33 DEBUG } 06:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728473" 06:54:33 INFO [http_cmd]: http status code OK 06:54:33 DEBUG --- status code --- 06:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:33 DEBUG --- http response --- 06:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:54:33 DEBUG ----- output ----- 06:54:33 DEBUG { 06:54:33 DEBUG "status": "success", 06:54:33 DEBUG "data": { 06:54:33 DEBUG "resultType": "vector", 06:54:33 DEBUG "result": [ 06:54:33 DEBUG { 06:54:33 DEBUG "metric": { 06:54:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:54:33 DEBUG }, 06:54:33 DEBUG "value": [ 06:54:33 DEBUG 06:54:33 DEBUG 06:54:33 DEBUG 06:54:33 DEBUG [...] 06:54:33 DEBUG 06:54:33 DEBUG 06:54:33 DEBUG 06:54:33 DEBUG "pod": "am-654c4fcbcf-487cv" 06:54:33 DEBUG }, 06:54:33 DEBUG "value": [ 06:54:33 DEBUG 1712728473, 06:54:33 DEBUG "0" 06:54:33 DEBUG ] 06:54:33 DEBUG } 06:54:33 DEBUG ] 06:54:33 DEBUG } 06:54:33 DEBUG } 06:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728473" 06:54:33 INFO [http_cmd]: http status code OK 06:54:33 DEBUG --- status code --- 06:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:33 DEBUG --- http response --- 06:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:54:33 DEBUG ----- output ----- 06:54:33 DEBUG { 06:54:33 DEBUG "status": "success", 06:54:33 DEBUG "data": { 06:54:33 DEBUG "resultType": "vector", 06:54:33 DEBUG "result": [ 06:54:33 DEBUG { 06:54:33 DEBUG "metric": { 06:54:33 DEBUG "pod": "ds-idrepo-0" 06:54:33 DEBUG }, 06:54:33 DEBUG "value": [ 06:54:33 DEBUG 06:54:33 DEBUG 06:54:33 DEBUG 06:54:33 DEBUG [...] 06:54:33 DEBUG 06:54:33 DEBUG 06:54:33 DEBUG 06:54:33 DEBUG "pod": "ds-idrepo-1" 06:54:33 DEBUG }, 06:54:33 DEBUG "value": [ 06:54:33 DEBUG 1712728473, 06:54:33 DEBUG "0" 06:54:33 DEBUG ] 06:54:33 DEBUG } 06:54:33 DEBUG ] 06:54:33 DEBUG } 06:54:33 DEBUG } 06:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728473" 06:54:33 INFO [http_cmd]: http status code OK 06:54:33 DEBUG --- status code --- 06:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:33 DEBUG --- http response --- 06:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:54:33 DEBUG ----- output ----- 06:54:33 DEBUG { 06:54:33 DEBUG "status": "success", 06:54:33 DEBUG "data": { 06:54:33 DEBUG "resultType": "vector", 06:54:33 DEBUG "result": [ 06:54:33 DEBUG { 06:54:33 DEBUG "metric": { 06:54:33 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:54:33 DEBUG }, 06:54:33 DEBUG "value": [ 06:54:33 DEBUG 06:54:33 DEBUG 06:54:33 DEBUG 06:54:33 DEBUG [...] 06:54:33 DEBUG 06:54:33 DEBUG 06:54:33 DEBUG 06:54:33 DEBUG "pod": "login-ui-5f964fb894-5zpvq" 06:54:33 DEBUG }, 06:54:33 DEBUG "value": [ 06:54:33 DEBUG 1712728473, 06:54:33 DEBUG "0" 06:54:33 DEBUG ] 06:54:33 DEBUG } 06:54:33 DEBUG ] 06:54:33 DEBUG } 06:54:33 DEBUG } 06:54:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728474" 06:54:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712728474" 06:54:34 INFO [http_cmd]: http status code OK 06:54:34 DEBUG --- status code --- 06:54:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:34 DEBUG --- http response --- 06:54:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:54:34 INFO [http_cmd]: http status code OK 06:54:34 DEBUG ----- output ----- 06:54:34 DEBUG --- status code --- 06:54:34 DEBUG { 06:54:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:34 DEBUG "status": "success", 06:54:34 DEBUG --- http response --- 06:54:34 DEBUG "data": { 06:54:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:54:34 DEBUG "resultType": "vector", 06:54:34 DEBUG ----- output ----- 06:54:34 DEBUG "result": [ 06:54:34 DEBUG { 06:54:34 DEBUG { 06:54:34 DEBUG "status": "success", 06:54:34 DEBUG "metric": { 06:54:34 DEBUG "data": { 06:54:34 DEBUG "instance": "10.142.0.25:9100" 06:54:34 DEBUG "resultType": "vector", 06:54:34 DEBUG }, 06:54:34 DEBUG "result": [ 06:54:34 DEBUG "value": [ 06:54:34 DEBUG { 06:54:34 DEBUG 06:54:34 DEBUG "metric": { 06:54:34 DEBUG 06:54:34 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:54:34 DEBUG 06:54:34 DEBUG }, 06:54:34 DEBUG [...] 06:54:34 DEBUG "value": [ 06:54:34 DEBUG 06:54:34 DEBUG 06:54:34 DEBUG 06:54:34 DEBUG 06:54:34 DEBUG 06:54:34 DEBUG 06:54:34 DEBUG "instance": "10.142.0.77:9100" 06:54:34 DEBUG [...] 06:54:34 DEBUG }, 06:54:34 DEBUG 06:54:34 DEBUG "value": [ 06:54:34 DEBUG 06:54:34 DEBUG 1712728474, 06:54:34 DEBUG 06:54:34 DEBUG "26119.066666666666" 06:54:34 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:54:34 DEBUG ] 06:54:34 DEBUG }, 06:54:34 DEBUG } 06:54:34 DEBUG "value": [ 06:54:34 DEBUG ] 06:54:34 DEBUG 1712728474, 06:54:34 DEBUG } 06:54:34 DEBUG "2487934976" 06:54:34 DEBUG } 06:54:34 DEBUG ] 06:54:34 DEBUG } 06:54:34 DEBUG ] 06:54:34 DEBUG } 06:54:34 DEBUG } 06:54:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728474" 06:54:34 INFO [http_cmd]: http status code OK 06:54:34 DEBUG --- status code --- 06:54:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:34 DEBUG --- http response --- 06:54:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:54:34 DEBUG ----- output ----- 06:54:34 DEBUG { 06:54:34 DEBUG "status": "success", 06:54:34 DEBUG "data": { 06:54:34 DEBUG "resultType": "vector", 06:54:34 DEBUG "result": [ 06:54:34 DEBUG { 06:54:34 DEBUG "metric": { 06:54:34 DEBUG "instance": "10.142.0.25:9100" 06:54:34 DEBUG }, 06:54:34 DEBUG "value": [ 06:54:34 DEBUG 06:54:34 DEBUG 06:54:34 DEBUG 06:54:34 DEBUG [...] 06:54:34 DEBUG 06:54:34 DEBUG 06:54:34 DEBUG 06:54:34 DEBUG "instance": "10.142.0.77:9100" 06:54:34 DEBUG }, 06:54:34 DEBUG "value": [ 06:54:34 DEBUG 1712728474, 06:54:34 DEBUG "0" 06:54:34 DEBUG ] 06:54:34 DEBUG } 06:54:34 DEBUG ] 06:54:34 DEBUG } 06:54:34 DEBUG } 06:54:54 INFO 06:54:54 INFO [loop_until]: kubectl --namespace=xlou top pods 06:54:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:54:55 INFO [loop_until]: OK (rc = 0) 06:54:55 DEBUG --- stdout --- 06:54:55 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 9Mi am-654c4fcbcf-487cv 7m 8870Mi am-654c4fcbcf-9scrj 7m 8378Mi am-654c4fcbcf-v5qb5 8m 8096Mi ds-cts-0 17m 12723Mi ds-cts-1 18m 12921Mi ds-cts-2 17m 11473Mi ds-idrepo-0 13m 5615Mi ds-idrepo-1 10m 5677Mi ds-idrepo-2 12m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 5Mi idm-6bfb7c96cc-qklk9 4m 1199Mi idm-6bfb7c96cc-vsb58 4m 2214Mi lodemon-b776b7d88-thrqf 6m 73Mi login-ui-5f964fb894-5zpvq 1m 4Mi overseer-0-6f8784665d-zq5hb 1007m 2930Mi 06:54:55 DEBUG --- stderr --- 06:54:55 DEBUG 06:55:03 INFO 06:55:03 INFO [loop_until]: kubectl --namespace=xlou top node 06:55:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:55:04 INFO [loop_until]: OK (rc = 0) 06:55:04 DEBUG --- stdout --- 06:55:04 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 97m 0% 2312Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 67m 0% 8991Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 76m 0% 3259Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 199m 1% 6514Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 70m 0% 9284Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 66m 0% 9733Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 76m 0% 860Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 69m 0% 6309Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 76m 0% 13459Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 71m 0% 12211Mi 20% gke-xlou-cdm-ds-32e4dcb1-gxzj 66m 0% 6402Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 69m 0% 13650Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 63m 0% 6368Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 1077m 6% 6609Mi 11% 06:55:04 DEBUG --- stderr --- 06:55:04 DEBUG 06:55:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712728530" 06:55:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728530" 06:55:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728530" 06:55:31 ERROR [http_cmd]: ERROR 06:55:31 INFO [http_cmd]: http status code OK 06:55:31 ERROR --- status code --- 06:55:31 DEBUG --- status code --- 06:55:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:55:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:31 ERROR --- http response --- 06:55:31 DEBUG --- http response --- 06:55:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:55:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:55:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:55:31 INFO [http_cmd]: http status code OK 06:55:31 DEBUG --- status code --- 06:55:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:31 DEBUG --- http response --- 06:55:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:55:31 DEBUG ----- output ----- 06:55:31 DEBUG { 06:55:31 DEBUG "status": "success", 06:55:31 DEBUG "data": { 06:55:31 DEBUG "resultType": "vector", 06:55:31 DEBUG "result": [ 06:55:31 DEBUG { 06:55:31 DEBUG "metric": { 06:55:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:55:31 DEBUG }, 06:55:31 DEBUG "value": [ 06:55:31 DEBUG 06:55:31 DEBUG 06:55:31 DEBUG 06:55:31 DEBUG [...] 06:55:31 DEBUG 06:55:31 DEBUG 06:55:31 DEBUG 06:55:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:55:31 DEBUG }, 06:55:31 DEBUG "value": [ 06:55:31 DEBUG 1712728530, 06:55:31 DEBUG "0" 06:55:31 DEBUG ] 06:55:31 DEBUG } 06:55:31 DEBUG ] 06:55:31 DEBUG } 06:55:31 DEBUG } 06:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728532" 06:55:32 INFO [http_cmd]: http status code OK 06:55:32 DEBUG --- status code --- 06:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:32 DEBUG --- http response --- 06:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:55:32 DEBUG ----- output ----- 06:55:32 DEBUG { 06:55:32 DEBUG "status": "success", 06:55:32 DEBUG "data": { 06:55:32 DEBUG "resultType": "vector", 06:55:32 DEBUG "result": [ 06:55:32 DEBUG { 06:55:32 DEBUG "metric": { 06:55:32 DEBUG "pod": "ds-idrepo-0" 06:55:32 DEBUG }, 06:55:32 DEBUG "value": [ 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG [...] 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG "pod": "ds-idrepo-1" 06:55:32 DEBUG }, 06:55:32 DEBUG "value": [ 06:55:32 DEBUG 1712728532, 06:55:32 DEBUG "0" 06:55:32 DEBUG ] 06:55:32 DEBUG } 06:55:32 DEBUG ] 06:55:32 DEBUG } 06:55:32 DEBUG } 06:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728532" 06:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728532" 06:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728532" 06:55:32 INFO [http_cmd]: http status code OK 06:55:32 DEBUG --- status code --- 06:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:32 DEBUG --- http response --- 06:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:55:32 INFO [http_cmd]: http status code OK 06:55:32 DEBUG ----- output ----- 06:55:32 DEBUG --- status code --- 06:55:32 DEBUG { 06:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:32 DEBUG "status": "success", 06:55:32 DEBUG --- http response --- 06:55:32 DEBUG "data": { 06:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:55:32 DEBUG "resultType": "vector", 06:55:32 DEBUG ----- output ----- 06:55:32 DEBUG "result": [ 06:55:32 DEBUG { 06:55:32 DEBUG { 06:55:32 DEBUG "status": "success", 06:55:32 DEBUG "metric": { 06:55:32 DEBUG "data": { 06:55:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:55:32 DEBUG "resultType": "vector", 06:55:32 DEBUG }, 06:55:32 DEBUG "result": [ 06:55:32 DEBUG "value": [ 06:55:32 DEBUG { 06:55:32 DEBUG 06:55:32 DEBUG "metric": { 06:55:32 DEBUG 06:55:32 DEBUG "instance": "10.142.0.25:9100" 06:55:32 DEBUG 06:55:32 DEBUG }, 06:55:32 DEBUG [...] 06:55:32 DEBUG "value": [ 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:55:32 DEBUG [...] 06:55:32 DEBUG }, 06:55:32 DEBUG 06:55:32 DEBUG "value": [ 06:55:32 DEBUG 06:55:32 DEBUG 1712728532, 06:55:32 DEBUG 06:55:32 DEBUG "0" 06:55:32 DEBUG "instance": "10.142.0.77:9100" 06:55:32 DEBUG ] 06:55:32 DEBUG }, 06:55:32 DEBUG } 06:55:32 DEBUG "value": [ 06:55:32 DEBUG ] 06:55:32 DEBUG 1712728532, 06:55:32 DEBUG } 06:55:32 DEBUG "25236" 06:55:32 DEBUG } 06:55:32 DEBUG ] 06:55:32 DEBUG } 06:55:32 DEBUG ] 06:55:32 DEBUG } 06:55:32 DEBUG } 06:55:32 INFO [http_cmd]: http status code OK 06:55:32 DEBUG --- status code --- 06:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:32 DEBUG --- http response --- 06:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:55:32 DEBUG ----- output ----- 06:55:32 DEBUG { 06:55:32 DEBUG "status": "success", 06:55:32 DEBUG "data": { 06:55:32 DEBUG "resultType": "vector", 06:55:32 DEBUG "result": [ 06:55:32 DEBUG { 06:55:32 DEBUG "metric": { 06:55:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:55:32 DEBUG }, 06:55:32 DEBUG "value": [ 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG [...] 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:55:32 DEBUG }, 06:55:32 DEBUG "value": [ 06:55:32 DEBUG 1712728532, 06:55:32 DEBUG "0" 06:55:32 DEBUG ] 06:55:32 DEBUG } 06:55:32 DEBUG ] 06:55:32 DEBUG } 06:55:32 DEBUG } 06:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728532" 06:55:32 INFO [http_cmd]: http status code OK 06:55:32 DEBUG --- status code --- 06:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:32 DEBUG --- http response --- 06:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:55:32 DEBUG ----- output ----- 06:55:32 DEBUG { 06:55:32 DEBUG "status": "success", 06:55:32 DEBUG "data": { 06:55:32 DEBUG "resultType": "vector", 06:55:32 DEBUG "result": [ 06:55:32 DEBUG { 06:55:32 DEBUG "metric": { 06:55:32 DEBUG "instance": "10.142.0.25:9100" 06:55:32 DEBUG }, 06:55:32 DEBUG "value": [ 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG [...] 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG "instance": "10.142.0.77:9100" 06:55:32 DEBUG }, 06:55:32 DEBUG "value": [ 06:55:32 DEBUG 1712728532, 06:55:32 DEBUG "438085686272" 06:55:32 DEBUG ] 06:55:32 DEBUG } 06:55:32 DEBUG ] 06:55:32 DEBUG } 06:55:32 DEBUG } 06:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712728532" 06:55:32 INFO [http_cmd]: http status code OK 06:55:32 DEBUG --- status code --- 06:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:32 DEBUG --- http response --- 06:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:55:32 DEBUG ----- output ----- 06:55:32 DEBUG { 06:55:32 DEBUG "status": "success", 06:55:32 DEBUG "data": { 06:55:32 DEBUG "resultType": "vector", 06:55:32 DEBUG "result": [ 06:55:32 DEBUG { 06:55:32 DEBUG "metric": { 06:55:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:55:32 DEBUG }, 06:55:32 DEBUG "value": [ 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG [...] 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:55:32 DEBUG }, 06:55:32 DEBUG "value": [ 06:55:32 DEBUG 1712728532, 06:55:32 DEBUG "0.4933622616254769" 06:55:32 DEBUG ] 06:55:32 DEBUG } 06:55:32 DEBUG ] 06:55:32 DEBUG } 06:55:32 DEBUG } 06:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728532" 06:55:32 INFO [http_cmd]: http status code OK 06:55:32 DEBUG --- status code --- 06:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:32 DEBUG --- http response --- 06:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:55:32 DEBUG ----- output ----- 06:55:32 DEBUG { 06:55:32 DEBUG "status": "success", 06:55:32 DEBUG "data": { 06:55:32 DEBUG "resultType": "vector", 06:55:32 DEBUG "result": [ 06:55:32 DEBUG { 06:55:32 DEBUG "metric": { 06:55:32 DEBUG "instance": "10.142.0.25:9100" 06:55:32 DEBUG }, 06:55:32 DEBUG "value": [ 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG [...] 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG "instance": "10.142.0.77:9100" 06:55:32 DEBUG }, 06:55:32 DEBUG "value": [ 06:55:32 DEBUG 1712728532, 06:55:32 DEBUG "0.0005333333333306693" 06:55:32 DEBUG ] 06:55:32 DEBUG } 06:55:32 DEBUG ] 06:55:32 DEBUG } 06:55:32 DEBUG } 06:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728532" 06:55:32 INFO [http_cmd]: http status code OK 06:55:32 DEBUG --- status code --- 06:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:32 DEBUG --- http response --- 06:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728532" 06:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:55:32 DEBUG ----- output ----- 06:55:32 DEBUG { 06:55:32 DEBUG "status": "success", 06:55:32 DEBUG "data": { 06:55:32 DEBUG "resultType": "vector", 06:55:32 DEBUG "result": [ 06:55:32 DEBUG { 06:55:32 DEBUG "metric": { 06:55:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:55:32 DEBUG }, 06:55:32 DEBUG "value": [ 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG [...] 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:55:32 DEBUG }, 06:55:32 DEBUG "value": [ 06:55:32 DEBUG 1712728532, 06:55:32 DEBUG "0" 06:55:32 DEBUG ] 06:55:32 DEBUG } 06:55:32 DEBUG ] 06:55:32 DEBUG } 06:55:32 DEBUG } 06:55:32 INFO [http_cmd]: http status code OK 06:55:32 DEBUG --- status code --- 06:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:32 DEBUG --- http response --- 06:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:55:32 DEBUG ----- output ----- 06:55:32 DEBUG { 06:55:32 DEBUG "status": "success", 06:55:32 DEBUG "data": { 06:55:32 DEBUG "resultType": "vector", 06:55:32 DEBUG "result": [ 06:55:32 DEBUG { 06:55:32 DEBUG "metric": { 06:55:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:55:32 DEBUG }, 06:55:32 DEBUG "value": [ 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG [...] 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:55:32 DEBUG }, 06:55:32 DEBUG "value": [ 06:55:32 DEBUG 1712728532, 06:55:32 DEBUG "0" 06:55:32 DEBUG ] 06:55:32 DEBUG } 06:55:32 DEBUG ] 06:55:32 DEBUG } 06:55:32 DEBUG } 06:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728532" 06:55:32 INFO [http_cmd]: http status code OK 06:55:32 DEBUG --- status code --- 06:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:32 DEBUG --- http response --- 06:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:55:32 DEBUG ----- output ----- 06:55:32 DEBUG { 06:55:32 DEBUG "status": "success", 06:55:32 DEBUG "data": { 06:55:32 DEBUG "resultType": "vector", 06:55:32 DEBUG "result": [ 06:55:32 DEBUG { 06:55:32 DEBUG "metric": { 06:55:32 DEBUG "pod": "ds-idrepo-0" 06:55:32 DEBUG }, 06:55:32 DEBUG "value": [ 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG [...] 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG "pod": "ds-cts-2" 06:55:32 DEBUG }, 06:55:32 DEBUG "value": [ 06:55:32 DEBUG 1712728532, 06:55:32 DEBUG "29.8" 06:55:32 DEBUG ] 06:55:32 DEBUG } 06:55:32 DEBUG ] 06:55:32 DEBUG } 06:55:32 DEBUG } 06:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728532" 06:55:32 INFO [http_cmd]: http status code OK 06:55:32 DEBUG --- status code --- 06:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:32 DEBUG --- http response --- 06:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:55:32 DEBUG ----- output ----- 06:55:32 DEBUG { 06:55:32 DEBUG "status": "success", 06:55:32 DEBUG "data": { 06:55:32 DEBUG "resultType": "vector", 06:55:32 DEBUG "result": [ 06:55:32 DEBUG { 06:55:32 DEBUG "metric": { 06:55:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:55:32 DEBUG }, 06:55:32 DEBUG "value": [ 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG [...] 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG "pod": "idm-6bfb7c96cc-vsb58" 06:55:32 DEBUG }, 06:55:32 DEBUG "value": [ 06:55:32 DEBUG 1712728532, 06:55:32 DEBUG "0" 06:55:32 DEBUG ] 06:55:32 DEBUG } 06:55:32 DEBUG ] 06:55:32 DEBUG } 06:55:32 DEBUG } 06:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712728532" 06:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728532" 06:55:32 INFO [http_cmd]: http status code OK 06:55:32 DEBUG --- status code --- 06:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:32 DEBUG --- http response --- 06:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:55:32 DEBUG ----- output ----- 06:55:32 DEBUG { 06:55:32 DEBUG "status": "success", 06:55:32 DEBUG "data": { 06:55:32 DEBUG "resultType": "vector", 06:55:32 DEBUG "result": [ 06:55:32 DEBUG { 06:55:32 DEBUG "metric": { 06:55:32 DEBUG "pod": "ds-idrepo-0" 06:55:32 DEBUG }, 06:55:32 DEBUG "value": [ 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG [...] 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG "pod": "ds-cts-2" 06:55:32 DEBUG }, 06:55:32 DEBUG "value": [ 06:55:32 DEBUG 1712728532, 06:55:32 DEBUG "0" 06:55:32 DEBUG ] 06:55:32 DEBUG } 06:55:32 DEBUG ] 06:55:32 DEBUG } 06:55:32 DEBUG } 06:55:32 INFO [http_cmd]: http status code OK 06:55:32 DEBUG --- status code --- 06:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:32 DEBUG --- http response --- 06:55:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:55:32 DEBUG ----- output ----- 06:55:32 DEBUG { 06:55:32 DEBUG "status": "success", 06:55:32 DEBUG "data": { 06:55:32 DEBUG "resultType": "vector", 06:55:32 DEBUG "result": [ 06:55:32 DEBUG { 06:55:32 DEBUG "metric": { 06:55:32 DEBUG "pod": "ds-idrepo-0" 06:55:32 DEBUG }, 06:55:32 DEBUG "value": [ 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG [...] 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG 06:55:32 DEBUG "pod": "ds-cts-2" 06:55:32 DEBUG }, 06:55:32 DEBUG "value": [ 06:55:32 DEBUG 1712728532, 06:55:32 DEBUG "0" 06:55:32 DEBUG ] 06:55:32 DEBUG } 06:55:32 DEBUG ] 06:55:32 DEBUG } 06:55:32 DEBUG } 06:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728533" 06:55:33 INFO [http_cmd]: http status code OK 06:55:33 DEBUG --- status code --- 06:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:33 DEBUG --- http response --- 06:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:55:33 DEBUG ----- output ----- 06:55:33 DEBUG { 06:55:33 DEBUG "status": "success", 06:55:33 DEBUG "data": { 06:55:33 DEBUG "resultType": "vector", 06:55:33 DEBUG "result": [ 06:55:33 DEBUG { 06:55:33 DEBUG "metric": { 06:55:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:55:33 DEBUG }, 06:55:33 DEBUG "value": [ 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG [...] 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG "pod": "am-654c4fcbcf-487cv" 06:55:33 DEBUG }, 06:55:33 DEBUG "value": [ 06:55:33 DEBUG 1712728533, 06:55:33 DEBUG "0" 06:55:33 DEBUG ] 06:55:33 DEBUG } 06:55:33 DEBUG ] 06:55:33 DEBUG } 06:55:33 DEBUG } 06:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712728533" 06:55:33 INFO [http_cmd]: http status code OK 06:55:33 DEBUG --- status code --- 06:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:33 DEBUG --- http response --- 06:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:55:33 DEBUG ----- output ----- 06:55:33 DEBUG { 06:55:33 DEBUG "status": "success", 06:55:33 DEBUG "data": { 06:55:33 DEBUG "resultType": "vector", 06:55:33 DEBUG "result": [ 06:55:33 DEBUG { 06:55:33 DEBUG "metric": { 06:55:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:55:33 DEBUG }, 06:55:33 DEBUG "value": [ 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG [...] 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG "pod": "am-654c4fcbcf-487cv" 06:55:33 DEBUG }, 06:55:33 DEBUG "value": [ 06:55:33 DEBUG 1712728533, 06:55:33 DEBUG "0.054" 06:55:33 DEBUG ] 06:55:33 DEBUG } 06:55:33 DEBUG ] 06:55:33 DEBUG } 06:55:33 DEBUG } 06:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728533" 06:55:33 INFO [http_cmd]: http status code OK 06:55:33 DEBUG --- status code --- 06:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:33 DEBUG --- http response --- 06:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:55:33 DEBUG ----- output ----- 06:55:33 DEBUG { 06:55:33 DEBUG "status": "success", 06:55:33 DEBUG "data": { 06:55:33 DEBUG "resultType": "vector", 06:55:33 DEBUG "result": [ 06:55:33 DEBUG { 06:55:33 DEBUG "metric": { 06:55:33 DEBUG "pod": "ds-idrepo-2" 06:55:33 DEBUG }, 06:55:33 DEBUG "value": [ 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG [...] 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG "pod": "ds-cts-2" 06:55:33 DEBUG }, 06:55:33 DEBUG "value": [ 06:55:33 DEBUG 1712728533, 06:55:33 DEBUG "0" 06:55:33 DEBUG ] 06:55:33 DEBUG } 06:55:33 DEBUG ] 06:55:33 DEBUG } 06:55:33 DEBUG } 06:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728533" 06:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728533" 06:55:33 INFO [http_cmd]: http status code OK 06:55:33 DEBUG --- status code --- 06:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:33 DEBUG --- http response --- 06:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:55:33 DEBUG ----- output ----- 06:55:33 DEBUG { 06:55:33 DEBUG "status": "success", 06:55:33 DEBUG "data": { 06:55:33 DEBUG "resultType": "vector", 06:55:33 DEBUG "result": [ 06:55:33 DEBUG { 06:55:33 DEBUG "metric": { 06:55:33 DEBUG "pod": "ds-idrepo-0" 06:55:33 DEBUG }, 06:55:33 DEBUG "value": [ 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG [...] 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG "pod": "ds-idrepo-1" 06:55:33 DEBUG }, 06:55:33 DEBUG "value": [ 06:55:33 DEBUG 1712728533, 06:55:33 DEBUG "0" 06:55:33 DEBUG ] 06:55:33 DEBUG } 06:55:33 DEBUG ] 06:55:33 DEBUG } 06:55:33 DEBUG } 06:55:33 INFO [http_cmd]: http status code OK 06:55:33 DEBUG --- status code --- 06:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:33 DEBUG --- http response --- 06:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:55:33 DEBUG ----- output ----- 06:55:33 DEBUG { 06:55:33 DEBUG "status": "success", 06:55:33 DEBUG "data": { 06:55:33 DEBUG "resultType": "vector", 06:55:33 DEBUG "result": [ 06:55:33 DEBUG { 06:55:33 DEBUG "metric": { 06:55:33 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:55:33 DEBUG }, 06:55:33 DEBUG "value": [ 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG [...] 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:55:33 DEBUG }, 06:55:33 DEBUG "value": [ 06:55:33 DEBUG 1712728533, 06:55:33 DEBUG "1.71448804092149" 06:55:33 DEBUG ] 06:55:33 DEBUG } 06:55:33 DEBUG ] 06:55:33 DEBUG } 06:55:33 DEBUG } 06:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728533" 06:55:33 INFO [http_cmd]: http status code OK 06:55:33 DEBUG --- status code --- 06:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:33 DEBUG --- http response --- 06:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:55:33 DEBUG ----- output ----- 06:55:33 DEBUG { 06:55:33 DEBUG "status": "success", 06:55:33 DEBUG "data": { 06:55:33 DEBUG "resultType": "vector", 06:55:33 DEBUG "result": [ 06:55:33 DEBUG { 06:55:33 DEBUG "metric": { 06:55:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:55:33 DEBUG }, 06:55:33 DEBUG "value": [ 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG [...] 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG "pod": "am-654c4fcbcf-487cv" 06:55:33 DEBUG }, 06:55:33 DEBUG "value": [ 06:55:33 DEBUG 1712728533, 06:55:33 DEBUG "0" 06:55:33 DEBUG ] 06:55:33 DEBUG } 06:55:33 DEBUG ] 06:55:33 DEBUG } 06:55:33 DEBUG } 06:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728533" 06:55:33 INFO [http_cmd]: http status code OK 06:55:33 DEBUG --- status code --- 06:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:33 DEBUG --- http response --- 06:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:55:33 DEBUG ----- output ----- 06:55:33 DEBUG { 06:55:33 DEBUG "status": "success", 06:55:33 DEBUG "data": { 06:55:33 DEBUG "resultType": "vector", 06:55:33 DEBUG "result": [ 06:55:33 DEBUG { 06:55:33 DEBUG "metric": { 06:55:33 DEBUG "pod": "ds-idrepo-0" 06:55:33 DEBUG }, 06:55:33 DEBUG "value": [ 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG [...] 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG "pod": "ds-idrepo-1" 06:55:33 DEBUG }, 06:55:33 DEBUG "value": [ 06:55:33 DEBUG 1712728533, 06:55:33 DEBUG "0" 06:55:33 DEBUG ] 06:55:33 DEBUG } 06:55:33 DEBUG ] 06:55:33 DEBUG } 06:55:33 DEBUG } 06:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728533" 06:55:33 INFO [http_cmd]: http status code OK 06:55:33 DEBUG --- status code --- 06:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:33 DEBUG --- http response --- 06:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:55:33 DEBUG ----- output ----- 06:55:33 DEBUG { 06:55:33 DEBUG "status": "success", 06:55:33 DEBUG "data": { 06:55:33 DEBUG "resultType": "vector", 06:55:33 DEBUG "result": [ 06:55:33 DEBUG { 06:55:33 DEBUG "metric": { 06:55:33 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:55:33 DEBUG }, 06:55:33 DEBUG "value": [ 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG [...] 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG 06:55:33 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:55:33 DEBUG }, 06:55:33 DEBUG "value": [ 06:55:33 DEBUG 1712728533, 06:55:33 DEBUG "4529.591214154973" 06:55:33 DEBUG ] 06:55:33 DEBUG } 06:55:33 DEBUG ] 06:55:33 DEBUG } 06:55:33 DEBUG } 06:55:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728534" 06:55:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712728534" 06:55:34 INFO [http_cmd]: http status code OK 06:55:34 INFO [http_cmd]: http status code OK 06:55:34 DEBUG --- status code --- 06:55:34 DEBUG --- status code --- 06:55:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:34 DEBUG --- http response --- 06:55:34 DEBUG --- http response --- 06:55:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:55:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:55:34 DEBUG ----- output ----- 06:55:34 DEBUG ----- output ----- 06:55:34 DEBUG { 06:55:34 DEBUG { 06:55:34 DEBUG "status": "success", 06:55:34 DEBUG "status": "success", 06:55:34 DEBUG "data": { 06:55:34 DEBUG "data": { 06:55:34 DEBUG "resultType": "vector", 06:55:34 DEBUG "resultType": "vector", 06:55:34 DEBUG "result": [ 06:55:34 DEBUG "result": [ 06:55:34 DEBUG { 06:55:34 DEBUG { 06:55:34 DEBUG "metric": { 06:55:34 DEBUG "metric": { 06:55:34 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:55:34 DEBUG "instance": "10.142.0.25:9100" 06:55:34 DEBUG }, 06:55:34 DEBUG }, 06:55:34 DEBUG "value": [ 06:55:34 DEBUG "value": [ 06:55:34 DEBUG 06:55:34 DEBUG 06:55:34 DEBUG 06:55:34 DEBUG 06:55:34 DEBUG 06:55:34 DEBUG 06:55:34 DEBUG [...] 06:55:34 DEBUG [...] 06:55:34 DEBUG 06:55:34 DEBUG 06:55:34 DEBUG 06:55:34 DEBUG 06:55:34 DEBUG 06:55:34 DEBUG 06:55:34 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:55:34 DEBUG "instance": "10.142.0.77:9100" 06:55:34 DEBUG }, 06:55:34 DEBUG }, 06:55:34 DEBUG "value": [ 06:55:34 DEBUG "value": [ 06:55:34 DEBUG 1712728534, 06:55:34 DEBUG 1712728534, 06:55:34 DEBUG "2487906304" 06:55:34 DEBUG "28404.666666666664" 06:55:34 DEBUG ] 06:55:34 DEBUG ] 06:55:34 DEBUG } 06:55:34 DEBUG } 06:55:34 DEBUG ] 06:55:34 DEBUG ] 06:55:34 DEBUG } 06:55:34 DEBUG } 06:55:34 DEBUG } 06:55:34 DEBUG } 06:55:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728534" 06:55:34 INFO [http_cmd]: http status code OK 06:55:34 DEBUG --- status code --- 06:55:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:34 DEBUG --- http response --- 06:55:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:55:34 DEBUG ----- output ----- 06:55:34 DEBUG { 06:55:34 DEBUG "status": "success", 06:55:34 DEBUG "data": { 06:55:34 DEBUG "resultType": "vector", 06:55:34 DEBUG "result": [ 06:55:34 DEBUG { 06:55:34 DEBUG "metric": { 06:55:34 DEBUG "instance": "10.142.0.25:9100" 06:55:34 DEBUG }, 06:55:34 DEBUG "value": [ 06:55:34 DEBUG 06:55:34 DEBUG 06:55:34 DEBUG 06:55:34 DEBUG [...] 06:55:34 DEBUG 06:55:34 DEBUG 06:55:34 DEBUG 06:55:34 DEBUG "instance": "10.142.0.77:9100" 06:55:34 DEBUG }, 06:55:34 DEBUG "value": [ 06:55:34 DEBUG 1712728534, 06:55:34 DEBUG "0" 06:55:34 DEBUG ] 06:55:34 DEBUG } 06:55:34 DEBUG ] 06:55:34 DEBUG } 06:55:34 DEBUG } 06:55:55 INFO 06:55:55 INFO [loop_until]: kubectl --namespace=xlou top pods 06:55:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:55:55 INFO [loop_until]: OK (rc = 0) 06:55:55 DEBUG --- stdout --- 06:55:55 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 9Mi am-654c4fcbcf-487cv 6m 8870Mi am-654c4fcbcf-9scrj 8m 8378Mi am-654c4fcbcf-v5qb5 9m 8096Mi ds-cts-0 17m 12723Mi ds-cts-1 17m 12921Mi ds-cts-2 16m 11473Mi ds-idrepo-0 14m 5620Mi ds-idrepo-1 10m 5677Mi ds-idrepo-2 13m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 5Mi idm-6bfb7c96cc-qklk9 4m 1198Mi idm-6bfb7c96cc-vsb58 4m 2214Mi lodemon-b776b7d88-thrqf 6m 73Mi login-ui-5f964fb894-5zpvq 1m 4Mi overseer-0-6f8784665d-zq5hb 1186m 3125Mi 06:55:55 DEBUG --- stderr --- 06:55:55 DEBUG 06:56:04 INFO 06:56:04 INFO [loop_until]: kubectl --namespace=xlou top node 06:56:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:56:04 INFO [loop_until]: OK (rc = 0) 06:56:04 DEBUG --- stdout --- 06:56:04 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 95m 0% 2317Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 66m 0% 8987Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 82m 0% 3257Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 208m 1% 6522Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 68m 0% 9283Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 67m 0% 9730Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 73m 0% 858Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 70m 0% 6315Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 74m 0% 13462Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 71m 0% 12213Mi 20% gke-xlou-cdm-ds-32e4dcb1-gxzj 64m 0% 6403Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 73m 0% 13654Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 68m 0% 6369Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 1174m 7% 6834Mi 11% 06:56:04 DEBUG --- stderr --- 06:56:04 DEBUG 06:56:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728591" 06:56:31 INFO [http_cmd]: http status code OK 06:56:31 DEBUG --- status code --- 06:56:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:31 DEBUG --- http response --- 06:56:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712728591" 06:56:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:56:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728591" 06:56:31 DEBUG ----- output ----- 06:56:31 DEBUG { 06:56:31 DEBUG "status": "success", 06:56:31 DEBUG "data": { 06:56:31 DEBUG "resultType": "vector", 06:56:31 DEBUG "result": [ 06:56:31 DEBUG { 06:56:31 DEBUG "metric": { 06:56:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:56:31 DEBUG }, 06:56:31 DEBUG "value": [ 06:56:31 DEBUG 06:56:31 DEBUG 06:56:31 DEBUG 06:56:31 DEBUG [...] 06:56:31 DEBUG 06:56:31 DEBUG 06:56:31 DEBUG 06:56:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:56:31 DEBUG }, 06:56:31 DEBUG "value": [ 06:56:31 DEBUG 1712728591, 06:56:31 DEBUG "0" 06:56:31 DEBUG ] 06:56:31 DEBUG } 06:56:31 DEBUG ] 06:56:31 DEBUG } 06:56:31 DEBUG } 06:56:31 ERROR [http_cmd]: ERROR 06:56:31 ERROR --- status code --- 06:56:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:56:31 ERROR --- http response --- 06:56:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:56:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:56:31 INFO [http_cmd]: http status code OK 06:56:31 DEBUG --- status code --- 06:56:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:31 DEBUG --- http response --- 06:56:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728592" 06:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728592" 06:56:32 INFO [http_cmd]: http status code OK 06:56:32 DEBUG --- status code --- 06:56:32 INFO [http_cmd]: http status code OK 06:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:32 DEBUG --- status code --- 06:56:32 DEBUG --- http response --- 06:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:56:32 DEBUG --- http response --- 06:56:32 DEBUG ----- output ----- 06:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:56:32 DEBUG { 06:56:32 DEBUG ----- output ----- 06:56:32 DEBUG "status": "success", 06:56:32 DEBUG { 06:56:32 DEBUG "data": { 06:56:32 DEBUG "status": "success", 06:56:32 DEBUG "resultType": "vector", 06:56:32 DEBUG "data": { 06:56:32 DEBUG "result": [ 06:56:32 DEBUG "resultType": "vector", 06:56:32 DEBUG { 06:56:32 DEBUG "result": [ 06:56:32 DEBUG "metric": { 06:56:32 DEBUG { 06:56:32 DEBUG "pod": "ds-idrepo-0" 06:56:32 DEBUG "metric": { 06:56:32 DEBUG }, 06:56:32 DEBUG "instance": "10.142.0.25:9100" 06:56:32 DEBUG "value": [ 06:56:32 DEBUG }, 06:56:32 DEBUG 06:56:32 DEBUG "value": [ 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG [...] 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG [...] 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG "pod": "ds-idrepo-1" 06:56:32 DEBUG 06:56:32 DEBUG }, 06:56:32 DEBUG "instance": "10.142.0.77:9100" 06:56:32 DEBUG "value": [ 06:56:32 DEBUG }, 06:56:32 DEBUG 1712728592, 06:56:32 DEBUG "value": [ 06:56:32 DEBUG "0" 06:56:32 DEBUG 1712728592, 06:56:32 DEBUG ] 06:56:32 DEBUG "25132.63333333333" 06:56:32 DEBUG } 06:56:32 DEBUG ] 06:56:32 DEBUG ] 06:56:32 DEBUG } 06:56:32 DEBUG } 06:56:32 DEBUG ] 06:56:32 DEBUG } 06:56:32 DEBUG } 06:56:32 DEBUG } 06:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728592" 06:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728592" 06:56:32 INFO [http_cmd]: http status code OK 06:56:32 DEBUG --- status code --- 06:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:32 DEBUG --- http response --- 06:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:56:32 DEBUG ----- output ----- 06:56:32 DEBUG { 06:56:32 DEBUG "status": "success", 06:56:32 DEBUG "data": { 06:56:32 DEBUG "resultType": "vector", 06:56:32 DEBUG "result": [ 06:56:32 DEBUG { 06:56:32 DEBUG "metric": { 06:56:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:56:32 DEBUG }, 06:56:32 DEBUG "value": [ 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG [...] 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:56:32 DEBUG }, 06:56:32 DEBUG "value": [ 06:56:32 DEBUG 1712728592, 06:56:32 DEBUG "0" 06:56:32 DEBUG ] 06:56:32 DEBUG } 06:56:32 DEBUG ] 06:56:32 DEBUG } 06:56:32 DEBUG } 06:56:32 INFO [http_cmd]: http status code OK 06:56:32 DEBUG --- status code --- 06:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:32 DEBUG --- http response --- 06:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:56:32 DEBUG ----- output ----- 06:56:32 DEBUG { 06:56:32 DEBUG "status": "success", 06:56:32 DEBUG "data": { 06:56:32 DEBUG "resultType": "vector", 06:56:32 DEBUG "result": [ 06:56:32 DEBUG { 06:56:32 DEBUG "metric": { 06:56:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:56:32 DEBUG }, 06:56:32 DEBUG "value": [ 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG [...] 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:56:32 DEBUG }, 06:56:32 DEBUG "value": [ 06:56:32 DEBUG 1712728592, 06:56:32 DEBUG "0" 06:56:32 DEBUG ] 06:56:32 DEBUG } 06:56:32 DEBUG ] 06:56:32 DEBUG } 06:56:32 DEBUG } 06:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728592" 06:56:32 INFO [http_cmd]: http status code OK 06:56:32 DEBUG --- status code --- 06:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:32 DEBUG --- http response --- 06:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:56:32 DEBUG ----- output ----- 06:56:32 DEBUG { 06:56:32 DEBUG "status": "success", 06:56:32 DEBUG "data": { 06:56:32 DEBUG "resultType": "vector", 06:56:32 DEBUG "result": [ 06:56:32 DEBUG { 06:56:32 DEBUG "metric": { 06:56:32 DEBUG "instance": "10.142.0.25:9100" 06:56:32 DEBUG }, 06:56:32 DEBUG "value": [ 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG [...] 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG "instance": "10.142.0.77:9100" 06:56:32 DEBUG }, 06:56:32 DEBUG "value": [ 06:56:32 DEBUG 1712728592, 06:56:32 DEBUG "438087371776" 06:56:32 DEBUG ] 06:56:32 DEBUG } 06:56:32 DEBUG ] 06:56:32 DEBUG } 06:56:32 DEBUG } 06:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712728592" 06:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728592" 06:56:32 INFO [http_cmd]: http status code OK 06:56:32 DEBUG --- status code --- 06:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:32 DEBUG --- http response --- 06:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:56:32 DEBUG ----- output ----- 06:56:32 DEBUG { 06:56:32 DEBUG "status": "success", 06:56:32 DEBUG "data": { 06:56:32 DEBUG "resultType": "vector", 06:56:32 DEBUG "result": [ 06:56:32 DEBUG { 06:56:32 DEBUG "metric": { 06:56:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:56:32 DEBUG }, 06:56:32 DEBUG "value": [ 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG [...] 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:56:32 DEBUG }, 06:56:32 DEBUG "value": [ 06:56:32 DEBUG 1712728592, 06:56:32 DEBUG "1.0288301869826004" 06:56:32 DEBUG ] 06:56:32 DEBUG } 06:56:32 DEBUG ] 06:56:32 DEBUG } 06:56:32 DEBUG } 06:56:32 INFO [http_cmd]: http status code OK 06:56:32 DEBUG --- status code --- 06:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:32 DEBUG --- http response --- 06:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:56:32 DEBUG ----- output ----- 06:56:32 DEBUG { 06:56:32 DEBUG "status": "success", 06:56:32 DEBUG "data": { 06:56:32 DEBUG "resultType": "vector", 06:56:32 DEBUG "result": [ 06:56:32 DEBUG { 06:56:32 DEBUG "metric": { 06:56:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:56:32 DEBUG }, 06:56:32 DEBUG "value": [ 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG [...] 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:56:32 DEBUG }, 06:56:32 DEBUG "value": [ 06:56:32 DEBUG 1712728592, 06:56:32 DEBUG "0" 06:56:32 DEBUG ] 06:56:32 DEBUG } 06:56:32 DEBUG ] 06:56:32 DEBUG } 06:56:32 DEBUG } 06:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728592" 06:56:32 INFO [http_cmd]: http status code OK 06:56:32 DEBUG --- status code --- 06:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:32 DEBUG --- http response --- 06:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:56:32 DEBUG ----- output ----- 06:56:32 DEBUG { 06:56:32 DEBUG "status": "success", 06:56:32 DEBUG "data": { 06:56:32 DEBUG "resultType": "vector", 06:56:32 DEBUG "result": [ 06:56:32 DEBUG { 06:56:32 DEBUG "metric": { 06:56:32 DEBUG "instance": "10.142.0.25:9100" 06:56:32 DEBUG }, 06:56:32 DEBUG "value": [ 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG [...] 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG "instance": "10.142.0.77:9100" 06:56:32 DEBUG }, 06:56:32 DEBUG "value": [ 06:56:32 DEBUG 1712728592, 06:56:32 DEBUG "0.0007666666666639324" 06:56:32 DEBUG ] 06:56:32 DEBUG } 06:56:32 DEBUG ] 06:56:32 DEBUG } 06:56:32 DEBUG } 06:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728592" 06:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728592" 06:56:32 INFO [http_cmd]: http status code OK 06:56:32 DEBUG --- status code --- 06:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:32 DEBUG --- http response --- 06:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:56:32 DEBUG ----- output ----- 06:56:32 DEBUG { 06:56:32 DEBUG "status": "success", 06:56:32 DEBUG "data": { 06:56:32 DEBUG "resultType": "vector", 06:56:32 DEBUG "result": [ 06:56:32 DEBUG { 06:56:32 DEBUG "metric": { 06:56:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:56:32 DEBUG }, 06:56:32 DEBUG "value": [ 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG [...] 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:56:32 DEBUG }, 06:56:32 DEBUG "value": [ 06:56:32 DEBUG 1712728592, 06:56:32 DEBUG "0" 06:56:32 DEBUG ] 06:56:32 DEBUG } 06:56:32 DEBUG ] 06:56:32 DEBUG } 06:56:32 DEBUG } 06:56:32 INFO [http_cmd]: http status code OK 06:56:32 DEBUG --- status code --- 06:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:32 DEBUG --- http response --- 06:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:56:32 DEBUG ----- output ----- 06:56:32 DEBUG { 06:56:32 DEBUG "status": "success", 06:56:32 DEBUG "data": { 06:56:32 DEBUG "resultType": "vector", 06:56:32 DEBUG "result": [ 06:56:32 DEBUG { 06:56:32 DEBUG "metric": { 06:56:32 DEBUG "pod": "ds-idrepo-0" 06:56:32 DEBUG }, 06:56:32 DEBUG "value": [ 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG [...] 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG "pod": "ds-cts-2" 06:56:32 DEBUG }, 06:56:32 DEBUG "value": [ 06:56:32 DEBUG 1712728592, 06:56:32 DEBUG "29.633333333333333" 06:56:32 DEBUG ] 06:56:32 DEBUG } 06:56:32 DEBUG ] 06:56:32 DEBUG } 06:56:32 DEBUG } 06:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728592" 06:56:32 INFO [http_cmd]: http status code OK 06:56:32 DEBUG --- status code --- 06:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:32 DEBUG --- http response --- 06:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:56:32 DEBUG ----- output ----- 06:56:32 DEBUG { 06:56:32 DEBUG "status": "success", 06:56:32 DEBUG "data": { 06:56:32 DEBUG "resultType": "vector", 06:56:32 DEBUG "result": [ 06:56:32 DEBUG { 06:56:32 DEBUG "metric": { 06:56:32 DEBUG "pod": "ds-idrepo-0" 06:56:32 DEBUG }, 06:56:32 DEBUG "value": [ 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG [...] 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG "pod": "ds-cts-2" 06:56:32 DEBUG }, 06:56:32 DEBUG "value": [ 06:56:32 DEBUG 1712728592, 06:56:32 DEBUG "0" 06:56:32 DEBUG ] 06:56:32 DEBUG } 06:56:32 DEBUG ] 06:56:32 DEBUG } 06:56:32 DEBUG } 06:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728592" 06:56:32 INFO [http_cmd]: http status code OK 06:56:32 DEBUG --- status code --- 06:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:32 DEBUG --- http response --- 06:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:56:32 DEBUG ----- output ----- 06:56:32 DEBUG { 06:56:32 DEBUG "status": "success", 06:56:32 DEBUG "data": { 06:56:32 DEBUG "resultType": "vector", 06:56:32 DEBUG "result": [ 06:56:32 DEBUG { 06:56:32 DEBUG "metric": { 06:56:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:56:32 DEBUG }, 06:56:32 DEBUG "value": [ 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG [...] 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:56:32 DEBUG }, 06:56:32 DEBUG "value": [ 06:56:32 DEBUG 1712728592, 06:56:32 DEBUG "23343931.701760445" 06:56:32 DEBUG ] 06:56:32 DEBUG } 06:56:32 DEBUG ] 06:56:32 DEBUG } 06:56:32 DEBUG } 06:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712728592" 06:56:32 INFO [http_cmd]: http status code OK 06:56:32 DEBUG --- status code --- 06:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:32 DEBUG --- http response --- 06:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:56:32 DEBUG ----- output ----- 06:56:32 DEBUG { 06:56:32 DEBUG "status": "success", 06:56:32 DEBUG "data": { 06:56:32 DEBUG "resultType": "vector", 06:56:32 DEBUG "result": [ 06:56:32 DEBUG { 06:56:32 DEBUG "metric": { 06:56:32 DEBUG "pod": "ds-idrepo-0" 06:56:32 DEBUG }, 06:56:32 DEBUG "value": [ 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG [...] 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG 06:56:32 DEBUG "pod": "ds-cts-2" 06:56:32 DEBUG }, 06:56:32 DEBUG "value": [ 06:56:32 DEBUG 1712728592, 06:56:32 DEBUG "0" 06:56:32 DEBUG ] 06:56:32 DEBUG } 06:56:32 DEBUG ] 06:56:32 DEBUG } 06:56:32 DEBUG } 06:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728593" 06:56:33 INFO [http_cmd]: http status code OK 06:56:33 DEBUG --- status code --- 06:56:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:33 DEBUG --- http response --- 06:56:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:56:33 DEBUG ----- output ----- 06:56:33 DEBUG { 06:56:33 DEBUG "status": "success", 06:56:33 DEBUG "data": { 06:56:33 DEBUG "resultType": "vector", 06:56:33 DEBUG "result": [ 06:56:33 DEBUG { 06:56:33 DEBUG "metric": { 06:56:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:56:33 DEBUG }, 06:56:33 DEBUG "value": [ 06:56:33 DEBUG 06:56:33 DEBUG 06:56:33 DEBUG 06:56:33 DEBUG [...] 06:56:33 DEBUG 06:56:33 DEBUG 06:56:33 DEBUG 06:56:33 DEBUG "pod": "am-654c4fcbcf-487cv" 06:56:33 DEBUG }, 06:56:33 DEBUG "value": [ 06:56:33 DEBUG 1712728593, 06:56:33 DEBUG "0" 06:56:33 DEBUG ] 06:56:33 DEBUG } 06:56:33 DEBUG ] 06:56:33 DEBUG } 06:56:33 DEBUG } 06:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712728593" 06:56:33 INFO [http_cmd]: http status code OK 06:56:33 DEBUG --- status code --- 06:56:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:33 DEBUG --- http response --- 06:56:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:56:33 DEBUG ----- output ----- 06:56:33 DEBUG { 06:56:33 DEBUG "status": "success", 06:56:33 DEBUG "data": { 06:56:33 DEBUG "resultType": "vector", 06:56:33 DEBUG "result": [ 06:56:33 DEBUG { 06:56:33 DEBUG "metric": { 06:56:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:56:33 DEBUG }, 06:56:33 DEBUG "value": [ 06:56:33 DEBUG 06:56:33 DEBUG 06:56:33 DEBUG 06:56:33 DEBUG [...] 06:56:33 DEBUG 06:56:33 DEBUG 06:56:33 DEBUG 06:56:33 DEBUG "pod": "am-654c4fcbcf-487cv" 06:56:33 DEBUG }, 06:56:33 DEBUG "value": [ 06:56:33 DEBUG 1712728593, 06:56:33 DEBUG "0.054" 06:56:33 DEBUG ] 06:56:33 DEBUG } 06:56:33 DEBUG ] 06:56:33 DEBUG } 06:56:33 DEBUG } 06:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728593" 06:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728593" 06:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728593" 06:56:33 INFO [http_cmd]: http status code OK 06:56:33 INFO [http_cmd]: http status code OK 06:56:33 DEBUG --- status code --- 06:56:33 DEBUG --- status code --- 06:56:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:33 DEBUG --- http response --- 06:56:33 DEBUG --- http response --- 06:56:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:56:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:56:33 DEBUG ----- output ----- 06:56:33 DEBUG ----- output ----- 06:56:33 DEBUG { 06:56:33 DEBUG { 06:56:33 DEBUG "status": "success", 06:56:33 DEBUG "status": "success", 06:56:33 DEBUG "data": { 06:56:33 DEBUG "data": { 06:56:33 DEBUG "resultType": "vector", 06:56:33 DEBUG "resultType": "vector", 06:56:33 DEBUG "result": [ 06:56:33 DEBUG "result": [ 06:56:33 DEBUG { 06:56:33 DEBUG { 06:56:33 DEBUG "metric": { 06:56:33 DEBUG "metric": { 06:56:33 DEBUG "pod": "ds-idrepo-2" 06:56:33 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:56:33 DEBUG }, 06:56:33 DEBUG }, 06:56:33 DEBUG "value": [ 06:56:33 DEBUG "value": [ 06:56:33 DEBUG 06:56:33 DEBUG 06:56:33 DEBUG 06:56:33 DEBUG 06:56:33 DEBUG 06:56:33 DEBUG 06:56:33 DEBUG [...] 06:56:33 DEBUG [...] 06:56:33 DEBUG 06:56:33 DEBUG 06:56:33 DEBUG 06:56:33 DEBUG 06:56:33 DEBUG 06:56:33 DEBUG 06:56:33 DEBUG "pod": "ds-cts-2" 06:56:33 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:56:33 DEBUG }, 06:56:33 DEBUG }, 06:56:33 DEBUG "value": [ 06:56:33 DEBUG "value": [ 06:56:33 DEBUG 1712728593, 06:56:33 DEBUG 1712728593, 06:56:33 DEBUG "0" 06:56:33 DEBUG "2.00924777908312" 06:56:33 DEBUG ] 06:56:33 DEBUG ] 06:56:33 DEBUG } 06:56:33 DEBUG } 06:56:33 DEBUG ] 06:56:33 DEBUG ] 06:56:33 DEBUG } 06:56:33 DEBUG } 06:56:33 DEBUG } 06:56:33 DEBUG } 06:56:33 INFO [http_cmd]: http status code OK 06:56:33 DEBUG --- status code --- 06:56:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:33 DEBUG --- http response --- 06:56:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:56:33 DEBUG ----- output ----- 06:56:33 DEBUG { 06:56:33 DEBUG "status": "success", 06:56:33 DEBUG "data": { 06:56:33 DEBUG "resultType": "vector", 06:56:33 DEBUG "result": [ 06:56:33 DEBUG { 06:56:33 DEBUG "metric": { 06:56:33 DEBUG "pod": "ds-idrepo-0" 06:56:33 DEBUG }, 06:56:33 DEBUG "value": [ 06:56:33 DEBUG 06:56:33 DEBUG 06:56:33 DEBUG 06:56:33 DEBUG [...] 06:56:33 DEBUG 06:56:33 DEBUG 06:56:33 DEBUG 06:56:33 DEBUG "pod": "ds-idrepo-1" 06:56:33 DEBUG }, 06:56:33 DEBUG "value": [ 06:56:33 DEBUG 1712728593, 06:56:33 DEBUG "0" 06:56:33 DEBUG ] 06:56:33 DEBUG } 06:56:33 DEBUG ] 06:56:33 DEBUG } 06:56:33 DEBUG } 06:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728593" 06:56:33 INFO [http_cmd]: http status code OK 06:56:33 DEBUG --- status code --- 06:56:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:33 DEBUG --- http response --- 06:56:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:56:33 DEBUG ----- output ----- 06:56:33 DEBUG { 06:56:33 DEBUG "status": "success", 06:56:33 DEBUG "data": { 06:56:33 DEBUG "resultType": "vector", 06:56:33 DEBUG "result": [ 06:56:33 DEBUG { 06:56:33 DEBUG "metric": { 06:56:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:56:33 DEBUG }, 06:56:33 DEBUG "value": [ 06:56:33 DEBUG 06:56:33 DEBUG 06:56:33 DEBUG 06:56:33 DEBUG [...] 06:56:33 DEBUG 06:56:33 DEBUG 06:56:33 DEBUG 06:56:33 DEBUG "pod": "am-654c4fcbcf-487cv" 06:56:33 DEBUG }, 06:56:33 DEBUG "value": [ 06:56:33 DEBUG 1712728593, 06:56:33 DEBUG "0" 06:56:33 DEBUG ] 06:56:33 DEBUG } 06:56:33 DEBUG ] 06:56:33 DEBUG } 06:56:33 DEBUG } 06:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728593" 06:56:33 INFO [http_cmd]: http status code OK 06:56:33 DEBUG --- status code --- 06:56:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:33 DEBUG --- http response --- 06:56:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:56:33 DEBUG ----- output ----- 06:56:33 DEBUG { 06:56:33 DEBUG "status": "success", 06:56:33 DEBUG "data": { 06:56:33 DEBUG "resultType": "vector", 06:56:33 DEBUG "result": [ 06:56:33 DEBUG { 06:56:33 DEBUG "metric": { 06:56:33 DEBUG "pod": "ds-idrepo-0" 06:56:33 DEBUG }, 06:56:33 DEBUG "value": [ 06:56:33 DEBUG 06:56:33 DEBUG 06:56:33 DEBUG 06:56:33 DEBUG [...] 06:56:33 DEBUG 06:56:33 DEBUG 06:56:33 DEBUG 06:56:33 DEBUG "pod": "ds-idrepo-1" 06:56:33 DEBUG }, 06:56:33 DEBUG "value": [ 06:56:33 DEBUG 1712728593, 06:56:33 DEBUG "0" 06:56:33 DEBUG ] 06:56:33 DEBUG } 06:56:33 DEBUG ] 06:56:33 DEBUG } 06:56:33 DEBUG } 06:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728593" 06:56:34 INFO [http_cmd]: http status code OK 06:56:34 DEBUG --- status code --- 06:56:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:34 DEBUG --- http response --- 06:56:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:56:34 DEBUG ----- output ----- 06:56:34 DEBUG { 06:56:34 DEBUG "status": "success", 06:56:34 DEBUG "data": { 06:56:34 DEBUG "resultType": "vector", 06:56:34 DEBUG "result": [ 06:56:34 DEBUG { 06:56:34 DEBUG "metric": { 06:56:34 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:56:34 DEBUG }, 06:56:34 DEBUG "value": [ 06:56:34 DEBUG 06:56:34 DEBUG 06:56:34 DEBUG 06:56:34 DEBUG [...] 06:56:34 DEBUG 06:56:34 DEBUG 06:56:34 DEBUG 06:56:34 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:56:34 DEBUG }, 06:56:34 DEBUG "value": [ 06:56:34 DEBUG 1712728593, 06:56:34 DEBUG "0" 06:56:34 DEBUG ] 06:56:34 DEBUG } 06:56:34 DEBUG ] 06:56:34 DEBUG } 06:56:34 DEBUG } 06:56:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728594" 06:56:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712728594" 06:56:34 INFO [http_cmd]: http status code OK 06:56:34 DEBUG --- status code --- 06:56:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:34 DEBUG --- http response --- 06:56:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:56:34 INFO [http_cmd]: http status code OK 06:56:34 DEBUG ----- output ----- 06:56:34 DEBUG --- status code --- 06:56:34 DEBUG { 06:56:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:34 DEBUG "status": "success", 06:56:34 DEBUG --- http response --- 06:56:34 DEBUG "data": { 06:56:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:56:34 DEBUG "resultType": "vector", 06:56:34 DEBUG ----- output ----- 06:56:34 DEBUG "result": [ 06:56:34 DEBUG { 06:56:34 DEBUG { 06:56:34 DEBUG "status": "success", 06:56:34 DEBUG "metric": { 06:56:34 DEBUG "data": { 06:56:34 DEBUG "instance": "10.142.0.25:9100" 06:56:34 DEBUG "resultType": "vector", 06:56:34 DEBUG }, 06:56:34 DEBUG "result": [ 06:56:34 DEBUG "value": [ 06:56:34 DEBUG { 06:56:34 DEBUG 06:56:34 DEBUG "metric": { 06:56:34 DEBUG 06:56:34 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:56:34 DEBUG 06:56:34 DEBUG }, 06:56:34 DEBUG [...] 06:56:34 DEBUG "value": [ 06:56:34 DEBUG 06:56:34 DEBUG 06:56:34 DEBUG 06:56:34 DEBUG 06:56:34 DEBUG 06:56:34 DEBUG 06:56:34 DEBUG "instance": "10.142.0.77:9100" 06:56:34 DEBUG [...] 06:56:34 DEBUG }, 06:56:34 DEBUG 06:56:34 DEBUG "value": [ 06:56:34 DEBUG 06:56:34 DEBUG 1712728594, 06:56:34 DEBUG 06:56:34 DEBUG "25943.033333333333" 06:56:34 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:56:34 DEBUG ] 06:56:34 DEBUG }, 06:56:34 DEBUG } 06:56:34 DEBUG "value": [ 06:56:34 DEBUG ] 06:56:34 DEBUG 1712728594, 06:56:34 DEBUG } 06:56:34 DEBUG "2490195968" 06:56:34 DEBUG } 06:56:34 DEBUG ] 06:56:34 DEBUG } 06:56:34 DEBUG ] 06:56:34 DEBUG } 06:56:34 DEBUG } 06:56:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728594" 06:56:34 INFO [http_cmd]: http status code OK 06:56:34 DEBUG --- status code --- 06:56:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:34 DEBUG --- http response --- 06:56:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:56:34 DEBUG ----- output ----- 06:56:34 DEBUG { 06:56:34 DEBUG "status": "success", 06:56:34 DEBUG "data": { 06:56:34 DEBUG "resultType": "vector", 06:56:34 DEBUG "result": [ 06:56:34 DEBUG { 06:56:34 DEBUG "metric": { 06:56:34 DEBUG "instance": "10.142.0.25:9100" 06:56:34 DEBUG }, 06:56:34 DEBUG "value": [ 06:56:34 DEBUG 06:56:34 DEBUG 06:56:34 DEBUG 06:56:34 DEBUG [...] 06:56:34 DEBUG 06:56:34 DEBUG 06:56:34 DEBUG 06:56:34 DEBUG "instance": "10.142.0.77:9100" 06:56:34 DEBUG }, 06:56:34 DEBUG "value": [ 06:56:34 DEBUG 1712728594, 06:56:34 DEBUG "0" 06:56:34 DEBUG ] 06:56:34 DEBUG } 06:56:34 DEBUG ] 06:56:34 DEBUG } 06:56:34 DEBUG } 06:56:55 INFO 06:56:55 INFO [loop_until]: kubectl --namespace=xlou top pods 06:56:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:56:55 INFO [loop_until]: OK (rc = 0) 06:56:55 DEBUG --- stdout --- 06:56:55 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 9Mi am-654c4fcbcf-487cv 7m 8870Mi am-654c4fcbcf-9scrj 8m 8378Mi am-654c4fcbcf-v5qb5 8m 8096Mi ds-cts-0 16m 12723Mi ds-cts-1 17m 12921Mi ds-cts-2 20m 11474Mi ds-idrepo-0 15m 5620Mi ds-idrepo-1 11m 5677Mi ds-idrepo-2 12m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 5Mi idm-6bfb7c96cc-qklk9 4m 1198Mi idm-6bfb7c96cc-vsb58 5m 2213Mi lodemon-b776b7d88-thrqf 4m 73Mi login-ui-5f964fb894-5zpvq 1m 4Mi overseer-0-6f8784665d-zq5hb 988m 3928Mi 06:56:55 DEBUG --- stderr --- 06:56:55 DEBUG 06:57:04 INFO 06:57:04 INFO [loop_until]: kubectl --namespace=xlou top node 06:57:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:57:04 INFO [loop_until]: OK (rc = 0) 06:57:04 DEBUG --- stdout --- 06:57:04 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 97m 0% 2315Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 66m 0% 8990Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 75m 0% 3255Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 195m 1% 6528Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 65m 0% 9283Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 66m 0% 9732Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 76m 0% 861Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 67m 0% 6317Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 69m 0% 13462Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 102m 0% 12210Mi 20% gke-xlou-cdm-ds-32e4dcb1-gxzj 73m 0% 6403Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 70m 0% 13654Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 66m 0% 6371Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 1074m 6% 7511Mi 12% 06:57:04 DEBUG --- stderr --- 06:57:04 DEBUG 06:57:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728651" 06:57:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712728651" 06:57:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728651" 06:57:31 INFO [http_cmd]: http status code OK 06:57:31 DEBUG --- status code --- 06:57:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:31 DEBUG --- http response --- 06:57:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:57:31 DEBUG ----- output ----- 06:57:31 DEBUG { 06:57:31 DEBUG "status": "success", 06:57:31 DEBUG "data": { 06:57:31 DEBUG "resultType": "vector", 06:57:31 DEBUG "result": [ 06:57:31 DEBUG { 06:57:31 DEBUG "metric": { 06:57:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:57:31 DEBUG }, 06:57:31 DEBUG "value": [ 06:57:31 DEBUG 06:57:31 DEBUG 06:57:31 DEBUG 06:57:31 DEBUG [...] 06:57:31 DEBUG 06:57:31 DEBUG 06:57:31 DEBUG 06:57:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:57:31 DEBUG }, 06:57:31 DEBUG "value": [ 06:57:31 DEBUG 1712728651, 06:57:31 DEBUG "0" 06:57:31 DEBUG ] 06:57:31 DEBUG } 06:57:31 DEBUG ] 06:57:31 DEBUG } 06:57:31 DEBUG } 06:57:31 ERROR [http_cmd]: ERROR 06:57:31 ERROR --- status code --- 06:57:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:57:31 ERROR --- http response --- 06:57:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:57:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:57:31 INFO [http_cmd]: http status code OK 06:57:31 DEBUG --- status code --- 06:57:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:31 DEBUG --- http response --- 06:57:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728652" 06:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728652" 06:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728652" 06:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728652" 06:57:32 INFO [http_cmd]: http status code OK 06:57:32 INFO [http_cmd]: http status code OK 06:57:32 INFO [http_cmd]: http status code OK 06:57:32 DEBUG --- status code --- 06:57:32 DEBUG --- status code --- 06:57:32 DEBUG --- status code --- 06:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:32 DEBUG --- http response --- 06:57:32 DEBUG --- http response --- 06:57:32 DEBUG --- http response --- 06:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:57:32 DEBUG ----- output ----- 06:57:32 DEBUG ----- output ----- 06:57:32 DEBUG ----- output ----- 06:57:32 DEBUG { 06:57:32 DEBUG { 06:57:32 DEBUG { 06:57:32 DEBUG "status": "success", 06:57:32 DEBUG "status": "success", 06:57:32 DEBUG "status": "success", 06:57:32 DEBUG "data": { 06:57:32 DEBUG "data": { 06:57:32 DEBUG "data": { 06:57:32 DEBUG "resultType": "vector", 06:57:32 DEBUG "resultType": "vector", 06:57:32 DEBUG "resultType": "vector", 06:57:32 DEBUG "result": [ 06:57:32 DEBUG "result": [ 06:57:32 DEBUG "result": [ 06:57:32 DEBUG { 06:57:32 DEBUG { 06:57:32 DEBUG { 06:57:32 DEBUG "metric": { 06:57:32 DEBUG "metric": { 06:57:32 DEBUG "metric": { 06:57:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:57:32 DEBUG "pod": "ds-idrepo-0" 06:57:32 DEBUG "instance": "10.142.0.25:9100" 06:57:32 DEBUG }, 06:57:32 DEBUG }, 06:57:32 DEBUG }, 06:57:32 DEBUG "value": [ 06:57:32 DEBUG "value": [ 06:57:32 DEBUG "value": [ 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG [...] 06:57:32 DEBUG [...] 06:57:32 DEBUG [...] 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:57:32 DEBUG "pod": "ds-idrepo-1" 06:57:32 DEBUG "instance": "10.142.0.77:9100" 06:57:32 DEBUG }, 06:57:32 DEBUG }, 06:57:32 DEBUG }, 06:57:32 DEBUG "value": [ 06:57:32 DEBUG "value": [ 06:57:32 INFO [http_cmd]: http status code OK 06:57:32 DEBUG "value": [ 06:57:32 DEBUG 1712728652, 06:57:32 DEBUG 1712728652, 06:57:32 DEBUG --- status code --- 06:57:32 DEBUG 1712728652, 06:57:32 DEBUG "0" 06:57:32 DEBUG "0" 06:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:32 DEBUG "25431.166666666664" 06:57:32 DEBUG ] 06:57:32 DEBUG ] 06:57:32 DEBUG --- http response --- 06:57:32 DEBUG ] 06:57:32 DEBUG } 06:57:32 DEBUG } 06:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:57:32 DEBUG } 06:57:32 DEBUG ] 06:57:32 DEBUG ] 06:57:32 DEBUG ----- output ----- 06:57:32 DEBUG ] 06:57:32 DEBUG } 06:57:32 DEBUG } 06:57:32 DEBUG { 06:57:32 DEBUG } 06:57:32 DEBUG } 06:57:32 DEBUG } 06:57:32 DEBUG "status": "success", 06:57:32 DEBUG } 06:57:32 DEBUG "data": { 06:57:32 DEBUG "resultType": "vector", 06:57:32 DEBUG "result": [ 06:57:32 DEBUG { 06:57:32 DEBUG "metric": { 06:57:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:57:32 DEBUG }, 06:57:32 DEBUG "value": [ 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG [...] 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:57:32 DEBUG }, 06:57:32 DEBUG "value": [ 06:57:32 DEBUG 1712728652, 06:57:32 DEBUG "0" 06:57:32 DEBUG ] 06:57:32 DEBUG } 06:57:32 DEBUG ] 06:57:32 DEBUG } 06:57:32 DEBUG } 06:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728652" 06:57:32 INFO [http_cmd]: http status code OK 06:57:32 DEBUG --- status code --- 06:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:32 DEBUG --- http response --- 06:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:57:32 DEBUG ----- output ----- 06:57:32 DEBUG { 06:57:32 DEBUG "status": "success", 06:57:32 DEBUG "data": { 06:57:32 DEBUG "resultType": "vector", 06:57:32 DEBUG "result": [ 06:57:32 DEBUG { 06:57:32 DEBUG "metric": { 06:57:32 DEBUG "instance": "10.142.0.25:9100" 06:57:32 DEBUG }, 06:57:32 DEBUG "value": [ 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG [...] 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG "instance": "10.142.0.77:9100" 06:57:32 DEBUG }, 06:57:32 DEBUG "value": [ 06:57:32 DEBUG 1712728652, 06:57:32 DEBUG "438088764416" 06:57:32 DEBUG ] 06:57:32 DEBUG } 06:57:32 DEBUG ] 06:57:32 DEBUG } 06:57:32 DEBUG } 06:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712728652" 06:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728652" 06:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728652" 06:57:32 INFO [http_cmd]: http status code OK 06:57:32 DEBUG --- status code --- 06:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:32 DEBUG --- http response --- 06:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:57:32 DEBUG ----- output ----- 06:57:32 DEBUG { 06:57:32 DEBUG "status": "success", 06:57:32 DEBUG "data": { 06:57:32 DEBUG "resultType": "vector", 06:57:32 DEBUG "result": [ 06:57:32 DEBUG { 06:57:32 DEBUG "metric": { 06:57:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:57:32 DEBUG }, 06:57:32 DEBUG "value": [ 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG [...] 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:57:32 DEBUG }, 06:57:32 DEBUG "value": [ 06:57:32 DEBUG 1712728652, 06:57:32 DEBUG "1.0324742402401845" 06:57:32 DEBUG ] 06:57:32 DEBUG } 06:57:32 DEBUG ] 06:57:32 DEBUG } 06:57:32 DEBUG } 06:57:32 INFO [http_cmd]: http status code OK 06:57:32 INFO [http_cmd]: http status code OK 06:57:32 DEBUG --- status code --- 06:57:32 DEBUG --- status code --- 06:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:32 DEBUG --- http response --- 06:57:32 DEBUG --- http response --- 06:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:57:32 DEBUG ----- output ----- 06:57:32 DEBUG ----- output ----- 06:57:32 DEBUG { 06:57:32 DEBUG { 06:57:32 DEBUG "status": "success", 06:57:32 DEBUG "status": "success", 06:57:32 DEBUG "data": { 06:57:32 DEBUG "data": { 06:57:32 DEBUG "resultType": "vector", 06:57:32 DEBUG "resultType": "vector", 06:57:32 DEBUG "result": [ 06:57:32 DEBUG "result": [ 06:57:32 DEBUG { 06:57:32 DEBUG { 06:57:32 DEBUG "metric": { 06:57:32 DEBUG "metric": { 06:57:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:57:32 DEBUG "instance": "10.142.0.25:9100" 06:57:32 DEBUG }, 06:57:32 DEBUG }, 06:57:32 DEBUG "value": [ 06:57:32 DEBUG "value": [ 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG [...] 06:57:32 DEBUG [...] 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG "pod": "login-ui-5f964fb894-5zpvq" 06:57:32 DEBUG "instance": "10.142.0.77:9100" 06:57:32 DEBUG }, 06:57:32 DEBUG }, 06:57:32 DEBUG "value": [ 06:57:32 DEBUG "value": [ 06:57:32 DEBUG 1712728652, 06:57:32 DEBUG 1712728652, 06:57:32 DEBUG "0" 06:57:32 DEBUG "0.0008999999999964814" 06:57:32 DEBUG ] 06:57:32 DEBUG ] 06:57:32 DEBUG } 06:57:32 DEBUG } 06:57:32 DEBUG ] 06:57:32 DEBUG ] 06:57:32 DEBUG } 06:57:32 DEBUG } 06:57:32 DEBUG } 06:57:32 DEBUG } 06:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728652" 06:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728652" 06:57:32 INFO [http_cmd]: http status code OK 06:57:32 DEBUG --- status code --- 06:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:32 DEBUG --- http response --- 06:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:57:32 DEBUG ----- output ----- 06:57:32 DEBUG { 06:57:32 DEBUG "status": "success", 06:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728652" 06:57:32 DEBUG "data": { 06:57:32 DEBUG "resultType": "vector", 06:57:32 DEBUG "result": [ 06:57:32 DEBUG { 06:57:32 DEBUG "metric": { 06:57:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:57:32 DEBUG }, 06:57:32 DEBUG "value": [ 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG [...] 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:57:32 DEBUG }, 06:57:32 DEBUG "value": [ 06:57:32 DEBUG 1712728652, 06:57:32 DEBUG "0" 06:57:32 DEBUG ] 06:57:32 DEBUG } 06:57:32 DEBUG ] 06:57:32 DEBUG } 06:57:32 DEBUG } 06:57:32 INFO [http_cmd]: http status code OK 06:57:32 DEBUG --- status code --- 06:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:32 DEBUG --- http response --- 06:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:57:32 DEBUG ----- output ----- 06:57:32 DEBUG { 06:57:32 DEBUG "status": "success", 06:57:32 DEBUG "data": { 06:57:32 DEBUG "resultType": "vector", 06:57:32 DEBUG "result": [ 06:57:32 DEBUG { 06:57:32 DEBUG "metric": { 06:57:32 DEBUG "pod": "ds-idrepo-0" 06:57:32 DEBUG }, 06:57:32 DEBUG "value": [ 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG [...] 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG "pod": "ds-cts-2" 06:57:32 DEBUG }, 06:57:32 DEBUG "value": [ 06:57:32 DEBUG 1712728652, 06:57:32 DEBUG "0" 06:57:32 DEBUG ] 06:57:32 DEBUG } 06:57:32 DEBUG ] 06:57:32 DEBUG } 06:57:32 DEBUG } 06:57:32 INFO [http_cmd]: http status code OK 06:57:32 DEBUG --- status code --- 06:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:32 DEBUG --- http response --- 06:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:57:32 DEBUG ----- output ----- 06:57:32 DEBUG { 06:57:32 DEBUG "status": "success", 06:57:32 DEBUG "data": { 06:57:32 DEBUG "resultType": "vector", 06:57:32 DEBUG "result": [ 06:57:32 DEBUG { 06:57:32 DEBUG "metric": { 06:57:32 DEBUG "pod": "ds-idrepo-0" 06:57:32 DEBUG }, 06:57:32 DEBUG "value": [ 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG [...] 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG "pod": "ds-cts-2" 06:57:32 DEBUG }, 06:57:32 DEBUG "value": [ 06:57:32 DEBUG 1712728652, 06:57:32 DEBUG "30" 06:57:32 DEBUG ] 06:57:32 DEBUG } 06:57:32 DEBUG ] 06:57:32 DEBUG } 06:57:32 DEBUG } 06:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728652" 06:57:32 INFO [http_cmd]: http status code OK 06:57:32 DEBUG --- status code --- 06:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:32 DEBUG --- http response --- 06:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:57:32 DEBUG ----- output ----- 06:57:32 DEBUG { 06:57:32 DEBUG "status": "success", 06:57:32 DEBUG "data": { 06:57:32 DEBUG "resultType": "vector", 06:57:32 DEBUG "result": [ 06:57:32 DEBUG { 06:57:32 DEBUG "metric": { 06:57:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:57:32 DEBUG }, 06:57:32 DEBUG "value": [ 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG [...] 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG "pod": "login-ui-5f964fb894-5zpvq" 06:57:32 DEBUG }, 06:57:32 DEBUG "value": [ 06:57:32 DEBUG 1712728652, 06:57:32 DEBUG "0" 06:57:32 DEBUG ] 06:57:32 DEBUG } 06:57:32 DEBUG ] 06:57:32 DEBUG } 06:57:32 DEBUG } 06:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712728652" 06:57:32 INFO [http_cmd]: http status code OK 06:57:32 DEBUG --- status code --- 06:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:32 DEBUG --- http response --- 06:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:57:32 DEBUG ----- output ----- 06:57:32 DEBUG { 06:57:32 DEBUG "status": "success", 06:57:32 DEBUG "data": { 06:57:32 DEBUG "resultType": "vector", 06:57:32 DEBUG "result": [ 06:57:32 DEBUG { 06:57:32 DEBUG "metric": { 06:57:32 DEBUG "pod": "ds-idrepo-0" 06:57:32 DEBUG }, 06:57:32 DEBUG "value": [ 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG [...] 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG 06:57:32 DEBUG "pod": "ds-cts-2" 06:57:32 DEBUG }, 06:57:32 DEBUG "value": [ 06:57:32 DEBUG 1712728652, 06:57:32 DEBUG "0" 06:57:32 DEBUG ] 06:57:32 DEBUG } 06:57:32 DEBUG ] 06:57:32 DEBUG } 06:57:32 DEBUG } 06:57:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728653" 06:57:33 INFO [http_cmd]: http status code OK 06:57:33 DEBUG --- status code --- 06:57:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:33 DEBUG --- http response --- 06:57:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:57:33 DEBUG ----- output ----- 06:57:33 DEBUG { 06:57:33 DEBUG "status": "success", 06:57:33 DEBUG "data": { 06:57:33 DEBUG "resultType": "vector", 06:57:33 DEBUG "result": [ 06:57:33 DEBUG { 06:57:33 DEBUG "metric": { 06:57:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:57:33 DEBUG }, 06:57:33 DEBUG "value": [ 06:57:33 DEBUG 06:57:33 DEBUG 06:57:33 DEBUG 06:57:33 DEBUG [...] 06:57:33 DEBUG 06:57:33 DEBUG 06:57:33 DEBUG 06:57:33 DEBUG "pod": "am-654c4fcbcf-487cv" 06:57:33 DEBUG }, 06:57:33 DEBUG "value": [ 06:57:33 DEBUG 1712728653, 06:57:33 DEBUG "0" 06:57:33 DEBUG ] 06:57:33 DEBUG } 06:57:33 DEBUG ] 06:57:33 DEBUG } 06:57:33 DEBUG } 06:57:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712728653" 06:57:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728653" 06:57:33 INFO [http_cmd]: http status code OK 06:57:33 DEBUG --- status code --- 06:57:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:33 DEBUG --- http response --- 06:57:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:57:33 DEBUG ----- output ----- 06:57:33 DEBUG { 06:57:33 DEBUG "status": "success", 06:57:33 DEBUG "data": { 06:57:33 DEBUG "resultType": "vector", 06:57:33 DEBUG "result": [ 06:57:33 DEBUG { 06:57:33 DEBUG "metric": { 06:57:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:57:33 DEBUG }, 06:57:33 DEBUG "value": [ 06:57:33 DEBUG 06:57:33 DEBUG 06:57:33 DEBUG 06:57:33 DEBUG [...] 06:57:33 DEBUG 06:57:33 DEBUG 06:57:33 DEBUG 06:57:33 DEBUG "pod": "am-654c4fcbcf-487cv" 06:57:33 DEBUG }, 06:57:33 DEBUG "value": [ 06:57:33 DEBUG 1712728653, 06:57:33 DEBUG "0.054" 06:57:33 DEBUG ] 06:57:33 DEBUG } 06:57:33 DEBUG ] 06:57:33 DEBUG } 06:57:33 DEBUG } 06:57:33 INFO [http_cmd]: http status code OK 06:57:33 DEBUG --- status code --- 06:57:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:33 DEBUG --- http response --- 06:57:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:57:33 DEBUG ----- output ----- 06:57:33 DEBUG { 06:57:33 DEBUG "status": "success", 06:57:33 DEBUG "data": { 06:57:33 DEBUG "resultType": "vector", 06:57:33 DEBUG "result": [ 06:57:33 DEBUG { 06:57:33 DEBUG "metric": { 06:57:33 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:57:33 DEBUG }, 06:57:33 DEBUG "value": [ 06:57:33 DEBUG 06:57:33 DEBUG 06:57:33 DEBUG 06:57:33 DEBUG [...] 06:57:33 DEBUG 06:57:33 DEBUG 06:57:33 DEBUG 06:57:33 DEBUG "pod": "login-ui-5f964fb894-5zpvq" 06:57:33 DEBUG }, 06:57:33 DEBUG "value": [ 06:57:33 DEBUG 1712728653, 06:57:33 DEBUG "0.000023430208258782643" 06:57:33 DEBUG ] 06:57:33 DEBUG } 06:57:33 DEBUG ] 06:57:33 DEBUG } 06:57:33 DEBUG } 06:57:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728653" 06:57:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728653" 06:57:33 INFO [http_cmd]: http status code OK 06:57:33 DEBUG --- status code --- 06:57:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:33 DEBUG --- http response --- 06:57:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:57:33 DEBUG ----- output ----- 06:57:33 DEBUG { 06:57:33 DEBUG "status": "success", 06:57:33 DEBUG "data": { 06:57:33 DEBUG "resultType": "vector", 06:57:33 DEBUG "result": [ 06:57:33 DEBUG { 06:57:33 DEBUG "metric": { 06:57:33 DEBUG "pod": "ds-idrepo-2" 06:57:33 DEBUG }, 06:57:33 DEBUG "value": [ 06:57:33 DEBUG 06:57:33 DEBUG 06:57:33 DEBUG 06:57:33 DEBUG [...] 06:57:33 DEBUG 06:57:33 DEBUG 06:57:33 DEBUG 06:57:33 DEBUG "pod": "ds-cts-2" 06:57:33 DEBUG }, 06:57:33 DEBUG "value": [ 06:57:33 DEBUG 1712728653, 06:57:33 DEBUG "0" 06:57:33 DEBUG ] 06:57:33 DEBUG } 06:57:33 DEBUG ] 06:57:33 DEBUG } 06:57:33 DEBUG } 06:57:33 INFO [http_cmd]: http status code OK 06:57:33 DEBUG --- status code --- 06:57:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:33 DEBUG --- http response --- 06:57:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:57:33 DEBUG ----- output ----- 06:57:33 DEBUG { 06:57:33 DEBUG "status": "success", 06:57:33 DEBUG "data": { 06:57:33 DEBUG "resultType": "vector", 06:57:33 DEBUG "result": [ 06:57:33 DEBUG { 06:57:33 DEBUG "metric": { 06:57:33 DEBUG "pod": "ds-idrepo-0" 06:57:33 DEBUG }, 06:57:33 DEBUG "value": [ 06:57:33 DEBUG 06:57:33 DEBUG 06:57:33 DEBUG 06:57:33 DEBUG [...] 06:57:33 DEBUG 06:57:33 DEBUG 06:57:33 DEBUG 06:57:33 DEBUG "pod": "ds-idrepo-1" 06:57:33 DEBUG }, 06:57:33 DEBUG "value": [ 06:57:33 DEBUG 1712728653, 06:57:33 DEBUG "0" 06:57:33 DEBUG ] 06:57:33 DEBUG } 06:57:33 DEBUG ] 06:57:33 DEBUG } 06:57:33 DEBUG } 06:57:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728653" 06:57:33 INFO [http_cmd]: http status code OK 06:57:33 DEBUG --- status code --- 06:57:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:33 DEBUG --- http response --- 06:57:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:57:33 DEBUG ----- output ----- 06:57:33 DEBUG { 06:57:33 DEBUG "status": "success", 06:57:33 DEBUG "data": { 06:57:33 DEBUG "resultType": "vector", 06:57:33 DEBUG "result": [ 06:57:33 DEBUG { 06:57:33 DEBUG "metric": { 06:57:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:57:33 DEBUG }, 06:57:33 DEBUG "value": [ 06:57:33 DEBUG 06:57:33 DEBUG 06:57:33 DEBUG 06:57:33 DEBUG [...] 06:57:33 DEBUG 06:57:33 DEBUG 06:57:33 DEBUG 06:57:33 DEBUG "pod": "am-654c4fcbcf-487cv" 06:57:33 DEBUG }, 06:57:33 DEBUG "value": [ 06:57:33 DEBUG 1712728653, 06:57:33 DEBUG "0" 06:57:33 DEBUG ] 06:57:33 DEBUG } 06:57:33 DEBUG ] 06:57:33 DEBUG } 06:57:33 DEBUG } 06:57:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728653" 06:57:33 INFO [http_cmd]: http status code OK 06:57:33 DEBUG --- status code --- 06:57:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:33 DEBUG --- http response --- 06:57:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:57:33 DEBUG ----- output ----- 06:57:33 DEBUG { 06:57:33 DEBUG "status": "success", 06:57:33 DEBUG "data": { 06:57:33 DEBUG "resultType": "vector", 06:57:33 DEBUG "result": [ 06:57:33 DEBUG { 06:57:33 DEBUG "metric": { 06:57:33 DEBUG "pod": "ds-idrepo-0" 06:57:33 DEBUG }, 06:57:33 DEBUG "value": [ 06:57:33 DEBUG 06:57:33 DEBUG 06:57:33 DEBUG 06:57:33 DEBUG [...] 06:57:33 DEBUG 06:57:33 DEBUG 06:57:33 DEBUG 06:57:33 DEBUG "pod": "ds-idrepo-1" 06:57:33 DEBUG }, 06:57:33 DEBUG "value": [ 06:57:33 DEBUG 1712728653, 06:57:33 DEBUG "0" 06:57:33 DEBUG ] 06:57:33 DEBUG } 06:57:33 DEBUG ] 06:57:33 DEBUG } 06:57:33 DEBUG } 06:57:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728654" 06:57:34 INFO [http_cmd]: http status code OK 06:57:34 DEBUG --- status code --- 06:57:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:34 DEBUG --- http response --- 06:57:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:57:34 DEBUG ----- output ----- 06:57:34 DEBUG { 06:57:34 DEBUG "status": "success", 06:57:34 DEBUG "data": { 06:57:34 DEBUG "resultType": "vector", 06:57:34 DEBUG "result": [ 06:57:34 DEBUG { 06:57:34 DEBUG "metric": { 06:57:34 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:57:34 DEBUG }, 06:57:34 DEBUG "value": [ 06:57:34 DEBUG 06:57:34 DEBUG 06:57:34 DEBUG 06:57:34 DEBUG [...] 06:57:34 DEBUG 06:57:34 DEBUG 06:57:34 DEBUG 06:57:34 DEBUG "pod": "login-ui-5f964fb894-5zpvq" 06:57:34 DEBUG }, 06:57:34 DEBUG "value": [ 06:57:34 DEBUG 1712728654, 06:57:34 DEBUG "0" 06:57:34 DEBUG ] 06:57:34 DEBUG } 06:57:34 DEBUG ] 06:57:34 DEBUG } 06:57:34 DEBUG } 06:57:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712728654" 06:57:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728654" 06:57:34 INFO [http_cmd]: http status code OK 06:57:34 DEBUG --- status code --- 06:57:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:34 DEBUG --- http response --- 06:57:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:57:34 DEBUG ----- output ----- 06:57:34 DEBUG { 06:57:34 DEBUG "status": "success", 06:57:34 DEBUG "data": { 06:57:34 DEBUG "resultType": "vector", 06:57:34 DEBUG "result": [ 06:57:34 DEBUG { 06:57:34 DEBUG "metric": { 06:57:34 DEBUG "instance": "10.142.0.25:9100" 06:57:34 DEBUG }, 06:57:34 DEBUG "value": [ 06:57:34 DEBUG 06:57:34 DEBUG 06:57:34 DEBUG 06:57:34 DEBUG [...] 06:57:34 DEBUG 06:57:34 DEBUG 06:57:34 DEBUG 06:57:34 DEBUG "instance": "10.142.0.77:9100" 06:57:34 DEBUG }, 06:57:34 DEBUG "value": [ 06:57:34 DEBUG 1712728654, 06:57:34 DEBUG "28101.233333333334" 06:57:34 DEBUG ] 06:57:34 DEBUG } 06:57:34 DEBUG ] 06:57:34 DEBUG } 06:57:34 DEBUG } 06:57:34 INFO [http_cmd]: http status code OK 06:57:34 DEBUG --- status code --- 06:57:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:34 DEBUG --- http response --- 06:57:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:57:34 DEBUG ----- output ----- 06:57:34 DEBUG { 06:57:34 DEBUG "status": "success", 06:57:34 DEBUG "data": { 06:57:34 DEBUG "resultType": "vector", 06:57:34 DEBUG "result": [ 06:57:34 DEBUG { 06:57:34 DEBUG "metric": { 06:57:34 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:57:34 DEBUG }, 06:57:34 DEBUG "value": [ 06:57:34 DEBUG 06:57:34 DEBUG 06:57:34 DEBUG 06:57:34 DEBUG [...] 06:57:34 DEBUG 06:57:34 DEBUG 06:57:34 DEBUG 06:57:34 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:57:34 DEBUG }, 06:57:34 DEBUG "value": [ 06:57:34 DEBUG 1712728654, 06:57:34 DEBUG "2487951360" 06:57:34 DEBUG ] 06:57:34 DEBUG } 06:57:34 DEBUG ] 06:57:34 DEBUG } 06:57:34 DEBUG } 06:57:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728654" 06:57:34 INFO [http_cmd]: http status code OK 06:57:34 DEBUG --- status code --- 06:57:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:34 DEBUG --- http response --- 06:57:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:57:34 DEBUG ----- output ----- 06:57:34 DEBUG { 06:57:34 DEBUG "status": "success", 06:57:34 DEBUG "data": { 06:57:34 DEBUG "resultType": "vector", 06:57:34 DEBUG "result": [ 06:57:34 DEBUG { 06:57:34 DEBUG "metric": { 06:57:34 DEBUG "instance": "10.142.0.25:9100" 06:57:34 DEBUG }, 06:57:34 DEBUG "value": [ 06:57:34 DEBUG 06:57:34 DEBUG 06:57:34 DEBUG 06:57:34 DEBUG [...] 06:57:34 DEBUG 06:57:34 DEBUG 06:57:34 DEBUG 06:57:34 DEBUG "instance": "10.142.0.77:9100" 06:57:34 DEBUG }, 06:57:34 DEBUG "value": [ 06:57:34 DEBUG 1712728654, 06:57:34 DEBUG "0" 06:57:34 DEBUG ] 06:57:34 DEBUG } 06:57:34 DEBUG ] 06:57:34 DEBUG } 06:57:34 DEBUG } 06:57:55 INFO 06:57:55 INFO [loop_until]: kubectl --namespace=xlou top pods 06:57:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:57:55 INFO [loop_until]: OK (rc = 0) 06:57:55 DEBUG --- stdout --- 06:57:55 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 9Mi am-654c4fcbcf-487cv 8m 8870Mi am-654c4fcbcf-9scrj 7m 8378Mi am-654c4fcbcf-v5qb5 8m 8096Mi ds-cts-0 16m 12723Mi ds-cts-1 17m 12921Mi ds-cts-2 14m 11473Mi ds-idrepo-0 16m 5620Mi ds-idrepo-1 10m 5677Mi ds-idrepo-2 12m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 5Mi idm-6bfb7c96cc-qklk9 4m 1198Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 5m 73Mi login-ui-5f964fb894-5zpvq 1m 4Mi overseer-0-6f8784665d-zq5hb 583m 4059Mi 06:57:55 DEBUG --- stderr --- 06:57:55 DEBUG 06:58:04 INFO 06:58:04 INFO [loop_until]: kubectl --namespace=xlou top node 06:58:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:58:04 INFO [loop_until]: OK (rc = 0) 06:58:04 DEBUG --- stdout --- 06:58:04 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 80m 0% 2319Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 68m 0% 8990Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 77m 0% 3257Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 211m 1% 6531Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 67m 0% 9281Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 67m 0% 9729Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 75m 0% 860Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 67m 0% 6320Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 73m 0% 13458Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 98m 0% 12214Mi 20% gke-xlou-cdm-ds-32e4dcb1-gxzj 65m 0% 6402Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 71m 0% 13657Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 66m 0% 6370Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 489m 3% 7683Mi 13% 06:58:04 DEBUG --- stderr --- 06:58:04 DEBUG 06:58:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728711" 06:58:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712728711" 06:58:31 ERROR [http_cmd]: ERROR 06:58:31 INFO [http_cmd]: http status code OK 06:58:31 ERROR --- status code --- 06:58:31 DEBUG --- status code --- 06:58:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:58:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:31 ERROR --- http response --- 06:58:31 DEBUG --- http response --- 06:58:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:58:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:58:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:58:31 DEBUG ----- output ----- 06:58:31 DEBUG { 06:58:31 DEBUG "status": "success", 06:58:31 DEBUG "data": { 06:58:31 DEBUG "resultType": "vector", 06:58:31 DEBUG "result": [ 06:58:31 DEBUG { 06:58:31 DEBUG "metric": { 06:58:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:58:31 DEBUG }, 06:58:31 DEBUG "value": [ 06:58:31 DEBUG 06:58:31 DEBUG 06:58:31 DEBUG 06:58:31 DEBUG [...] 06:58:31 DEBUG 06:58:31 DEBUG 06:58:31 DEBUG 06:58:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:58:31 DEBUG }, 06:58:31 DEBUG "value": [ 06:58:31 DEBUG 1712728711, 06:58:31 DEBUG "0" 06:58:31 DEBUG ] 06:58:31 DEBUG } 06:58:31 DEBUG ] 06:58:31 DEBUG } 06:58:31 DEBUG } 06:58:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728711" 06:58:31 INFO [http_cmd]: http status code OK 06:58:31 DEBUG --- status code --- 06:58:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:31 DEBUG --- http response --- 06:58:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728712" 06:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728712" 06:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728712" 06:58:32 INFO [http_cmd]: http status code OK 06:58:32 DEBUG --- status code --- 06:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:32 DEBUG --- http response --- 06:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:58:32 DEBUG ----- output ----- 06:58:32 DEBUG { 06:58:32 DEBUG "status": "success", 06:58:32 DEBUG "data": { 06:58:32 DEBUG "resultType": "vector", 06:58:32 DEBUG "result": [ 06:58:32 DEBUG { 06:58:32 DEBUG "metric": { 06:58:32 DEBUG "instance": "10.142.0.25:9100" 06:58:32 DEBUG }, 06:58:32 DEBUG "value": [ 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG [...] 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG "instance": "10.142.0.77:9100" 06:58:32 DEBUG }, 06:58:32 DEBUG "value": [ 06:58:32 DEBUG 1712728712, 06:58:32 DEBUG "25128" 06:58:32 DEBUG ] 06:58:32 DEBUG } 06:58:32 DEBUG ] 06:58:32 DEBUG } 06:58:32 DEBUG } 06:58:32 INFO [http_cmd]: http status code OK 06:58:32 DEBUG --- status code --- 06:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:32 DEBUG --- http response --- 06:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:58:32 DEBUG ----- output ----- 06:58:32 DEBUG { 06:58:32 DEBUG "status": "success", 06:58:32 DEBUG "data": { 06:58:32 DEBUG "resultType": "vector", 06:58:32 DEBUG "result": [ 06:58:32 DEBUG { 06:58:32 DEBUG "metric": { 06:58:32 DEBUG "pod": "ds-idrepo-0" 06:58:32 DEBUG }, 06:58:32 DEBUG "value": [ 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG [...] 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG "pod": "ds-idrepo-1" 06:58:32 DEBUG }, 06:58:32 DEBUG "value": [ 06:58:32 DEBUG 1712728712, 06:58:32 DEBUG "0" 06:58:32 DEBUG ] 06:58:32 DEBUG } 06:58:32 DEBUG ] 06:58:32 DEBUG } 06:58:32 DEBUG } 06:58:32 INFO [http_cmd]: http status code OK 06:58:32 DEBUG --- status code --- 06:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:32 DEBUG --- http response --- 06:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:58:32 DEBUG ----- output ----- 06:58:32 DEBUG { 06:58:32 DEBUG "status": "success", 06:58:32 DEBUG "data": { 06:58:32 DEBUG "resultType": "vector", 06:58:32 DEBUG "result": [ 06:58:32 DEBUG { 06:58:32 DEBUG "metric": { 06:58:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:58:32 DEBUG }, 06:58:32 DEBUG "value": [ 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG [...] 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:58:32 DEBUG }, 06:58:32 DEBUG "value": [ 06:58:32 DEBUG 1712728712, 06:58:32 DEBUG "0" 06:58:32 DEBUG ] 06:58:32 DEBUG } 06:58:32 DEBUG ] 06:58:32 DEBUG } 06:58:32 DEBUG } 06:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728712" 06:58:32 INFO [http_cmd]: http status code OK 06:58:32 DEBUG --- status code --- 06:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:32 DEBUG --- http response --- 06:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:58:32 DEBUG ----- output ----- 06:58:32 DEBUG { 06:58:32 DEBUG "status": "success", 06:58:32 DEBUG "data": { 06:58:32 DEBUG "resultType": "vector", 06:58:32 DEBUG "result": [ 06:58:32 DEBUG { 06:58:32 DEBUG "metric": { 06:58:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:58:32 DEBUG }, 06:58:32 DEBUG "value": [ 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG [...] 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:58:32 DEBUG }, 06:58:32 DEBUG "value": [ 06:58:32 DEBUG 1712728712, 06:58:32 DEBUG "0" 06:58:32 DEBUG ] 06:58:32 DEBUG } 06:58:32 DEBUG ] 06:58:32 DEBUG } 06:58:32 DEBUG } 06:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728712" 06:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712728712" 06:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728712" 06:58:32 INFO [http_cmd]: http status code OK 06:58:32 INFO [http_cmd]: http status code OK 06:58:32 DEBUG --- status code --- 06:58:32 DEBUG --- status code --- 06:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:32 INFO [http_cmd]: http status code OK 06:58:32 DEBUG --- http response --- 06:58:32 DEBUG --- http response --- 06:58:32 DEBUG --- status code --- 06:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:32 DEBUG ----- output ----- 06:58:32 DEBUG ----- output ----- 06:58:32 DEBUG --- http response --- 06:58:32 DEBUG { 06:58:32 DEBUG { 06:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:58:32 DEBUG "status": "success", 06:58:32 DEBUG "status": "success", 06:58:32 DEBUG ----- output ----- 06:58:32 DEBUG "data": { 06:58:32 DEBUG "data": { 06:58:32 DEBUG { 06:58:32 DEBUG "resultType": "vector", 06:58:32 DEBUG "resultType": "vector", 06:58:32 DEBUG "status": "success", 06:58:32 DEBUG "result": [ 06:58:32 DEBUG "result": [ 06:58:32 DEBUG "data": { 06:58:32 DEBUG { 06:58:32 DEBUG { 06:58:32 DEBUG "resultType": "vector", 06:58:32 DEBUG "metric": { 06:58:32 DEBUG "metric": { 06:58:32 DEBUG "result": [ 06:58:32 DEBUG "instance": "10.142.0.25:9100" 06:58:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:58:32 DEBUG { 06:58:32 DEBUG }, 06:58:32 DEBUG }, 06:58:32 DEBUG "metric": { 06:58:32 DEBUG "value": [ 06:58:32 DEBUG "value": [ 06:58:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG }, 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG "value": [ 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG [...] 06:58:32 DEBUG [...] 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG [...] 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG "instance": "10.142.0.77:9100" 06:58:32 DEBUG "pod": "ds-cts-0" 06:58:32 DEBUG 06:58:32 DEBUG }, 06:58:32 DEBUG }, 06:58:32 DEBUG 06:58:32 DEBUG "value": [ 06:58:32 DEBUG "value": [ 06:58:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:58:32 DEBUG 1712728712, 06:58:32 DEBUG 1712728712, 06:58:32 DEBUG }, 06:58:32 DEBUG "438090322944" 06:58:32 DEBUG "0" 06:58:32 DEBUG "value": [ 06:58:32 DEBUG ] 06:58:32 DEBUG ] 06:58:32 DEBUG 1712728712, 06:58:32 DEBUG } 06:58:32 DEBUG } 06:58:32 DEBUG "0.748399445965407" 06:58:32 DEBUG ] 06:58:32 DEBUG ] 06:58:32 DEBUG ] 06:58:32 DEBUG } 06:58:32 DEBUG } 06:58:32 DEBUG } 06:58:32 DEBUG } 06:58:32 DEBUG } 06:58:32 DEBUG ] 06:58:32 DEBUG } 06:58:32 DEBUG } 06:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728712" 06:58:32 INFO [http_cmd]: http status code OK 06:58:32 DEBUG --- status code --- 06:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:32 DEBUG --- http response --- 06:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:58:32 DEBUG ----- output ----- 06:58:32 DEBUG { 06:58:32 DEBUG "status": "success", 06:58:32 DEBUG "data": { 06:58:32 DEBUG "resultType": "vector", 06:58:32 DEBUG "result": [ 06:58:32 DEBUG { 06:58:32 DEBUG "metric": { 06:58:32 DEBUG "pod": "ds-idrepo-0" 06:58:32 DEBUG }, 06:58:32 DEBUG "value": [ 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG [...] 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG "pod": "ds-cts-2" 06:58:32 DEBUG }, 06:58:32 DEBUG "value": [ 06:58:32 DEBUG 1712728712, 06:58:32 DEBUG "0" 06:58:32 DEBUG ] 06:58:32 DEBUG } 06:58:32 DEBUG ] 06:58:32 DEBUG } 06:58:32 DEBUG } 06:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728712" 06:58:32 INFO [http_cmd]: http status code OK 06:58:32 DEBUG --- status code --- 06:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:32 DEBUG --- http response --- 06:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:58:32 DEBUG ----- output ----- 06:58:32 DEBUG { 06:58:32 DEBUG "status": "success", 06:58:32 DEBUG "data": { 06:58:32 DEBUG "resultType": "vector", 06:58:32 DEBUG "result": [ 06:58:32 DEBUG { 06:58:32 DEBUG "metric": { 06:58:32 DEBUG "instance": "10.142.0.25:9100" 06:58:32 DEBUG }, 06:58:32 DEBUG "value": [ 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG [...] 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG "instance": "10.142.0.77:9100" 06:58:32 DEBUG }, 06:58:32 DEBUG "value": [ 06:58:32 DEBUG 1712728712, 06:58:32 DEBUG "0.0008333333333283121" 06:58:32 DEBUG ] 06:58:32 DEBUG } 06:58:32 DEBUG ] 06:58:32 DEBUG } 06:58:32 DEBUG } 06:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728712" 06:58:32 INFO [http_cmd]: http status code OK 06:58:32 DEBUG --- status code --- 06:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:32 DEBUG --- http response --- 06:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:58:32 DEBUG ----- output ----- 06:58:32 DEBUG { 06:58:32 DEBUG "status": "success", 06:58:32 DEBUG "data": { 06:58:32 DEBUG "resultType": "vector", 06:58:32 DEBUG "result": [ 06:58:32 DEBUG { 06:58:32 DEBUG "metric": { 06:58:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:58:32 DEBUG }, 06:58:32 DEBUG "value": [ 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG [...] 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:58:32 DEBUG }, 06:58:32 DEBUG "value": [ 06:58:32 DEBUG 1712728712, 06:58:32 DEBUG "0" 06:58:32 DEBUG ] 06:58:32 DEBUG } 06:58:32 DEBUG ] 06:58:32 DEBUG } 06:58:32 DEBUG } 06:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728712" 06:58:32 INFO [http_cmd]: http status code OK 06:58:32 DEBUG --- status code --- 06:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:32 DEBUG --- http response --- 06:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:58:32 DEBUG ----- output ----- 06:58:32 DEBUG { 06:58:32 DEBUG "status": "success", 06:58:32 DEBUG "data": { 06:58:32 DEBUG "resultType": "vector", 06:58:32 DEBUG "result": [ 06:58:32 DEBUG { 06:58:32 DEBUG "metric": { 06:58:32 DEBUG "pod": "ds-idrepo-0" 06:58:32 DEBUG }, 06:58:32 DEBUG "value": [ 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG [...] 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG "pod": "ds-cts-2" 06:58:32 DEBUG }, 06:58:32 DEBUG "value": [ 06:58:32 DEBUG 1712728712, 06:58:32 DEBUG "28.8" 06:58:32 DEBUG ] 06:58:32 DEBUG } 06:58:32 DEBUG ] 06:58:32 DEBUG } 06:58:32 DEBUG } 06:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728712" 06:58:32 INFO [http_cmd]: http status code OK 06:58:32 DEBUG --- status code --- 06:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:32 DEBUG --- http response --- 06:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:58:32 DEBUG ----- output ----- 06:58:32 DEBUG { 06:58:32 DEBUG "status": "success", 06:58:32 DEBUG "data": { 06:58:32 DEBUG "resultType": "vector", 06:58:32 DEBUG "result": [ 06:58:32 DEBUG { 06:58:32 DEBUG "metric": { 06:58:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:58:32 DEBUG }, 06:58:32 DEBUG "value": [ 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG [...] 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG "pod": "ds-cts-0" 06:58:32 DEBUG }, 06:58:32 DEBUG "value": [ 06:58:32 DEBUG 1712728712, 06:58:32 DEBUG "1237.1300087254178" 06:58:32 DEBUG ] 06:58:32 DEBUG } 06:58:32 DEBUG ] 06:58:32 DEBUG } 06:58:32 DEBUG } 06:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712728712" 06:58:32 INFO [http_cmd]: http status code OK 06:58:32 DEBUG --- status code --- 06:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:32 DEBUG --- http response --- 06:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:58:32 DEBUG ----- output ----- 06:58:32 DEBUG { 06:58:32 DEBUG "status": "success", 06:58:32 DEBUG "data": { 06:58:32 DEBUG "resultType": "vector", 06:58:32 DEBUG "result": [ 06:58:32 DEBUG { 06:58:32 DEBUG "metric": { 06:58:32 DEBUG "pod": "ds-idrepo-0" 06:58:32 DEBUG }, 06:58:32 DEBUG "value": [ 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG [...] 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG 06:58:32 DEBUG "pod": "ds-cts-2" 06:58:32 DEBUG }, 06:58:32 DEBUG "value": [ 06:58:32 DEBUG 1712728712, 06:58:32 DEBUG "0" 06:58:32 DEBUG ] 06:58:32 DEBUG } 06:58:32 DEBUG ] 06:58:32 DEBUG } 06:58:32 DEBUG } 06:58:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728713" 06:58:33 INFO [http_cmd]: http status code OK 06:58:33 DEBUG --- status code --- 06:58:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:33 DEBUG --- http response --- 06:58:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:58:33 DEBUG ----- output ----- 06:58:33 DEBUG { 06:58:33 DEBUG "status": "success", 06:58:33 DEBUG "data": { 06:58:33 DEBUG "resultType": "vector", 06:58:33 DEBUG "result": [ 06:58:33 DEBUG { 06:58:33 DEBUG "metric": { 06:58:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:58:33 DEBUG }, 06:58:33 DEBUG "value": [ 06:58:33 DEBUG 06:58:33 DEBUG 06:58:33 DEBUG 06:58:33 DEBUG [...] 06:58:33 DEBUG 06:58:33 DEBUG 06:58:33 DEBUG 06:58:33 DEBUG "pod": "am-654c4fcbcf-487cv" 06:58:33 DEBUG }, 06:58:33 DEBUG "value": [ 06:58:33 DEBUG 1712728713, 06:58:33 DEBUG "0" 06:58:33 DEBUG ] 06:58:33 DEBUG } 06:58:33 DEBUG ] 06:58:33 DEBUG } 06:58:33 DEBUG } 06:58:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712728713" 06:58:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728713" 06:58:33 INFO [http_cmd]: http status code OK 06:58:33 DEBUG --- status code --- 06:58:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:33 DEBUG --- http response --- 06:58:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:58:33 DEBUG ----- output ----- 06:58:33 DEBUG { 06:58:33 DEBUG "status": "success", 06:58:33 DEBUG "data": { 06:58:33 DEBUG "resultType": "vector", 06:58:33 DEBUG "result": [ 06:58:33 DEBUG { 06:58:33 DEBUG "metric": { 06:58:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:58:33 DEBUG }, 06:58:33 DEBUG "value": [ 06:58:33 DEBUG 06:58:33 DEBUG 06:58:33 DEBUG 06:58:33 DEBUG [...] 06:58:33 DEBUG 06:58:33 DEBUG 06:58:33 DEBUG 06:58:33 DEBUG "pod": "am-654c4fcbcf-487cv" 06:58:33 DEBUG }, 06:58:33 DEBUG "value": [ 06:58:33 DEBUG 1712728713, 06:58:33 DEBUG "0.054" 06:58:33 DEBUG ] 06:58:33 DEBUG } 06:58:33 DEBUG ] 06:58:33 DEBUG } 06:58:33 DEBUG } 06:58:33 INFO [http_cmd]: http status code OK 06:58:33 DEBUG --- status code --- 06:58:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:33 DEBUG --- http response --- 06:58:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:58:33 DEBUG ----- output ----- 06:58:33 DEBUG { 06:58:33 DEBUG "status": "success", 06:58:33 DEBUG "data": { 06:58:33 DEBUG "resultType": "vector", 06:58:33 DEBUG "result": [ 06:58:33 DEBUG { 06:58:33 DEBUG "metric": { 06:58:33 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:58:33 DEBUG }, 06:58:33 DEBUG "value": [ 06:58:33 DEBUG 06:58:33 DEBUG 06:58:33 DEBUG 06:58:33 DEBUG [...] 06:58:33 DEBUG 06:58:33 DEBUG 06:58:33 DEBUG 06:58:33 DEBUG "pod": "ds-cts-0" 06:58:33 DEBUG }, 06:58:33 DEBUG "value": [ 06:58:33 DEBUG 1712728713, 06:58:33 DEBUG "0.01615708916702875" 06:58:33 DEBUG ] 06:58:33 DEBUG } 06:58:33 DEBUG ] 06:58:33 DEBUG } 06:58:33 DEBUG } 06:58:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728713" 06:58:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728713" 06:58:33 INFO [http_cmd]: http status code OK 06:58:33 DEBUG --- status code --- 06:58:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:33 DEBUG --- http response --- 06:58:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:58:33 DEBUG ----- output ----- 06:58:33 DEBUG { 06:58:33 DEBUG "status": "success", 06:58:33 DEBUG "data": { 06:58:33 DEBUG "resultType": "vector", 06:58:33 DEBUG "result": [ 06:58:33 DEBUG { 06:58:33 DEBUG "metric": { 06:58:33 DEBUG "pod": "ds-idrepo-2" 06:58:33 DEBUG }, 06:58:33 DEBUG "value": [ 06:58:33 DEBUG 06:58:33 DEBUG 06:58:33 DEBUG 06:58:33 DEBUG [...] 06:58:33 DEBUG 06:58:33 DEBUG 06:58:33 DEBUG 06:58:33 DEBUG "pod": "ds-cts-2" 06:58:33 DEBUG }, 06:58:33 INFO [http_cmd]: http status code OK 06:58:33 DEBUG "value": [ 06:58:33 DEBUG --- status code --- 06:58:33 DEBUG 1712728713, 06:58:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:33 DEBUG "0" 06:58:33 DEBUG --- http response --- 06:58:33 DEBUG ] 06:58:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:58:33 DEBUG } 06:58:33 DEBUG ----- output ----- 06:58:33 DEBUG ] 06:58:33 DEBUG { 06:58:33 DEBUG } 06:58:33 DEBUG "status": "success", 06:58:33 DEBUG } 06:58:33 DEBUG "data": { 06:58:33 DEBUG "resultType": "vector", 06:58:33 DEBUG "result": [ 06:58:33 DEBUG { 06:58:33 DEBUG "metric": { 06:58:33 DEBUG "pod": "ds-idrepo-0" 06:58:33 DEBUG }, 06:58:33 DEBUG "value": [ 06:58:33 DEBUG 06:58:33 DEBUG 06:58:33 DEBUG 06:58:33 DEBUG [...] 06:58:33 DEBUG 06:58:33 DEBUG 06:58:33 DEBUG 06:58:33 DEBUG "pod": "ds-idrepo-1" 06:58:33 DEBUG }, 06:58:33 DEBUG "value": [ 06:58:33 DEBUG 1712728713, 06:58:33 DEBUG "0" 06:58:33 DEBUG ] 06:58:33 DEBUG } 06:58:33 DEBUG ] 06:58:33 DEBUG } 06:58:33 DEBUG } 06:58:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728713" 06:58:33 INFO [http_cmd]: http status code OK 06:58:33 DEBUG --- status code --- 06:58:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:33 DEBUG --- http response --- 06:58:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:58:33 DEBUG ----- output ----- 06:58:33 DEBUG { 06:58:33 DEBUG "status": "success", 06:58:33 DEBUG "data": { 06:58:33 DEBUG "resultType": "vector", 06:58:33 DEBUG "result": [ 06:58:33 DEBUG { 06:58:33 DEBUG "metric": { 06:58:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:58:33 DEBUG }, 06:58:33 DEBUG "value": [ 06:58:33 DEBUG 06:58:33 DEBUG 06:58:33 DEBUG 06:58:33 DEBUG [...] 06:58:33 DEBUG 06:58:33 DEBUG 06:58:33 DEBUG 06:58:33 DEBUG "pod": "am-654c4fcbcf-487cv" 06:58:33 DEBUG }, 06:58:33 DEBUG "value": [ 06:58:33 DEBUG 1712728713, 06:58:33 DEBUG "0" 06:58:33 DEBUG ] 06:58:33 DEBUG } 06:58:33 DEBUG ] 06:58:33 DEBUG } 06:58:33 DEBUG } 06:58:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728713" 06:58:33 INFO [http_cmd]: http status code OK 06:58:33 DEBUG --- status code --- 06:58:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:33 DEBUG --- http response --- 06:58:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:58:33 DEBUG ----- output ----- 06:58:33 DEBUG { 06:58:33 DEBUG "status": "success", 06:58:33 DEBUG "data": { 06:58:33 DEBUG "resultType": "vector", 06:58:33 DEBUG "result": [ 06:58:33 DEBUG { 06:58:33 DEBUG "metric": { 06:58:33 DEBUG "pod": "ds-idrepo-0" 06:58:33 DEBUG }, 06:58:33 DEBUG "value": [ 06:58:33 DEBUG 06:58:33 DEBUG 06:58:33 DEBUG 06:58:33 DEBUG [...] 06:58:33 DEBUG 06:58:33 DEBUG 06:58:33 DEBUG 06:58:33 DEBUG "pod": "ds-idrepo-1" 06:58:33 DEBUG }, 06:58:33 DEBUG "value": [ 06:58:33 DEBUG 1712728713, 06:58:33 DEBUG "0" 06:58:33 DEBUG ] 06:58:33 DEBUG } 06:58:33 DEBUG ] 06:58:33 DEBUG } 06:58:33 DEBUG } 06:58:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728714" 06:58:34 INFO [http_cmd]: http status code OK 06:58:34 DEBUG --- status code --- 06:58:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:34 DEBUG --- http response --- 06:58:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:58:34 DEBUG ----- output ----- 06:58:34 DEBUG { 06:58:34 DEBUG "status": "success", 06:58:34 DEBUG "data": { 06:58:34 DEBUG "resultType": "vector", 06:58:34 DEBUG "result": [ 06:58:34 DEBUG { 06:58:34 DEBUG "metric": { 06:58:34 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:58:34 DEBUG }, 06:58:34 DEBUG "value": [ 06:58:34 DEBUG 06:58:34 DEBUG 06:58:34 DEBUG 06:58:34 DEBUG [...] 06:58:34 DEBUG 06:58:34 DEBUG 06:58:34 DEBUG 06:58:34 DEBUG "pod": "ds-cts-0" 06:58:34 DEBUG }, 06:58:34 DEBUG "value": [ 06:58:34 DEBUG 1712728714, 06:58:34 DEBUG "14708.101214846634" 06:58:34 DEBUG ] 06:58:34 DEBUG } 06:58:34 DEBUG ] 06:58:34 DEBUG } 06:58:34 DEBUG } 06:58:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712728714" 06:58:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728714" 06:58:34 INFO [http_cmd]: http status code OK 06:58:34 DEBUG --- status code --- 06:58:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:34 DEBUG --- http response --- 06:58:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:58:34 INFO [http_cmd]: http status code OK 06:58:34 DEBUG ----- output ----- 06:58:34 DEBUG --- status code --- 06:58:34 DEBUG { 06:58:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:34 DEBUG "status": "success", 06:58:34 DEBUG --- http response --- 06:58:34 DEBUG "data": { 06:58:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:58:34 DEBUG "resultType": "vector", 06:58:34 DEBUG ----- output ----- 06:58:34 DEBUG "result": [ 06:58:34 DEBUG { 06:58:34 DEBUG { 06:58:34 DEBUG "status": "success", 06:58:34 DEBUG "metric": { 06:58:34 DEBUG "data": { 06:58:34 DEBUG "instance": "10.142.0.25:9100" 06:58:34 DEBUG "resultType": "vector", 06:58:34 DEBUG }, 06:58:34 DEBUG "result": [ 06:58:34 DEBUG "value": [ 06:58:34 DEBUG { 06:58:34 DEBUG 06:58:34 DEBUG "metric": { 06:58:34 DEBUG 06:58:34 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:58:34 DEBUG 06:58:34 DEBUG }, 06:58:34 DEBUG [...] 06:58:34 DEBUG "value": [ 06:58:34 DEBUG 06:58:34 DEBUG 06:58:34 DEBUG 06:58:34 DEBUG 06:58:34 DEBUG 06:58:34 DEBUG 06:58:34 DEBUG "instance": "10.142.0.77:9100" 06:58:34 DEBUG [...] 06:58:34 DEBUG }, 06:58:34 DEBUG 06:58:34 DEBUG "value": [ 06:58:34 DEBUG 06:58:34 DEBUG 1712728714, 06:58:34 DEBUG 06:58:34 DEBUG "25929.533333333333" 06:58:34 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:58:34 DEBUG ] 06:58:34 DEBUG }, 06:58:34 DEBUG } 06:58:34 DEBUG "value": [ 06:58:34 DEBUG ] 06:58:34 DEBUG 1712728714, 06:58:34 DEBUG } 06:58:34 DEBUG "2485981184" 06:58:34 DEBUG } 06:58:34 DEBUG ] 06:58:34 DEBUG } 06:58:34 DEBUG ] 06:58:34 DEBUG } 06:58:34 DEBUG } 06:58:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728714" 06:58:34 INFO [http_cmd]: http status code OK 06:58:34 DEBUG --- status code --- 06:58:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:34 DEBUG --- http response --- 06:58:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:58:34 DEBUG ----- output ----- 06:58:34 DEBUG { 06:58:34 DEBUG "status": "success", 06:58:34 DEBUG "data": { 06:58:34 DEBUG "resultType": "vector", 06:58:34 DEBUG "result": [ 06:58:34 DEBUG { 06:58:34 DEBUG "metric": { 06:58:34 DEBUG "instance": "10.142.0.25:9100" 06:58:34 DEBUG }, 06:58:34 DEBUG "value": [ 06:58:34 DEBUG 06:58:34 DEBUG 06:58:34 DEBUG 06:58:34 DEBUG [...] 06:58:34 DEBUG 06:58:34 DEBUG 06:58:34 DEBUG 06:58:34 DEBUG "instance": "10.142.0.77:9100" 06:58:34 DEBUG }, 06:58:34 DEBUG "value": [ 06:58:34 DEBUG 1712728714, 06:58:34 DEBUG "0" 06:58:34 DEBUG ] 06:58:34 DEBUG } 06:58:34 DEBUG ] 06:58:34 DEBUG } 06:58:34 DEBUG } 06:58:55 INFO 06:58:55 INFO [loop_until]: kubectl --namespace=xlou top pods 06:58:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:58:55 INFO [loop_until]: OK (rc = 0) 06:58:55 DEBUG --- stdout --- 06:58:55 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 9Mi am-654c4fcbcf-487cv 8m 8870Mi am-654c4fcbcf-9scrj 7m 8378Mi am-654c4fcbcf-v5qb5 8m 8096Mi ds-cts-0 17m 12723Mi ds-cts-1 17m 12921Mi ds-cts-2 15m 11473Mi ds-idrepo-0 14m 5620Mi ds-idrepo-1 11m 5677Mi ds-idrepo-2 12m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 5Mi idm-6bfb7c96cc-qklk9 5m 1198Mi idm-6bfb7c96cc-vsb58 4m 2214Mi lodemon-b776b7d88-thrqf 7m 74Mi login-ui-5f964fb894-5zpvq 1m 4Mi overseer-0-6f8784665d-zq5hb 974m 4759Mi 06:58:55 DEBUG --- stderr --- 06:58:55 DEBUG 06:59:04 INFO 06:59:04 INFO [loop_until]: kubectl --namespace=xlou top node 06:59:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:59:04 INFO [loop_until]: OK (rc = 0) 06:59:04 DEBUG --- stdout --- 06:59:04 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 94m 0% 2316Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 69m 0% 8986Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 76m 0% 3254Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 212m 1% 6537Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 66m 0% 9282Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 64m 0% 9734Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 72m 0% 859Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 68m 0% 6320Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 71m 0% 13463Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 102m 0% 12209Mi 20% gke-xlou-cdm-ds-32e4dcb1-gxzj 66m 0% 6402Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 71m 0% 13657Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 64m 0% 6369Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 1077m 6% 8506Mi 14% 06:59:04 DEBUG --- stderr --- 06:59:04 DEBUG 06:59:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728771" 06:59:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712728771" 06:59:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728771" 06:59:31 ERROR [http_cmd]: ERROR 06:59:31 INFO [http_cmd]: http status code OK 06:59:31 ERROR --- status code --- 06:59:31 DEBUG --- status code --- 06:59:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:59:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:31 ERROR --- http response --- 06:59:31 DEBUG --- http response --- 06:59:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:59:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:59:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:59:31 DEBUG ----- output ----- 06:59:31 DEBUG { 06:59:31 DEBUG "status": "success", 06:59:31 DEBUG "data": { 06:59:31 DEBUG "resultType": "vector", 06:59:31 DEBUG "result": [ 06:59:31 DEBUG { 06:59:31 DEBUG "metric": { 06:59:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:59:31 DEBUG }, 06:59:31 DEBUG "value": [ 06:59:31 DEBUG 06:59:31 DEBUG 06:59:31 DEBUG 06:59:31 DEBUG [...] 06:59:31 DEBUG 06:59:31 DEBUG 06:59:31 DEBUG 06:59:31 DEBUG "pod": "am-654c4fcbcf-487cv" 06:59:31 DEBUG }, 06:59:31 DEBUG "value": [ 06:59:31 DEBUG 1712728771, 06:59:31 DEBUG "0" 06:59:31 DEBUG ] 06:59:31 DEBUG } 06:59:31 DEBUG ] 06:59:31 DEBUG } 06:59:31 DEBUG } 06:59:31 INFO [http_cmd]: http status code OK 06:59:31 DEBUG --- status code --- 06:59:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:31 DEBUG --- http response --- 06:59:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728772" 06:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728772" 06:59:32 INFO [http_cmd]: http status code OK 06:59:32 INFO [http_cmd]: http status code OK 06:59:32 DEBUG --- status code --- 06:59:32 DEBUG --- status code --- 06:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:32 DEBUG --- http response --- 06:59:32 DEBUG --- http response --- 06:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:59:32 DEBUG ----- output ----- 06:59:32 DEBUG ----- output ----- 06:59:32 DEBUG { 06:59:32 DEBUG { 06:59:32 DEBUG "status": "success", 06:59:32 DEBUG "status": "success", 06:59:32 DEBUG "data": { 06:59:32 DEBUG "data": { 06:59:32 DEBUG "resultType": "vector", 06:59:32 DEBUG "resultType": "vector", 06:59:32 DEBUG "result": [ 06:59:32 DEBUG "result": [ 06:59:32 DEBUG { 06:59:32 DEBUG { 06:59:32 DEBUG "metric": { 06:59:32 DEBUG "metric": { 06:59:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:59:32 DEBUG "instance": "10.142.0.25:9100" 06:59:32 DEBUG }, 06:59:32 DEBUG }, 06:59:32 DEBUG "value": [ 06:59:32 DEBUG "value": [ 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG [...] 06:59:32 DEBUG [...] 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:59:32 DEBUG "instance": "10.142.0.77:9100" 06:59:32 DEBUG }, 06:59:32 DEBUG }, 06:59:32 DEBUG "value": [ 06:59:32 DEBUG "value": [ 06:59:32 DEBUG 1712728772, 06:59:32 DEBUG 1712728772, 06:59:32 DEBUG "0" 06:59:32 DEBUG "25093.233333333334" 06:59:32 DEBUG ] 06:59:32 DEBUG ] 06:59:32 DEBUG } 06:59:32 DEBUG } 06:59:32 DEBUG ] 06:59:32 DEBUG ] 06:59:32 DEBUG } 06:59:32 DEBUG } 06:59:32 DEBUG } 06:59:32 DEBUG } 06:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728772" 06:59:32 INFO [http_cmd]: http status code OK 06:59:32 DEBUG --- status code --- 06:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:32 DEBUG --- http response --- 06:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:59:32 DEBUG ----- output ----- 06:59:32 DEBUG { 06:59:32 DEBUG "status": "success", 06:59:32 DEBUG "data": { 06:59:32 DEBUG "resultType": "vector", 06:59:32 DEBUG "result": [ 06:59:32 DEBUG { 06:59:32 DEBUG "metric": { 06:59:32 DEBUG "pod": "ds-idrepo-0" 06:59:32 DEBUG }, 06:59:32 DEBUG "value": [ 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG [...] 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG "pod": "ds-idrepo-1" 06:59:32 DEBUG }, 06:59:32 DEBUG "value": [ 06:59:32 DEBUG 1712728772, 06:59:32 DEBUG "0" 06:59:32 DEBUG ] 06:59:32 DEBUG } 06:59:32 DEBUG ] 06:59:32 DEBUG } 06:59:32 DEBUG } 06:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728772" 06:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728772" 06:59:32 INFO [http_cmd]: http status code OK 06:59:32 DEBUG --- status code --- 06:59:32 INFO [http_cmd]: http status code OK 06:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:32 DEBUG --- status code --- 06:59:32 DEBUG --- http response --- 06:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:59:32 DEBUG --- http response --- 06:59:32 DEBUG ----- output ----- 06:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:59:32 DEBUG { 06:59:32 DEBUG ----- output ----- 06:59:32 DEBUG "status": "success", 06:59:32 DEBUG { 06:59:32 DEBUG "data": { 06:59:32 DEBUG "status": "success", 06:59:32 DEBUG "resultType": "vector", 06:59:32 DEBUG "data": { 06:59:32 DEBUG "result": [ 06:59:32 DEBUG "resultType": "vector", 06:59:32 DEBUG { 06:59:32 DEBUG "result": [ 06:59:32 DEBUG "metric": { 06:59:32 DEBUG { 06:59:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:59:32 DEBUG "metric": { 06:59:32 DEBUG }, 06:59:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:59:32 DEBUG "value": [ 06:59:32 DEBUG }, 06:59:32 DEBUG 06:59:32 DEBUG "value": [ 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG [...] 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG [...] 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:59:32 DEBUG 06:59:32 DEBUG }, 06:59:32 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:59:32 DEBUG "value": [ 06:59:32 DEBUG }, 06:59:32 DEBUG 1712728772, 06:59:32 DEBUG "value": [ 06:59:32 DEBUG "0" 06:59:32 DEBUG 1712728772, 06:59:32 DEBUG ] 06:59:32 DEBUG "0" 06:59:32 DEBUG } 06:59:32 DEBUG ] 06:59:32 DEBUG ] 06:59:32 DEBUG } 06:59:32 DEBUG } 06:59:32 DEBUG ] 06:59:32 DEBUG } 06:59:32 DEBUG } 06:59:32 DEBUG } 06:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728772" 06:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728772" 06:59:32 INFO [http_cmd]: http status code OK 06:59:32 DEBUG --- status code --- 06:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:32 DEBUG --- http response --- 06:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:59:32 DEBUG ----- output ----- 06:59:32 DEBUG { 06:59:32 DEBUG "status": "success", 06:59:32 DEBUG "data": { 06:59:32 DEBUG "resultType": "vector", 06:59:32 DEBUG "result": [ 06:59:32 DEBUG { 06:59:32 DEBUG "metric": { 06:59:32 DEBUG "pod": "ds-idrepo-0" 06:59:32 DEBUG }, 06:59:32 DEBUG "value": [ 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG [...] 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG "pod": "ds-cts-2" 06:59:32 DEBUG }, 06:59:32 DEBUG "value": [ 06:59:32 DEBUG 1712728772, 06:59:32 DEBUG "0" 06:59:32 DEBUG ] 06:59:32 DEBUG } 06:59:32 DEBUG ] 06:59:32 DEBUG } 06:59:32 DEBUG } 06:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712728772" 06:59:32 INFO [http_cmd]: http status code OK 06:59:32 DEBUG --- status code --- 06:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:32 DEBUG --- http response --- 06:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:59:32 DEBUG ----- output ----- 06:59:32 DEBUG { 06:59:32 DEBUG "status": "success", 06:59:32 DEBUG "data": { 06:59:32 DEBUG "resultType": "vector", 06:59:32 DEBUG "result": [ 06:59:32 DEBUG { 06:59:32 DEBUG "metric": { 06:59:32 DEBUG "instance": "10.142.0.25:9100" 06:59:32 DEBUG }, 06:59:32 DEBUG "value": [ 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG [...] 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG "instance": "10.142.0.77:9100" 06:59:32 DEBUG }, 06:59:32 DEBUG "value": [ 06:59:32 DEBUG 1712728772, 06:59:32 DEBUG "438091895808" 06:59:32 DEBUG ] 06:59:32 DEBUG } 06:59:32 DEBUG ] 06:59:32 DEBUG } 06:59:32 DEBUG } 06:59:32 INFO [http_cmd]: http status code OK 06:59:32 DEBUG --- status code --- 06:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:32 DEBUG --- http response --- 06:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:59:32 DEBUG ----- output ----- 06:59:32 DEBUG { 06:59:32 DEBUG "status": "success", 06:59:32 DEBUG "data": { 06:59:32 DEBUG "resultType": "vector", 06:59:32 DEBUG "result": [ 06:59:32 DEBUG { 06:59:32 DEBUG "metric": { 06:59:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:59:32 DEBUG }, 06:59:32 DEBUG "value": [ 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG [...] 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:59:32 DEBUG }, 06:59:32 DEBUG "value": [ 06:59:32 DEBUG 1712728772, 06:59:32 DEBUG "1.0359496729405133" 06:59:32 DEBUG ] 06:59:32 DEBUG } 06:59:32 DEBUG ] 06:59:32 DEBUG } 06:59:32 DEBUG } 06:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728772" 06:59:32 INFO [http_cmd]: http status code OK 06:59:32 DEBUG --- status code --- 06:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:32 DEBUG --- http response --- 06:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:59:32 DEBUG ----- output ----- 06:59:32 DEBUG { 06:59:32 DEBUG "status": "success", 06:59:32 DEBUG "data": { 06:59:32 DEBUG "resultType": "vector", 06:59:32 DEBUG "result": [ 06:59:32 DEBUG { 06:59:32 DEBUG "metric": { 06:59:32 DEBUG "instance": "10.142.0.25:9100" 06:59:32 DEBUG }, 06:59:32 DEBUG "value": [ 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG [...] 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG "instance": "10.142.0.77:9100" 06:59:32 DEBUG }, 06:59:32 DEBUG "value": [ 06:59:32 DEBUG 1712728772, 06:59:32 DEBUG "0.0007000000000042899" 06:59:32 DEBUG ] 06:59:32 DEBUG } 06:59:32 DEBUG ] 06:59:32 DEBUG } 06:59:32 DEBUG } 06:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728772" 06:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728772" 06:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728772" 06:59:32 INFO [http_cmd]: http status code OK 06:59:32 DEBUG --- status code --- 06:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:32 INFO [http_cmd]: http status code OK 06:59:32 DEBUG --- http response --- 06:59:32 DEBUG --- status code --- 06:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:59:32 DEBUG --- http response --- 06:59:32 DEBUG ----- output ----- 06:59:32 DEBUG { 06:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:59:32 INFO [http_cmd]: http status code OK 06:59:32 DEBUG "status": "success", 06:59:32 DEBUG ----- output ----- 06:59:32 DEBUG --- status code --- 06:59:32 DEBUG "data": { 06:59:32 DEBUG { 06:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:32 DEBUG "resultType": "vector", 06:59:32 DEBUG "status": "success", 06:59:32 DEBUG --- http response --- 06:59:32 DEBUG "result": [ 06:59:32 DEBUG "data": { 06:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:59:32 DEBUG { 06:59:32 DEBUG "resultType": "vector", 06:59:32 DEBUG ----- output ----- 06:59:32 DEBUG "metric": { 06:59:32 DEBUG "result": [ 06:59:32 DEBUG { 06:59:32 DEBUG "pod": "ds-idrepo-0" 06:59:32 DEBUG { 06:59:32 DEBUG "status": "success", 06:59:32 DEBUG }, 06:59:32 DEBUG "metric": { 06:59:32 DEBUG "data": { 06:59:32 DEBUG "value": [ 06:59:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:59:32 DEBUG "resultType": "vector", 06:59:32 DEBUG 06:59:32 DEBUG }, 06:59:32 DEBUG "result": [ 06:59:32 DEBUG 06:59:32 DEBUG "value": [ 06:59:32 DEBUG { 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG "metric": { 06:59:32 DEBUG [...] 06:59:32 DEBUG 06:59:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG }, 06:59:32 DEBUG 06:59:32 DEBUG [...] 06:59:32 DEBUG "value": [ 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG "pod": "ds-cts-2" 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG }, 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG "value": [ 06:59:32 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 06:59:32 DEBUG [...] 06:59:32 DEBUG 1712728772, 06:59:32 DEBUG }, 06:59:32 DEBUG 06:59:32 DEBUG "27.633333333333333" 06:59:32 DEBUG "value": [ 06:59:32 DEBUG 06:59:32 DEBUG ] 06:59:32 DEBUG 1712728772, 06:59:32 DEBUG 06:59:32 DEBUG } 06:59:32 DEBUG "28896293.30612512" 06:59:32 DEBUG "pod": "am-654c4fcbcf-487cv" 06:59:32 DEBUG ] 06:59:32 DEBUG ] 06:59:32 DEBUG }, 06:59:32 DEBUG } 06:59:32 DEBUG } 06:59:32 DEBUG "value": [ 06:59:32 DEBUG } 06:59:32 DEBUG ] 06:59:32 DEBUG 1712728772, 06:59:32 DEBUG } 06:59:32 DEBUG "0" 06:59:32 DEBUG } 06:59:32 DEBUG ] 06:59:32 DEBUG } 06:59:32 DEBUG ] 06:59:32 DEBUG } 06:59:32 DEBUG } 06:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712728772" 06:59:32 INFO [http_cmd]: http status code OK 06:59:32 DEBUG --- status code --- 06:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:32 DEBUG --- http response --- 06:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:59:32 DEBUG ----- output ----- 06:59:32 DEBUG { 06:59:32 DEBUG "status": "success", 06:59:32 DEBUG "data": { 06:59:32 DEBUG "resultType": "vector", 06:59:32 DEBUG "result": [ 06:59:32 DEBUG { 06:59:32 DEBUG "metric": { 06:59:32 DEBUG "pod": "ds-idrepo-0" 06:59:32 DEBUG }, 06:59:32 DEBUG "value": [ 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG [...] 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG 06:59:32 DEBUG "pod": "ds-cts-2" 06:59:32 DEBUG }, 06:59:32 DEBUG "value": [ 06:59:32 DEBUG 1712728772, 06:59:32 DEBUG "0" 06:59:32 DEBUG ] 06:59:32 DEBUG } 06:59:32 DEBUG ] 06:59:32 DEBUG } 06:59:32 DEBUG } 06:59:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728773" 06:59:33 INFO [http_cmd]: http status code OK 06:59:33 DEBUG --- status code --- 06:59:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:33 DEBUG --- http response --- 06:59:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:59:33 DEBUG ----- output ----- 06:59:33 DEBUG { 06:59:33 DEBUG "status": "success", 06:59:33 DEBUG "data": { 06:59:33 DEBUG "resultType": "vector", 06:59:33 DEBUG "result": [ 06:59:33 DEBUG { 06:59:33 DEBUG "metric": { 06:59:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:59:33 DEBUG }, 06:59:33 DEBUG "value": [ 06:59:33 DEBUG 06:59:33 DEBUG 06:59:33 DEBUG 06:59:33 DEBUG [...] 06:59:33 DEBUG 06:59:33 DEBUG 06:59:33 DEBUG 06:59:33 DEBUG "pod": "am-654c4fcbcf-487cv" 06:59:33 DEBUG }, 06:59:33 DEBUG "value": [ 06:59:33 DEBUG 1712728773, 06:59:33 DEBUG "0" 06:59:33 DEBUG ] 06:59:33 DEBUG } 06:59:33 DEBUG ] 06:59:33 DEBUG } 06:59:33 DEBUG } 06:59:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712728773" 06:59:33 INFO [http_cmd]: http status code OK 06:59:33 DEBUG --- status code --- 06:59:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:33 DEBUG --- http response --- 06:59:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:59:33 DEBUG ----- output ----- 06:59:33 DEBUG { 06:59:33 DEBUG "status": "success", 06:59:33 DEBUG "data": { 06:59:33 DEBUG "resultType": "vector", 06:59:33 DEBUG "result": [ 06:59:33 DEBUG { 06:59:33 DEBUG "metric": { 06:59:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:59:33 DEBUG }, 06:59:33 DEBUG "value": [ 06:59:33 DEBUG 06:59:33 DEBUG 06:59:33 DEBUG 06:59:33 DEBUG [...] 06:59:33 DEBUG 06:59:33 DEBUG 06:59:33 DEBUG 06:59:33 DEBUG "pod": "am-654c4fcbcf-487cv" 06:59:33 DEBUG }, 06:59:33 DEBUG "value": [ 06:59:33 DEBUG 1712728773, 06:59:33 DEBUG "0.054" 06:59:33 DEBUG ] 06:59:33 DEBUG } 06:59:33 DEBUG ] 06:59:33 DEBUG } 06:59:33 DEBUG } 06:59:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728773" 06:59:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728773" 06:59:33 INFO [http_cmd]: http status code OK 06:59:33 DEBUG --- status code --- 06:59:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:33 DEBUG --- http response --- 06:59:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:59:33 DEBUG ----- output ----- 06:59:33 DEBUG { 06:59:33 DEBUG "status": "success", 06:59:33 DEBUG "data": { 06:59:33 DEBUG "resultType": "vector", 06:59:33 DEBUG "result": [ 06:59:33 DEBUG { 06:59:33 DEBUG "metric": { 06:59:33 DEBUG "pod": "ds-idrepo-2" 06:59:33 DEBUG }, 06:59:33 DEBUG "value": [ 06:59:33 DEBUG 06:59:33 DEBUG 06:59:33 DEBUG 06:59:33 DEBUG [...] 06:59:33 DEBUG 06:59:33 DEBUG 06:59:33 DEBUG 06:59:33 DEBUG "pod": "ds-cts-2" 06:59:33 DEBUG }, 06:59:33 DEBUG "value": [ 06:59:33 DEBUG 1712728773, 06:59:33 DEBUG "0" 06:59:33 DEBUG ] 06:59:33 DEBUG } 06:59:33 DEBUG ] 06:59:33 DEBUG } 06:59:33 DEBUG } 06:59:33 INFO [http_cmd]: http status code OK 06:59:33 DEBUG --- status code --- 06:59:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:33 DEBUG --- http response --- 06:59:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:59:33 DEBUG ----- output ----- 06:59:33 DEBUG { 06:59:33 DEBUG "status": "success", 06:59:33 DEBUG "data": { 06:59:33 DEBUG "resultType": "vector", 06:59:33 DEBUG "result": [ 06:59:33 DEBUG { 06:59:33 DEBUG "metric": { 06:59:33 DEBUG "pod": "ds-cts-2" 06:59:33 DEBUG }, 06:59:33 DEBUG "value": [ 06:59:33 DEBUG 06:59:33 DEBUG 06:59:33 DEBUG 06:59:33 DEBUG [...] 06:59:33 DEBUG 06:59:33 DEBUG 06:59:33 DEBUG 06:59:33 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 06:59:33 DEBUG }, 06:59:33 DEBUG "value": [ 06:59:33 DEBUG 1712728773, 06:59:33 DEBUG "0.000028119607099998016" 06:59:33 DEBUG ] 06:59:33 DEBUG } 06:59:33 DEBUG ] 06:59:33 DEBUG } 06:59:33 DEBUG } 06:59:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728773" 06:59:33 INFO [http_cmd]: http status code OK 06:59:33 DEBUG --- status code --- 06:59:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:33 DEBUG --- http response --- 06:59:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:59:33 DEBUG ----- output ----- 06:59:33 DEBUG { 06:59:33 DEBUG "status": "success", 06:59:33 DEBUG "data": { 06:59:33 DEBUG "resultType": "vector", 06:59:33 DEBUG "result": [ 06:59:33 DEBUG { 06:59:33 DEBUG "metric": { 06:59:33 DEBUG "pod": "ds-idrepo-0" 06:59:33 DEBUG }, 06:59:33 DEBUG "value": [ 06:59:33 DEBUG 06:59:33 DEBUG 06:59:33 DEBUG 06:59:33 DEBUG [...] 06:59:33 DEBUG 06:59:33 DEBUG 06:59:33 DEBUG 06:59:33 DEBUG "pod": "ds-idrepo-1" 06:59:33 DEBUG }, 06:59:33 DEBUG "value": [ 06:59:33 DEBUG 1712728773, 06:59:33 DEBUG "0" 06:59:33 DEBUG ] 06:59:33 DEBUG } 06:59:33 DEBUG ] 06:59:33 DEBUG } 06:59:33 DEBUG } 06:59:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728773" 06:59:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728773" 06:59:34 INFO [http_cmd]: http status code OK 06:59:34 INFO [http_cmd]: http status code OK 06:59:34 DEBUG --- status code --- 06:59:34 DEBUG --- status code --- 06:59:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:34 DEBUG --- http response --- 06:59:34 DEBUG --- http response --- 06:59:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:59:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:59:34 DEBUG ----- output ----- 06:59:34 DEBUG ----- output ----- 06:59:34 DEBUG { 06:59:34 DEBUG { 06:59:34 DEBUG "status": "success", 06:59:34 DEBUG "status": "success", 06:59:34 DEBUG "data": { 06:59:34 DEBUG "data": { 06:59:34 DEBUG "resultType": "vector", 06:59:34 DEBUG "resultType": "vector", 06:59:34 DEBUG "result": [ 06:59:34 DEBUG "result": [ 06:59:34 DEBUG { 06:59:34 DEBUG { 06:59:34 DEBUG "metric": { 06:59:34 DEBUG "metric": { 06:59:34 DEBUG "pod": "am-654c4fcbcf-v5qb5" 06:59:34 DEBUG "pod": "ds-idrepo-0" 06:59:34 DEBUG }, 06:59:34 DEBUG }, 06:59:34 DEBUG "value": [ 06:59:34 DEBUG "value": [ 06:59:34 DEBUG 06:59:34 DEBUG 06:59:34 DEBUG 06:59:34 DEBUG 06:59:34 DEBUG 06:59:34 DEBUG 06:59:34 DEBUG [...] 06:59:34 DEBUG [...] 06:59:34 DEBUG 06:59:34 DEBUG 06:59:34 DEBUG 06:59:34 DEBUG 06:59:34 DEBUG 06:59:34 DEBUG 06:59:34 DEBUG "pod": "am-654c4fcbcf-487cv" 06:59:34 DEBUG "pod": "ds-idrepo-1" 06:59:34 DEBUG }, 06:59:34 DEBUG }, 06:59:34 DEBUG "value": [ 06:59:34 DEBUG "value": [ 06:59:34 DEBUG 1712728773, 06:59:34 DEBUG 1712728773, 06:59:34 DEBUG "0" 06:59:34 DEBUG "0" 06:59:34 DEBUG ] 06:59:34 DEBUG ] 06:59:34 DEBUG } 06:59:34 DEBUG } 06:59:34 DEBUG ] 06:59:34 DEBUG ] 06:59:34 DEBUG } 06:59:34 DEBUG } 06:59:34 DEBUG } 06:59:34 DEBUG } 06:59:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728774" 06:59:34 INFO [http_cmd]: http status code OK 06:59:34 DEBUG --- status code --- 06:59:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:34 DEBUG --- http response --- 06:59:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:59:34 DEBUG ----- output ----- 06:59:34 DEBUG { 06:59:34 DEBUG "status": "success", 06:59:34 DEBUG "data": { 06:59:34 DEBUG "resultType": "vector", 06:59:34 DEBUG "result": [ 06:59:34 DEBUG { 06:59:34 DEBUG "metric": { 06:59:34 DEBUG "pod": "ds-cts-2" 06:59:34 DEBUG }, 06:59:34 DEBUG "value": [ 06:59:34 DEBUG 06:59:34 DEBUG 06:59:34 DEBUG 06:59:34 DEBUG [...] 06:59:34 DEBUG 06:59:34 DEBUG 06:59:34 DEBUG 06:59:34 DEBUG "pod": "ds-idrepo-0" 06:59:34 DEBUG }, 06:59:34 DEBUG "value": [ 06:59:34 DEBUG 1712728774, 06:59:34 DEBUG "3812.278503669232" 06:59:34 DEBUG ] 06:59:34 DEBUG } 06:59:34 DEBUG ] 06:59:34 DEBUG } 06:59:34 DEBUG } 06:59:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712728774" 06:59:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728774" 06:59:34 INFO [http_cmd]: http status code OK 06:59:34 DEBUG --- status code --- 06:59:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:34 DEBUG --- http response --- 06:59:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:59:34 DEBUG ----- output ----- 06:59:34 DEBUG { 06:59:34 DEBUG "status": "success", 06:59:34 DEBUG "data": { 06:59:34 DEBUG "resultType": "vector", 06:59:34 DEBUG "result": [ 06:59:34 DEBUG { 06:59:34 DEBUG "metric": { 06:59:34 DEBUG "instance": "10.142.0.25:9100" 06:59:34 DEBUG }, 06:59:34 DEBUG "value": [ 06:59:34 DEBUG 06:59:34 DEBUG 06:59:34 DEBUG 06:59:34 DEBUG [...] 06:59:34 DEBUG 06:59:34 DEBUG 06:59:34 DEBUG 06:59:34 DEBUG "instance": "10.142.0.77:9100" 06:59:34 DEBUG }, 06:59:34 DEBUG "value": [ 06:59:34 DEBUG 1712728774, 06:59:34 DEBUG "26367.8" 06:59:34 DEBUG ] 06:59:34 DEBUG } 06:59:34 DEBUG ] 06:59:34 DEBUG } 06:59:34 DEBUG } 06:59:34 INFO [http_cmd]: http status code OK 06:59:34 DEBUG --- status code --- 06:59:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:34 DEBUG --- http response --- 06:59:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:59:34 DEBUG ----- output ----- 06:59:34 DEBUG { 06:59:34 DEBUG "status": "success", 06:59:34 DEBUG "data": { 06:59:34 DEBUG "resultType": "vector", 06:59:34 DEBUG "result": [ 06:59:34 DEBUG { 06:59:34 DEBUG "metric": { 06:59:34 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 06:59:34 DEBUG }, 06:59:34 DEBUG "value": [ 06:59:34 DEBUG 06:59:34 DEBUG 06:59:34 DEBUG 06:59:34 DEBUG [...] 06:59:34 DEBUG 06:59:34 DEBUG 06:59:34 DEBUG 06:59:34 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 06:59:34 DEBUG }, 06:59:34 DEBUG "value": [ 06:59:34 DEBUG 1712728774, 06:59:34 DEBUG "2488393728" 06:59:34 DEBUG ] 06:59:34 DEBUG } 06:59:34 DEBUG ] 06:59:34 DEBUG } 06:59:34 DEBUG } 06:59:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728774" 06:59:34 INFO [http_cmd]: http status code OK 06:59:34 DEBUG --- status code --- 06:59:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:34 DEBUG --- http response --- 06:59:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:59:34 DEBUG ----- output ----- 06:59:34 DEBUG { 06:59:34 DEBUG "status": "success", 06:59:34 DEBUG "data": { 06:59:34 DEBUG "resultType": "vector", 06:59:34 DEBUG "result": [ 06:59:34 DEBUG { 06:59:34 DEBUG "metric": { 06:59:34 DEBUG "instance": "10.142.0.25:9100" 06:59:34 DEBUG }, 06:59:34 DEBUG "value": [ 06:59:34 DEBUG 06:59:34 DEBUG 06:59:34 DEBUG 06:59:34 DEBUG [...] 06:59:34 DEBUG 06:59:34 DEBUG 06:59:34 DEBUG 06:59:34 DEBUG "instance": "10.142.0.77:9100" 06:59:34 DEBUG }, 06:59:34 DEBUG "value": [ 06:59:34 DEBUG 1712728774, 06:59:34 DEBUG "0" 06:59:34 DEBUG ] 06:59:34 DEBUG } 06:59:34 DEBUG ] 06:59:34 DEBUG } 06:59:34 DEBUG } 06:59:55 INFO 06:59:55 INFO [loop_until]: kubectl --namespace=xlou top pods 06:59:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:59:55 INFO [loop_until]: OK (rc = 0) 06:59:55 DEBUG --- stdout --- 06:59:55 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 9Mi am-654c4fcbcf-487cv 7m 8870Mi am-654c4fcbcf-9scrj 8m 8378Mi am-654c4fcbcf-v5qb5 8m 8096Mi ds-cts-0 17m 12723Mi ds-cts-1 22m 12921Mi ds-cts-2 14m 11474Mi ds-idrepo-0 14m 5620Mi ds-idrepo-1 11m 5677Mi ds-idrepo-2 12m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 5Mi idm-6bfb7c96cc-qklk9 5m 1198Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 7m 74Mi login-ui-5f964fb894-5zpvq 1m 4Mi overseer-0-6f8784665d-zq5hb 1020m 5712Mi 06:59:55 DEBUG --- stderr --- 06:59:55 DEBUG 07:00:04 INFO 07:00:04 INFO [loop_until]: kubectl --namespace=xlou top node 07:00:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:00:04 INFO [loop_until]: OK (rc = 0) 07:00:04 DEBUG --- stdout --- 07:00:04 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 97m 0% 2318Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 67m 0% 8989Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 76m 0% 3263Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 208m 1% 6516Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 71m 0% 9278Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 65m 0% 9735Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 76m 0% 855Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 68m 0% 6322Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 75m 0% 13460Mi 22% gke-xlou-cdm-ds-32e4dcb1-4hls 82m 0% 12211Mi 20% gke-xlou-cdm-ds-32e4dcb1-gxzj 64m 0% 6403Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 76m 0% 13659Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 64m 0% 6372Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 1075m 6% 9337Mi 15% 07:00:04 DEBUG --- stderr --- 07:00:04 DEBUG 07:00:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728831" 07:00:31 INFO [http_cmd]: http status code OK 07:00:31 DEBUG --- status code --- 07:00:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:31 DEBUG --- http response --- 07:00:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:00:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728831" 07:00:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712728831" 07:00:31 ERROR [http_cmd]: ERROR 07:00:31 INFO [http_cmd]: http status code OK 07:00:31 ERROR --- status code --- 07:00:31 DEBUG --- status code --- 07:00:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:00:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:31 ERROR --- http response --- 07:00:31 DEBUG --- http response --- 07:00:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:00:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:00:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:00:31 DEBUG ----- output ----- 07:00:31 DEBUG { 07:00:31 DEBUG "status": "success", 07:00:31 DEBUG "data": { 07:00:31 DEBUG "resultType": "vector", 07:00:31 DEBUG "result": [ 07:00:31 DEBUG { 07:00:31 DEBUG "metric": { 07:00:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 07:00:31 DEBUG }, 07:00:31 DEBUG "value": [ 07:00:31 DEBUG 07:00:31 DEBUG 07:00:31 DEBUG 07:00:31 DEBUG [...] 07:00:31 DEBUG 07:00:31 DEBUG 07:00:31 DEBUG 07:00:31 DEBUG "pod": "am-654c4fcbcf-487cv" 07:00:31 DEBUG }, 07:00:31 DEBUG "value": [ 07:00:31 DEBUG 1712728831, 07:00:31 DEBUG "0" 07:00:31 DEBUG ] 07:00:31 DEBUG } 07:00:31 DEBUG ] 07:00:31 DEBUG } 07:00:31 DEBUG } 07:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728832" 07:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728832" 07:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728832" 07:00:32 INFO [http_cmd]: http status code OK 07:00:32 INFO [http_cmd]: http status code OK 07:00:32 INFO [http_cmd]: http status code OK 07:00:32 DEBUG --- status code --- 07:00:32 DEBUG --- status code --- 07:00:32 DEBUG --- status code --- 07:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:32 DEBUG --- http response --- 07:00:32 DEBUG --- http response --- 07:00:32 DEBUG --- http response --- 07:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:00:32 DEBUG ----- output ----- 07:00:32 DEBUG ----- output ----- 07:00:32 DEBUG ----- output ----- 07:00:32 DEBUG { 07:00:32 DEBUG { 07:00:32 DEBUG { 07:00:32 DEBUG "status": "success", 07:00:32 DEBUG "status": "success", 07:00:32 DEBUG "status": "success", 07:00:32 DEBUG "data": { 07:00:32 DEBUG "data": { 07:00:32 DEBUG "data": { 07:00:32 DEBUG "resultType": "vector", 07:00:32 DEBUG "resultType": "vector", 07:00:32 DEBUG "resultType": "vector", 07:00:32 DEBUG "result": [ 07:00:32 DEBUG "result": [ 07:00:32 DEBUG "result": [ 07:00:32 DEBUG { 07:00:32 DEBUG { 07:00:32 DEBUG { 07:00:32 DEBUG "metric": { 07:00:32 DEBUG "metric": { 07:00:32 DEBUG "metric": { 07:00:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 07:00:32 DEBUG "pod": "ds-idrepo-0" 07:00:32 DEBUG "instance": "10.142.0.25:9100" 07:00:32 DEBUG }, 07:00:32 DEBUG }, 07:00:32 DEBUG }, 07:00:32 DEBUG "value": [ 07:00:32 DEBUG "value": [ 07:00:32 DEBUG "value": [ 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG [...] 07:00:32 DEBUG [...] 07:00:32 DEBUG [...] 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG "pod": "am-654c4fcbcf-487cv" 07:00:32 DEBUG "pod": "ds-idrepo-1" 07:00:32 DEBUG "instance": "10.142.0.77:9100" 07:00:32 DEBUG }, 07:00:32 DEBUG }, 07:00:32 DEBUG }, 07:00:32 DEBUG "value": [ 07:00:32 DEBUG "value": [ 07:00:32 DEBUG "value": [ 07:00:32 DEBUG 1712728832, 07:00:32 DEBUG 1712728832, 07:00:32 DEBUG 1712728832, 07:00:32 DEBUG "0" 07:00:32 DEBUG "0" 07:00:32 DEBUG "25093.766666666666" 07:00:32 DEBUG ] 07:00:32 DEBUG ] 07:00:32 DEBUG ] 07:00:32 DEBUG } 07:00:32 DEBUG } 07:00:32 DEBUG } 07:00:32 DEBUG ] 07:00:32 DEBUG ] 07:00:32 DEBUG ] 07:00:32 DEBUG } 07:00:32 DEBUG } 07:00:32 DEBUG } 07:00:32 DEBUG } 07:00:32 DEBUG } 07:00:32 DEBUG } 07:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728832" 07:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728832" 07:00:32 INFO [http_cmd]: http status code OK 07:00:32 DEBUG --- status code --- 07:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:32 DEBUG --- http response --- 07:00:32 INFO [http_cmd]: http status code OK 07:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:00:32 DEBUG --- status code --- 07:00:32 DEBUG ----- output ----- 07:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:32 DEBUG { 07:00:32 DEBUG --- http response --- 07:00:32 DEBUG "status": "success", 07:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 07:00:32 DEBUG "data": { 07:00:32 DEBUG ----- output ----- 07:00:32 DEBUG "resultType": "vector", 07:00:32 DEBUG { 07:00:32 DEBUG "result": [ 07:00:32 DEBUG "status": "success", 07:00:32 DEBUG { 07:00:32 DEBUG "data": { 07:00:32 DEBUG "metric": { 07:00:32 DEBUG "resultType": "vector", 07:00:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 07:00:32 DEBUG "result": [ 07:00:32 DEBUG }, 07:00:32 DEBUG { 07:00:32 DEBUG "value": [ 07:00:32 DEBUG "metric": { 07:00:32 DEBUG 07:00:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 07:00:32 DEBUG 07:00:32 DEBUG }, 07:00:32 DEBUG 07:00:32 DEBUG "value": [ 07:00:32 DEBUG [...] 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG [...] 07:00:32 DEBUG "pod": "am-654c4fcbcf-487cv" 07:00:32 DEBUG 07:00:32 DEBUG }, 07:00:32 DEBUG 07:00:32 DEBUG "value": [ 07:00:32 DEBUG 07:00:32 DEBUG 1712728832, 07:00:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 07:00:32 DEBUG "0" 07:00:32 DEBUG }, 07:00:32 DEBUG ] 07:00:32 DEBUG "value": [ 07:00:32 DEBUG } 07:00:32 DEBUG 1712728832, 07:00:32 DEBUG ] 07:00:32 DEBUG "0" 07:00:32 DEBUG } 07:00:32 DEBUG ] 07:00:32 DEBUG } 07:00:32 DEBUG } 07:00:32 DEBUG ] 07:00:32 DEBUG } 07:00:32 DEBUG } 07:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728832" 07:00:32 INFO [http_cmd]: http status code OK 07:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728832" 07:00:32 DEBUG --- status code --- 07:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:32 DEBUG --- http response --- 07:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:00:32 DEBUG ----- output ----- 07:00:32 DEBUG { 07:00:32 DEBUG "status": "success", 07:00:32 DEBUG "data": { 07:00:32 DEBUG "resultType": "vector", 07:00:32 DEBUG "result": [ 07:00:32 DEBUG { 07:00:32 DEBUG "metric": { 07:00:32 DEBUG "instance": "10.142.0.25:9100" 07:00:32 DEBUG }, 07:00:32 DEBUG "value": [ 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG [...] 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG "instance": "10.142.0.77:9100" 07:00:32 DEBUG }, 07:00:32 DEBUG "value": [ 07:00:32 DEBUG 1712728832, 07:00:32 DEBUG "438093530112" 07:00:32 DEBUG ] 07:00:32 DEBUG } 07:00:32 DEBUG ] 07:00:32 DEBUG } 07:00:32 DEBUG } 07:00:32 INFO [http_cmd]: http status code OK 07:00:32 DEBUG --- status code --- 07:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:32 DEBUG --- http response --- 07:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:00:32 DEBUG ----- output ----- 07:00:32 DEBUG { 07:00:32 DEBUG "status": "success", 07:00:32 DEBUG "data": { 07:00:32 DEBUG "resultType": "vector", 07:00:32 DEBUG "result": [ 07:00:32 DEBUG { 07:00:32 DEBUG "metric": { 07:00:32 DEBUG "pod": "ds-idrepo-0" 07:00:32 DEBUG }, 07:00:32 DEBUG "value": [ 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG [...] 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG "pod": "ds-cts-2" 07:00:32 DEBUG }, 07:00:32 DEBUG "value": [ 07:00:32 DEBUG 1712728832, 07:00:32 DEBUG "0" 07:00:32 DEBUG ] 07:00:32 DEBUG } 07:00:32 DEBUG ] 07:00:32 DEBUG } 07:00:32 DEBUG } 07:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712728832" 07:00:32 INFO [http_cmd]: http status code OK 07:00:32 DEBUG --- status code --- 07:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:32 DEBUG --- http response --- 07:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:00:32 DEBUG ----- output ----- 07:00:32 DEBUG { 07:00:32 DEBUG "status": "success", 07:00:32 DEBUG "data": { 07:00:32 DEBUG "resultType": "vector", 07:00:32 DEBUG "result": [ 07:00:32 DEBUG { 07:00:32 DEBUG "metric": { 07:00:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 07:00:32 DEBUG }, 07:00:32 DEBUG "value": [ 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG [...] 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:00:32 DEBUG }, 07:00:32 DEBUG "value": [ 07:00:32 DEBUG 1712728832, 07:00:32 DEBUG "0.7202237748465441" 07:00:32 DEBUG ] 07:00:32 DEBUG } 07:00:32 DEBUG ] 07:00:32 DEBUG } 07:00:32 DEBUG } 07:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728832" 07:00:32 INFO [http_cmd]: http status code OK 07:00:32 DEBUG --- status code --- 07:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:32 DEBUG --- http response --- 07:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:00:32 DEBUG ----- output ----- 07:00:32 DEBUG { 07:00:32 DEBUG "status": "success", 07:00:32 DEBUG "data": { 07:00:32 DEBUG "resultType": "vector", 07:00:32 DEBUG "result": [ 07:00:32 DEBUG { 07:00:32 DEBUG "metric": { 07:00:32 DEBUG "instance": "10.142.0.25:9100" 07:00:32 DEBUG }, 07:00:32 DEBUG "value": [ 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG [...] 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG "instance": "10.142.0.77:9100" 07:00:32 DEBUG }, 07:00:32 DEBUG "value": [ 07:00:32 DEBUG 1712728832, 07:00:32 DEBUG "0.0007666666666648799" 07:00:32 DEBUG ] 07:00:32 DEBUG } 07:00:32 DEBUG ] 07:00:32 DEBUG } 07:00:32 DEBUG } 07:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728832" 07:00:32 INFO [http_cmd]: http status code OK 07:00:32 DEBUG --- status code --- 07:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:32 DEBUG --- http response --- 07:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:00:32 DEBUG ----- output ----- 07:00:32 DEBUG { 07:00:32 DEBUG "status": "success", 07:00:32 DEBUG "data": { 07:00:32 DEBUG "resultType": "vector", 07:00:32 DEBUG "result": [ 07:00:32 DEBUG { 07:00:32 DEBUG "metric": { 07:00:32 DEBUG "pod": "ds-idrepo-0" 07:00:32 DEBUG }, 07:00:32 DEBUG "value": [ 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG [...] 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG "pod": "ds-cts-2" 07:00:32 DEBUG }, 07:00:32 DEBUG "value": [ 07:00:32 DEBUG 1712728832, 07:00:32 DEBUG "0" 07:00:32 DEBUG ] 07:00:32 DEBUG } 07:00:32 DEBUG ] 07:00:32 DEBUG } 07:00:32 DEBUG } 07:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728832" 07:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728832" 07:00:32 INFO [http_cmd]: http status code OK 07:00:32 DEBUG --- status code --- 07:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:32 DEBUG --- http response --- 07:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:00:32 INFO [http_cmd]: http status code OK 07:00:32 DEBUG ----- output ----- 07:00:32 DEBUG --- status code --- 07:00:32 DEBUG { 07:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:32 DEBUG "status": "success", 07:00:32 DEBUG --- http response --- 07:00:32 DEBUG "data": { 07:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 07:00:32 DEBUG "resultType": "vector", 07:00:32 DEBUG ----- output ----- 07:00:32 DEBUG "result": [ 07:00:32 DEBUG { 07:00:32 DEBUG { 07:00:32 DEBUG "status": "success", 07:00:32 DEBUG "metric": { 07:00:32 DEBUG "data": { 07:00:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 07:00:32 DEBUG "resultType": "vector", 07:00:32 DEBUG }, 07:00:32 DEBUG "result": [ 07:00:32 DEBUG "value": [ 07:00:32 DEBUG { 07:00:32 DEBUG 07:00:32 DEBUG "metric": { 07:00:32 DEBUG 07:00:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 07:00:32 DEBUG 07:00:32 DEBUG }, 07:00:32 DEBUG [...] 07:00:32 DEBUG "value": [ 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG 07:00:32 DEBUG "pod": "am-654c4fcbcf-487cv" 07:00:32 DEBUG [...] 07:00:32 DEBUG }, 07:00:32 DEBUG 07:00:32 DEBUG "value": [ 07:00:32 DEBUG 07:00:32 DEBUG 1712728832, 07:00:32 DEBUG 07:00:32 DEBUG "0" 07:00:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 07:00:32 DEBUG ] 07:00:32 DEBUG }, 07:00:32 DEBUG } 07:00:32 DEBUG "value": [ 07:00:32 DEBUG ] 07:00:32 DEBUG 1712728832, 07:00:32 DEBUG } 07:00:32 DEBUG "0" 07:00:32 DEBUG } 07:00:32 DEBUG ] 07:00:32 DEBUG } 07:00:32 DEBUG ] 07:00:32 DEBUG } 07:00:32 DEBUG } 07:00:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712728833" 07:00:33 INFO [http_cmd]: http status code OK 07:00:33 DEBUG --- status code --- 07:00:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:33 DEBUG --- http response --- 07:00:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:00:33 DEBUG ----- output ----- 07:00:33 DEBUG { 07:00:33 DEBUG "status": "success", 07:00:33 DEBUG "data": { 07:00:33 DEBUG "resultType": "vector", 07:00:33 DEBUG "result": [ 07:00:33 DEBUG { 07:00:33 DEBUG "metric": { 07:00:33 DEBUG "pod": "ds-idrepo-0" 07:00:33 DEBUG }, 07:00:33 DEBUG "value": [ 07:00:33 DEBUG 07:00:33 DEBUG 07:00:33 DEBUG 07:00:33 DEBUG [...] 07:00:33 DEBUG 07:00:33 DEBUG 07:00:33 DEBUG 07:00:33 DEBUG "pod": "ds-cts-2" 07:00:33 DEBUG }, 07:00:33 DEBUG "value": [ 07:00:33 DEBUG 1712728833, 07:00:33 DEBUG "0" 07:00:33 DEBUG ] 07:00:33 DEBUG } 07:00:33 DEBUG ] 07:00:33 DEBUG } 07:00:33 DEBUG } 07:00:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728833" 07:00:33 INFO [http_cmd]: http status code OK 07:00:33 DEBUG --- status code --- 07:00:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:33 DEBUG --- http response --- 07:00:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:00:33 DEBUG ----- output ----- 07:00:33 DEBUG { 07:00:33 DEBUG "status": "success", 07:00:33 DEBUG "data": { 07:00:33 DEBUG "resultType": "vector", 07:00:33 DEBUG "result": [ 07:00:33 DEBUG { 07:00:33 DEBUG "metric": { 07:00:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 07:00:33 DEBUG }, 07:00:33 DEBUG "value": [ 07:00:33 DEBUG 07:00:33 DEBUG 07:00:33 DEBUG 07:00:33 DEBUG [...] 07:00:33 DEBUG 07:00:33 DEBUG 07:00:33 DEBUG 07:00:33 DEBUG "pod": "am-654c4fcbcf-487cv" 07:00:33 DEBUG }, 07:00:33 DEBUG "value": [ 07:00:33 DEBUG 1712728833, 07:00:33 DEBUG "0" 07:00:33 DEBUG ] 07:00:33 DEBUG } 07:00:33 DEBUG ] 07:00:33 DEBUG } 07:00:33 DEBUG } 07:00:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712728833" 07:00:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728833" 07:00:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728833" 07:00:33 INFO [http_cmd]: http status code OK 07:00:33 DEBUG --- status code --- 07:00:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:33 DEBUG --- http response --- 07:00:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:00:33 DEBUG ----- output ----- 07:00:33 DEBUG { 07:00:33 DEBUG "status": "success", 07:00:33 DEBUG "data": { 07:00:33 DEBUG "resultType": "vector", 07:00:33 DEBUG "result": [ 07:00:33 DEBUG { 07:00:33 DEBUG "metric": { 07:00:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 07:00:33 DEBUG }, 07:00:33 DEBUG "value": [ 07:00:33 DEBUG 07:00:33 DEBUG 07:00:33 DEBUG 07:00:33 DEBUG [...] 07:00:33 DEBUG 07:00:33 DEBUG 07:00:33 DEBUG 07:00:33 DEBUG "pod": "am-654c4fcbcf-487cv" 07:00:33 DEBUG }, 07:00:33 DEBUG "value": [ 07:00:33 DEBUG 1712728833, 07:00:33 DEBUG "0.054" 07:00:33 DEBUG ] 07:00:33 DEBUG } 07:00:33 DEBUG ] 07:00:33 DEBUG } 07:00:33 DEBUG } 07:00:33 INFO [http_cmd]: http status code OK 07:00:33 DEBUG --- status code --- 07:00:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:33 DEBUG --- http response --- 07:00:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:00:33 DEBUG ----- output ----- 07:00:33 DEBUG { 07:00:33 DEBUG "status": "success", 07:00:33 DEBUG "data": { 07:00:33 DEBUG "resultType": "vector", 07:00:33 DEBUG "result": [ 07:00:33 DEBUG { 07:00:33 DEBUG "metric": { 07:00:33 DEBUG "pod": "ds-idrepo-2" 07:00:33 DEBUG }, 07:00:33 DEBUG "value": [ 07:00:33 DEBUG 07:00:33 DEBUG 07:00:33 DEBUG 07:00:33 DEBUG [...] 07:00:33 DEBUG 07:00:33 DEBUG 07:00:33 DEBUG 07:00:33 DEBUG "pod": "ds-cts-2" 07:00:33 DEBUG }, 07:00:33 DEBUG "value": [ 07:00:33 DEBUG 1712728833, 07:00:33 DEBUG "0" 07:00:33 DEBUG ] 07:00:33 DEBUG } 07:00:33 DEBUG ] 07:00:33 DEBUG } 07:00:33 DEBUG } 07:00:33 INFO [http_cmd]: http status code OK 07:00:33 DEBUG --- status code --- 07:00:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:33 DEBUG --- http response --- 07:00:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 07:00:33 DEBUG ----- output ----- 07:00:33 DEBUG { 07:00:33 DEBUG "status": "success", 07:00:33 DEBUG "data": { 07:00:33 DEBUG "resultType": "vector", 07:00:33 DEBUG "result": [ 07:00:33 DEBUG { 07:00:33 DEBUG "metric": { 07:00:33 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 07:00:33 DEBUG }, 07:00:33 DEBUG "value": [ 07:00:33 DEBUG 07:00:33 DEBUG 07:00:33 DEBUG 07:00:33 DEBUG [...] 07:00:33 DEBUG 07:00:33 DEBUG 07:00:33 DEBUG 07:00:33 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 07:00:33 DEBUG }, 07:00:33 DEBUG "value": [ 07:00:33 DEBUG 1712728833, 07:00:33 DEBUG "1.710869373898697" 07:00:33 DEBUG ] 07:00:33 DEBUG } 07:00:33 DEBUG ] 07:00:33 DEBUG } 07:00:33 DEBUG } 07:00:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728833" 07:00:33 INFO [http_cmd]: http status code OK 07:00:33 DEBUG --- status code --- 07:00:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:33 DEBUG --- http response --- 07:00:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:00:33 DEBUG ----- output ----- 07:00:33 DEBUG { 07:00:33 DEBUG "status": "success", 07:00:33 DEBUG "data": { 07:00:33 DEBUG "resultType": "vector", 07:00:33 DEBUG "result": [ 07:00:33 DEBUG { 07:00:33 DEBUG "metric": { 07:00:33 DEBUG "pod": "ds-idrepo-0" 07:00:33 DEBUG }, 07:00:33 DEBUG "value": [ 07:00:33 DEBUG 07:00:33 DEBUG 07:00:33 DEBUG 07:00:33 DEBUG [...] 07:00:33 DEBUG 07:00:33 DEBUG 07:00:33 DEBUG 07:00:33 DEBUG "pod": "ds-idrepo-1" 07:00:33 DEBUG }, 07:00:33 DEBUG "value": [ 07:00:33 DEBUG 1712728833, 07:00:33 DEBUG "0" 07:00:33 DEBUG ] 07:00:33 DEBUG } 07:00:33 DEBUG ] 07:00:33 DEBUG } 07:00:33 DEBUG } 07:00:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728834" 07:00:34 INFO [http_cmd]: http status code OK 07:00:34 DEBUG --- status code --- 07:00:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:34 DEBUG --- http response --- 07:00:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:00:34 DEBUG ----- output ----- 07:00:34 DEBUG { 07:00:34 DEBUG "status": "success", 07:00:34 DEBUG "data": { 07:00:34 DEBUG "resultType": "vector", 07:00:34 DEBUG "result": [ 07:00:34 DEBUG { 07:00:34 DEBUG "metric": { 07:00:34 DEBUG "pod": "ds-idrepo-0" 07:00:34 DEBUG }, 07:00:34 DEBUG "value": [ 07:00:34 DEBUG 07:00:34 DEBUG 07:00:34 DEBUG 07:00:34 DEBUG [...] 07:00:34 DEBUG 07:00:34 DEBUG 07:00:34 DEBUG 07:00:34 DEBUG "pod": "ds-idrepo-1" 07:00:34 DEBUG }, 07:00:34 DEBUG "value": [ 07:00:34 DEBUG 1712728834, 07:00:34 DEBUG "0" 07:00:34 DEBUG ] 07:00:34 DEBUG } 07:00:34 DEBUG ] 07:00:34 DEBUG } 07:00:34 DEBUG } 07:00:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728834" 07:00:34 INFO [http_cmd]: http status code OK 07:00:34 DEBUG --- status code --- 07:00:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:34 DEBUG --- http response --- 07:00:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:00:34 DEBUG ----- output ----- 07:00:34 DEBUG { 07:00:34 DEBUG "status": "success", 07:00:34 DEBUG "data": { 07:00:34 DEBUG "resultType": "vector", 07:00:34 DEBUG "result": [ 07:00:34 DEBUG { 07:00:34 DEBUG "metric": { 07:00:34 DEBUG "pod": "am-654c4fcbcf-v5qb5" 07:00:34 DEBUG }, 07:00:34 DEBUG "value": [ 07:00:34 DEBUG 07:00:34 DEBUG 07:00:34 DEBUG 07:00:34 DEBUG [...] 07:00:34 DEBUG 07:00:34 DEBUG 07:00:34 DEBUG 07:00:34 DEBUG "pod": "am-654c4fcbcf-487cv" 07:00:34 DEBUG }, 07:00:34 DEBUG "value": [ 07:00:34 DEBUG 1712728834, 07:00:34 DEBUG "0" 07:00:34 DEBUG ] 07:00:34 DEBUG } 07:00:34 DEBUG ] 07:00:34 DEBUG } 07:00:34 DEBUG } 07:00:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728834" 07:00:34 INFO [http_cmd]: http status code OK 07:00:34 DEBUG --- status code --- 07:00:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:34 DEBUG --- http response --- 07:00:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 07:00:34 DEBUG ----- output ----- 07:00:34 DEBUG { 07:00:34 DEBUG "status": "success", 07:00:34 DEBUG "data": { 07:00:34 DEBUG "resultType": "vector", 07:00:34 DEBUG "result": [ 07:00:34 DEBUG { 07:00:34 DEBUG "metric": { 07:00:34 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 07:00:34 DEBUG }, 07:00:34 DEBUG "value": [ 07:00:34 DEBUG 07:00:34 DEBUG 07:00:34 DEBUG 07:00:34 DEBUG [...] 07:00:34 DEBUG 07:00:34 DEBUG 07:00:34 DEBUG 07:00:34 DEBUG "pod": "overseer-0-6f8784665d-zq5hb" 07:00:34 DEBUG }, 07:00:34 DEBUG "value": [ 07:00:34 DEBUG 1712728834, 07:00:34 DEBUG "6766.327064619288" 07:00:34 DEBUG ] 07:00:34 DEBUG } 07:00:34 DEBUG ] 07:00:34 DEBUG } 07:00:34 DEBUG } 07:00:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728834" 07:00:34 INFO [http_cmd]: http status code OK 07:00:34 DEBUG --- status code --- 07:00:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:34 DEBUG --- http response --- 07:00:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:00:34 DEBUG ----- output ----- 07:00:34 DEBUG { 07:00:34 DEBUG "status": "success", 07:00:34 DEBUG "data": { 07:00:34 DEBUG "resultType": "vector", 07:00:34 DEBUG "result": [ 07:00:34 DEBUG { 07:00:34 DEBUG "metric": { 07:00:34 DEBUG "instance": "10.142.0.25:9100" 07:00:34 DEBUG }, 07:00:34 DEBUG "value": [ 07:00:34 DEBUG 07:00:34 DEBUG 07:00:34 DEBUG 07:00:34 DEBUG [...] 07:00:34 DEBUG 07:00:34 DEBUG 07:00:34 DEBUG 07:00:34 DEBUG "instance": "10.142.0.77:9100" 07:00:34 DEBUG }, 07:00:34 DEBUG "value": [ 07:00:34 DEBUG 1712728834, 07:00:34 DEBUG "28169.733333333334" 07:00:34 DEBUG ] 07:00:34 DEBUG } 07:00:34 DEBUG ] 07:00:34 DEBUG } 07:00:34 DEBUG } 07:00:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712728834" 07:00:34 INFO [http_cmd]: http status code OK 07:00:34 DEBUG --- status code --- 07:00:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:34 DEBUG --- http response --- 07:00:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:00:34 DEBUG ----- output ----- 07:00:34 DEBUG { 07:00:34 DEBUG "status": "success", 07:00:34 DEBUG "data": { 07:00:34 DEBUG "resultType": "vector", 07:00:34 DEBUG "result": [ 07:00:34 DEBUG { 07:00:34 DEBUG "metric": { 07:00:34 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 07:00:34 DEBUG }, 07:00:34 DEBUG "value": [ 07:00:34 DEBUG 07:00:34 DEBUG 07:00:34 DEBUG 07:00:34 DEBUG [...] 07:00:34 DEBUG 07:00:34 DEBUG 07:00:34 DEBUG 07:00:34 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 07:00:34 DEBUG }, 07:00:34 DEBUG "value": [ 07:00:34 DEBUG 1712728834, 07:00:34 DEBUG "2485452800" 07:00:34 DEBUG ] 07:00:34 DEBUG } 07:00:34 DEBUG ] 07:00:34 DEBUG } 07:00:34 DEBUG } 07:00:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728834" 07:00:34 INFO [http_cmd]: http status code OK 07:00:34 DEBUG --- status code --- 07:00:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:34 DEBUG --- http response --- 07:00:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:00:34 DEBUG ----- output ----- 07:00:34 DEBUG { 07:00:34 DEBUG "status": "success", 07:00:34 DEBUG "data": { 07:00:34 DEBUG "resultType": "vector", 07:00:34 DEBUG "result": [ 07:00:34 DEBUG { 07:00:34 DEBUG "metric": { 07:00:34 DEBUG "instance": "10.142.0.25:9100" 07:00:34 DEBUG }, 07:00:34 DEBUG "value": [ 07:00:34 DEBUG 07:00:34 DEBUG 07:00:34 DEBUG 07:00:34 DEBUG [...] 07:00:34 DEBUG 07:00:34 DEBUG 07:00:34 DEBUG 07:00:34 DEBUG "instance": "10.142.0.77:9100" 07:00:34 DEBUG }, 07:00:34 DEBUG "value": [ 07:00:34 DEBUG 1712728834, 07:00:34 DEBUG "0" 07:00:34 DEBUG ] 07:00:34 DEBUG } 07:00:34 DEBUG ] 07:00:34 DEBUG } 07:00:34 DEBUG } 07:00:55 INFO 07:00:55 INFO [loop_until]: kubectl --namespace=xlou top pods 07:00:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:00:55 INFO [loop_until]: OK (rc = 0) 07:00:55 DEBUG --- stdout --- 07:00:55 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 9Mi am-654c4fcbcf-487cv 7m 8870Mi am-654c4fcbcf-9scrj 7m 8378Mi am-654c4fcbcf-v5qb5 8m 8096Mi ds-cts-0 12m 11768Mi ds-cts-1 13m 11965Mi ds-cts-2 14m 11474Mi ds-idrepo-0 14m 5620Mi ds-idrepo-1 11m 5677Mi ds-idrepo-2 12m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 5Mi idm-6bfb7c96cc-qklk9 3m 1198Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 6m 73Mi login-ui-5f964fb894-5zpvq 1m 4Mi overseer-0-6f8784665d-zq5hb 1115m 5957Mi 07:00:55 DEBUG --- stderr --- 07:00:55 DEBUG 07:01:04 INFO 07:01:04 INFO [loop_until]: kubectl --namespace=xlou top node 07:01:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:01:04 INFO [loop_until]: OK (rc = 0) 07:01:04 DEBUG --- stdout --- 07:01:04 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 95m 0% 2318Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 69m 0% 8989Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 76m 0% 3254Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 208m 1% 6533Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 66m 0% 9281Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 68m 0% 9733Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 79m 0% 862Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 66m 0% 6322Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 70m 0% 12507Mi 21% gke-xlou-cdm-ds-32e4dcb1-4hls 65m 0% 12209Mi 20% gke-xlou-cdm-ds-32e4dcb1-gxzj 68m 0% 6405Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 69m 0% 12700Mi 21% gke-xlou-cdm-ds-32e4dcb1-rr7g 64m 0% 6373Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 1145m 7% 9587Mi 16% 07:01:04 DEBUG --- stderr --- 07:01:04 DEBUG 07:01:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728891" 07:01:31 INFO [http_cmd]: http status code OK 07:01:31 DEBUG --- status code --- 07:01:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:31 DEBUG --- http response --- 07:01:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:01:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712728891" 07:01:31 ERROR [http_cmd]: ERROR 07:01:31 ERROR --- status code --- 07:01:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:01:31 ERROR --- http response --- 07:01:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:01:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:01:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728891" 07:01:31 INFO [http_cmd]: http status code OK 07:01:31 DEBUG --- status code --- 07:01:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:31 DEBUG --- http response --- 07:01:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:01:31 DEBUG ----- output ----- 07:01:31 DEBUG { 07:01:31 DEBUG "status": "success", 07:01:31 DEBUG "data": { 07:01:31 DEBUG "resultType": "vector", 07:01:31 DEBUG "result": [ 07:01:31 DEBUG { 07:01:31 DEBUG "metric": { 07:01:31 DEBUG "pod": "am-654c4fcbcf-v5qb5" 07:01:31 DEBUG }, 07:01:31 DEBUG "value": [ 07:01:31 DEBUG 07:01:31 DEBUG 07:01:31 DEBUG 07:01:31 DEBUG [...] 07:01:31 DEBUG 07:01:31 DEBUG 07:01:31 DEBUG 07:01:31 DEBUG "pod": "am-654c4fcbcf-487cv" 07:01:31 DEBUG }, 07:01:31 DEBUG "value": [ 07:01:31 DEBUG 1712728891, 07:01:31 DEBUG "0" 07:01:31 DEBUG ] 07:01:31 DEBUG } 07:01:31 DEBUG ] 07:01:31 DEBUG } 07:01:31 DEBUG } 07:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728892" 07:01:32 INFO [http_cmd]: http status code OK 07:01:32 DEBUG --- status code --- 07:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:32 DEBUG --- http response --- 07:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:01:32 DEBUG ----- output ----- 07:01:32 DEBUG { 07:01:32 DEBUG "status": "success", 07:01:32 DEBUG "data": { 07:01:32 DEBUG "resultType": "vector", 07:01:32 DEBUG "result": [ 07:01:32 DEBUG { 07:01:32 DEBUG "metric": { 07:01:32 DEBUG "pod": "ds-idrepo-0" 07:01:32 DEBUG }, 07:01:32 DEBUG "value": [ 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG [...] 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG "pod": "ds-idrepo-1" 07:01:32 DEBUG }, 07:01:32 DEBUG "value": [ 07:01:32 DEBUG 1712728892, 07:01:32 DEBUG "0" 07:01:32 DEBUG ] 07:01:32 DEBUG } 07:01:32 DEBUG ] 07:01:32 DEBUG } 07:01:32 DEBUG } 07:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728892" 07:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728892" 07:01:32 INFO [http_cmd]: http status code OK 07:01:32 INFO [http_cmd]: http status code OK 07:01:32 DEBUG --- status code --- 07:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728892" 07:01:32 DEBUG --- status code --- 07:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:32 DEBUG --- http response --- 07:01:32 DEBUG --- http response --- 07:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:01:32 DEBUG ----- output ----- 07:01:32 DEBUG ----- output ----- 07:01:32 DEBUG { 07:01:32 DEBUG { 07:01:32 DEBUG "status": "success", 07:01:32 DEBUG "status": "success", 07:01:32 DEBUG "data": { 07:01:32 DEBUG "data": { 07:01:32 DEBUG "resultType": "vector", 07:01:32 DEBUG "resultType": "vector", 07:01:32 DEBUG "result": [ 07:01:32 DEBUG "result": [ 07:01:32 DEBUG { 07:01:32 DEBUG { 07:01:32 DEBUG "metric": { 07:01:32 DEBUG "metric": { 07:01:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 07:01:32 DEBUG "instance": "10.142.0.25:9100" 07:01:32 DEBUG }, 07:01:32 DEBUG }, 07:01:32 DEBUG "value": [ 07:01:32 DEBUG "value": [ 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG [...] 07:01:32 DEBUG [...] 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG "pod": "am-654c4fcbcf-487cv" 07:01:32 DEBUG "instance": "10.142.0.77:9100" 07:01:32 DEBUG }, 07:01:32 DEBUG }, 07:01:32 DEBUG "value": [ 07:01:32 DEBUG "value": [ 07:01:32 DEBUG 1712728892, 07:01:32 DEBUG 1712728892, 07:01:32 DEBUG "0" 07:01:32 DEBUG "24967.933333333334" 07:01:32 DEBUG ] 07:01:32 DEBUG ] 07:01:32 DEBUG } 07:01:32 DEBUG } 07:01:32 DEBUG ] 07:01:32 DEBUG ] 07:01:32 DEBUG } 07:01:32 DEBUG } 07:01:32 DEBUG } 07:01:32 DEBUG } 07:01:32 INFO [http_cmd]: http status code OK 07:01:32 DEBUG --- status code --- 07:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:32 DEBUG --- http response --- 07:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:01:32 DEBUG ----- output ----- 07:01:32 DEBUG { 07:01:32 DEBUG "status": "success", 07:01:32 DEBUG "data": { 07:01:32 DEBUG "resultType": "vector", 07:01:32 DEBUG "result": [ 07:01:32 DEBUG { 07:01:32 DEBUG "metric": { 07:01:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 07:01:32 DEBUG }, 07:01:32 DEBUG "value": [ 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG [...] 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG "pod": "am-654c4fcbcf-487cv" 07:01:32 DEBUG }, 07:01:32 DEBUG "value": [ 07:01:32 DEBUG 1712728892, 07:01:32 DEBUG "0" 07:01:32 DEBUG ] 07:01:32 DEBUG } 07:01:32 DEBUG ] 07:01:32 DEBUG } 07:01:32 DEBUG } 07:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728892" 07:01:32 INFO [http_cmd]: http status code OK 07:01:32 DEBUG --- status code --- 07:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:32 DEBUG --- http response --- 07:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 07:01:32 DEBUG ----- output ----- 07:01:32 DEBUG { 07:01:32 DEBUG "status": "success", 07:01:32 DEBUG "data": { 07:01:32 DEBUG "resultType": "vector", 07:01:32 DEBUG "result": [ 07:01:32 DEBUG { 07:01:32 DEBUG "metric": { 07:01:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 07:01:32 DEBUG }, 07:01:32 DEBUG "value": [ 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG [...] 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 07:01:32 DEBUG }, 07:01:32 DEBUG "value": [ 07:01:32 DEBUG 1712728892, 07:01:32 DEBUG "0" 07:01:32 DEBUG ] 07:01:32 DEBUG } 07:01:32 DEBUG ] 07:01:32 DEBUG } 07:01:32 DEBUG } 07:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728892" 07:01:32 INFO [http_cmd]: http status code OK 07:01:32 DEBUG --- status code --- 07:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:32 DEBUG --- http response --- 07:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:01:32 DEBUG ----- output ----- 07:01:32 DEBUG { 07:01:32 DEBUG "status": "success", 07:01:32 DEBUG "data": { 07:01:32 DEBUG "resultType": "vector", 07:01:32 DEBUG "result": [ 07:01:32 DEBUG { 07:01:32 DEBUG "metric": { 07:01:32 DEBUG "instance": "10.142.0.25:9100" 07:01:32 DEBUG }, 07:01:32 DEBUG "value": [ 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG [...] 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG "instance": "10.142.0.77:9100" 07:01:32 DEBUG }, 07:01:32 DEBUG "value": [ 07:01:32 DEBUG 1712728892, 07:01:32 DEBUG "438094969856" 07:01:32 DEBUG ] 07:01:32 DEBUG } 07:01:32 DEBUG ] 07:01:32 DEBUG } 07:01:32 DEBUG } 07:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728892" 07:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712728892" 07:01:32 INFO [http_cmd]: http status code OK 07:01:32 DEBUG --- status code --- 07:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:32 DEBUG --- http response --- 07:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:01:32 DEBUG ----- output ----- 07:01:32 DEBUG { 07:01:32 DEBUG "status": "success", 07:01:32 DEBUG "data": { 07:01:32 DEBUG "resultType": "vector", 07:01:32 DEBUG "result": [ 07:01:32 DEBUG { 07:01:32 DEBUG "metric": { 07:01:32 DEBUG "pod": "ds-idrepo-0" 07:01:32 DEBUG }, 07:01:32 DEBUG "value": [ 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG [...] 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG "pod": "ds-cts-2" 07:01:32 DEBUG }, 07:01:32 DEBUG "value": [ 07:01:32 DEBUG 1712728892, 07:01:32 DEBUG "0" 07:01:32 DEBUG ] 07:01:32 DEBUG } 07:01:32 DEBUG ] 07:01:32 DEBUG } 07:01:32 DEBUG } 07:01:32 INFO [http_cmd]: http status code OK 07:01:32 DEBUG --- status code --- 07:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:32 DEBUG --- http response --- 07:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:01:32 DEBUG ----- output ----- 07:01:32 DEBUG { 07:01:32 DEBUG "status": "success", 07:01:32 DEBUG "data": { 07:01:32 DEBUG "resultType": "vector", 07:01:32 DEBUG "result": [ 07:01:32 DEBUG { 07:01:32 DEBUG "metric": { 07:01:32 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 07:01:32 DEBUG }, 07:01:32 DEBUG "value": [ 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG [...] 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:01:32 DEBUG }, 07:01:32 DEBUG "value": [ 07:01:32 DEBUG 1712728892, 07:01:32 DEBUG "1.0436621728069504" 07:01:32 DEBUG ] 07:01:32 DEBUG } 07:01:32 DEBUG ] 07:01:32 DEBUG } 07:01:32 DEBUG } 07:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728892" 07:01:32 INFO [http_cmd]: http status code OK 07:01:32 DEBUG --- status code --- 07:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:32 DEBUG --- http response --- 07:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:01:32 DEBUG ----- output ----- 07:01:32 DEBUG { 07:01:32 DEBUG "status": "success", 07:01:32 DEBUG "data": { 07:01:32 DEBUG "resultType": "vector", 07:01:32 DEBUG "result": [ 07:01:32 DEBUG { 07:01:32 DEBUG "metric": { 07:01:32 DEBUG "instance": "10.142.0.25:9100" 07:01:32 DEBUG }, 07:01:32 DEBUG "value": [ 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG [...] 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG "instance": "10.142.0.77:9100" 07:01:32 DEBUG }, 07:01:32 DEBUG "value": [ 07:01:32 DEBUG 1712728892, 07:01:32 DEBUG "0.0003666666666634436" 07:01:32 DEBUG ] 07:01:32 DEBUG } 07:01:32 DEBUG ] 07:01:32 DEBUG } 07:01:32 DEBUG } 07:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728892" 07:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728892" 07:01:32 INFO [http_cmd]: http status code OK 07:01:32 INFO [http_cmd]: http status code OK 07:01:32 DEBUG --- status code --- 07:01:32 DEBUG --- status code --- 07:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:32 DEBUG --- http response --- 07:01:32 DEBUG --- http response --- 07:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 07:01:32 DEBUG ----- output ----- 07:01:32 DEBUG ----- output ----- 07:01:32 DEBUG { 07:01:32 DEBUG { 07:01:32 DEBUG "status": "success", 07:01:32 DEBUG "status": "success", 07:01:32 DEBUG "data": { 07:01:32 DEBUG "data": { 07:01:32 DEBUG "resultType": "vector", 07:01:32 DEBUG "resultType": "vector", 07:01:32 DEBUG "result": [ 07:01:32 DEBUG "result": [ 07:01:32 DEBUG { 07:01:32 DEBUG { 07:01:32 DEBUG "metric": { 07:01:32 DEBUG "metric": { 07:01:32 DEBUG "pod": "ds-idrepo-0" 07:01:32 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 07:01:32 DEBUG }, 07:01:32 DEBUG }, 07:01:32 DEBUG "value": [ 07:01:32 DEBUG "value": [ 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG [...] 07:01:32 DEBUG [...] 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG 07:01:32 DEBUG "pod": "ds-cts-2" 07:01:32 DEBUG "pod": "am-654c4fcbcf-v5qb5" 07:01:32 DEBUG }, 07:01:32 DEBUG }, 07:01:32 DEBUG "value": [ 07:01:32 DEBUG "value": [ 07:01:32 DEBUG 1712728892, 07:01:32 DEBUG 1712728892, 07:01:32 DEBUG "0" 07:01:32 DEBUG "14537.710736468502" 07:01:32 DEBUG ] 07:01:32 DEBUG ] 07:01:32 DEBUG } 07:01:32 DEBUG } 07:01:32 DEBUG ] 07:01:32 DEBUG ] 07:01:32 DEBUG } 07:01:32 DEBUG } 07:01:32 DEBUG } 07:01:32 DEBUG } 07:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728892" 07:01:33 INFO [http_cmd]: http status code OK 07:01:33 DEBUG --- status code --- 07:01:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:33 DEBUG --- http response --- 07:01:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:01:33 DEBUG ----- output ----- 07:01:33 DEBUG { 07:01:33 DEBUG "status": "success", 07:01:33 DEBUG "data": { 07:01:33 DEBUG "resultType": "vector", 07:01:33 DEBUG "result": [ 07:01:33 DEBUG { 07:01:33 DEBUG "metric": { 07:01:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 07:01:33 DEBUG }, 07:01:33 DEBUG "value": [ 07:01:33 DEBUG 07:01:33 DEBUG 07:01:33 DEBUG 07:01:33 DEBUG [...] 07:01:33 DEBUG 07:01:33 DEBUG 07:01:33 DEBUG 07:01:33 DEBUG "pod": "am-654c4fcbcf-487cv" 07:01:33 DEBUG }, 07:01:33 DEBUG "value": [ 07:01:33 DEBUG 1712728892, 07:01:33 DEBUG "0" 07:01:33 DEBUG ] 07:01:33 DEBUG } 07:01:33 DEBUG ] 07:01:33 DEBUG } 07:01:33 DEBUG } 07:01:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712728893" 07:01:33 INFO [http_cmd]: http status code OK 07:01:33 DEBUG --- status code --- 07:01:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:33 DEBUG --- http response --- 07:01:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:01:33 DEBUG ----- output ----- 07:01:33 DEBUG { 07:01:33 DEBUG "status": "success", 07:01:33 DEBUG "data": { 07:01:33 DEBUG "resultType": "vector", 07:01:33 DEBUG "result": [ 07:01:33 DEBUG { 07:01:33 DEBUG "metric": { 07:01:33 DEBUG "pod": "ds-idrepo-0" 07:01:33 DEBUG }, 07:01:33 DEBUG "value": [ 07:01:33 DEBUG 07:01:33 DEBUG 07:01:33 DEBUG 07:01:33 DEBUG [...] 07:01:33 DEBUG 07:01:33 DEBUG 07:01:33 DEBUG 07:01:33 DEBUG "pod": "ds-cts-2" 07:01:33 DEBUG }, 07:01:33 DEBUG "value": [ 07:01:33 DEBUG 1712728893, 07:01:33 DEBUG "0" 07:01:33 DEBUG ] 07:01:33 DEBUG } 07:01:33 DEBUG ] 07:01:33 DEBUG } 07:01:33 DEBUG } 07:01:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728893" 07:01:33 INFO [http_cmd]: http status code OK 07:01:33 DEBUG --- status code --- 07:01:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:33 DEBUG --- http response --- 07:01:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:01:33 DEBUG ----- output ----- 07:01:33 DEBUG { 07:01:33 DEBUG "status": "success", 07:01:33 DEBUG "data": { 07:01:33 DEBUG "resultType": "vector", 07:01:33 DEBUG "result": [ 07:01:33 DEBUG { 07:01:33 DEBUG "metric": { 07:01:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 07:01:33 DEBUG }, 07:01:33 DEBUG "value": [ 07:01:33 DEBUG 07:01:33 DEBUG 07:01:33 DEBUG 07:01:33 DEBUG [...] 07:01:33 DEBUG 07:01:33 DEBUG 07:01:33 DEBUG 07:01:33 DEBUG "pod": "am-654c4fcbcf-487cv" 07:01:33 DEBUG }, 07:01:33 DEBUG "value": [ 07:01:33 DEBUG 1712728893, 07:01:33 DEBUG "0" 07:01:33 DEBUG ] 07:01:33 DEBUG } 07:01:33 DEBUG ] 07:01:33 DEBUG } 07:01:33 DEBUG } 07:01:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712728893" 07:01:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728893" 07:01:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728893" 07:01:33 INFO [http_cmd]: http status code OK 07:01:33 DEBUG --- status code --- 07:01:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:33 DEBUG --- http response --- 07:01:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:01:33 DEBUG ----- output ----- 07:01:33 DEBUG { 07:01:33 DEBUG "status": "success", 07:01:33 DEBUG "data": { 07:01:33 DEBUG "resultType": "vector", 07:01:33 DEBUG "result": [ 07:01:33 DEBUG { 07:01:33 DEBUG "metric": { 07:01:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 07:01:33 DEBUG }, 07:01:33 DEBUG "value": [ 07:01:33 DEBUG 07:01:33 DEBUG 07:01:33 DEBUG 07:01:33 DEBUG [...] 07:01:33 DEBUG 07:01:33 DEBUG 07:01:33 DEBUG 07:01:33 DEBUG "pod": "am-654c4fcbcf-487cv" 07:01:33 DEBUG }, 07:01:33 DEBUG "value": [ 07:01:33 DEBUG 1712728893, 07:01:33 DEBUG "0.054" 07:01:33 DEBUG ] 07:01:33 DEBUG } 07:01:33 DEBUG ] 07:01:33 DEBUG } 07:01:33 DEBUG } 07:01:33 INFO [http_cmd]: http status code OK 07:01:33 DEBUG --- status code --- 07:01:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:33 DEBUG --- http response --- 07:01:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:01:33 DEBUG ----- output ----- 07:01:33 DEBUG { 07:01:33 DEBUG "status": "success", 07:01:33 DEBUG "data": { 07:01:33 DEBUG "resultType": "vector", 07:01:33 DEBUG "result": [ 07:01:33 DEBUG { 07:01:33 DEBUG "metric": { 07:01:33 DEBUG "pod": "ds-idrepo-2" 07:01:33 DEBUG }, 07:01:33 DEBUG "value": [ 07:01:33 DEBUG 07:01:33 DEBUG 07:01:33 DEBUG 07:01:33 DEBUG [...] 07:01:33 DEBUG 07:01:33 DEBUG 07:01:33 DEBUG 07:01:33 DEBUG "pod": "ds-cts-2" 07:01:33 DEBUG }, 07:01:33 DEBUG "value": [ 07:01:33 DEBUG 1712728893, 07:01:33 DEBUG "0" 07:01:33 DEBUG ] 07:01:33 DEBUG } 07:01:33 DEBUG ] 07:01:33 DEBUG } 07:01:33 DEBUG } 07:01:33 INFO [http_cmd]: http status code OK 07:01:33 DEBUG --- status code --- 07:01:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:33 DEBUG --- http response --- 07:01:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 07:01:33 DEBUG ----- output ----- 07:01:33 DEBUG { 07:01:33 DEBUG "status": "success", 07:01:33 DEBUG "data": { 07:01:33 DEBUG "resultType": "vector", 07:01:33 DEBUG "result": [ 07:01:33 DEBUG { 07:01:33 DEBUG "metric": { 07:01:33 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 07:01:33 DEBUG }, 07:01:33 DEBUG "value": [ 07:01:33 DEBUG 07:01:33 DEBUG 07:01:33 DEBUG 07:01:33 DEBUG [...] 07:01:33 DEBUG 07:01:33 DEBUG 07:01:33 DEBUG 07:01:33 DEBUG "pod": "am-654c4fcbcf-v5qb5" 07:01:33 DEBUG }, 07:01:33 DEBUG "value": [ 07:01:33 DEBUG 1712728893, 07:01:33 DEBUG "0.00869937322565566" 07:01:33 DEBUG ] 07:01:33 DEBUG } 07:01:33 DEBUG ] 07:01:33 DEBUG } 07:01:33 DEBUG } 07:01:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728893" 07:01:33 INFO [http_cmd]: http status code OK 07:01:33 DEBUG --- status code --- 07:01:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:33 DEBUG --- http response --- 07:01:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:01:33 DEBUG ----- output ----- 07:01:33 DEBUG { 07:01:33 DEBUG "status": "success", 07:01:33 DEBUG "data": { 07:01:33 DEBUG "resultType": "vector", 07:01:33 DEBUG "result": [ 07:01:33 DEBUG { 07:01:33 DEBUG "metric": { 07:01:33 DEBUG "pod": "ds-idrepo-0" 07:01:33 DEBUG }, 07:01:33 DEBUG "value": [ 07:01:33 DEBUG 07:01:33 DEBUG 07:01:33 DEBUG 07:01:33 DEBUG [...] 07:01:33 DEBUG 07:01:33 DEBUG 07:01:33 DEBUG 07:01:33 DEBUG "pod": "ds-idrepo-1" 07:01:33 DEBUG }, 07:01:33 DEBUG "value": [ 07:01:33 DEBUG 1712728893, 07:01:33 DEBUG "0" 07:01:33 DEBUG ] 07:01:33 DEBUG } 07:01:33 DEBUG ] 07:01:33 DEBUG } 07:01:33 DEBUG } 07:01:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728894" 07:01:34 INFO [http_cmd]: http status code OK 07:01:34 DEBUG --- status code --- 07:01:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:34 DEBUG --- http response --- 07:01:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:01:34 DEBUG ----- output ----- 07:01:34 DEBUG { 07:01:34 DEBUG "status": "success", 07:01:34 DEBUG "data": { 07:01:34 DEBUG "resultType": "vector", 07:01:34 DEBUG "result": [ 07:01:34 DEBUG { 07:01:34 DEBUG "metric": { 07:01:34 DEBUG "pod": "ds-idrepo-0" 07:01:34 DEBUG }, 07:01:34 DEBUG "value": [ 07:01:34 DEBUG 07:01:34 DEBUG 07:01:34 DEBUG 07:01:34 DEBUG [...] 07:01:34 DEBUG 07:01:34 DEBUG 07:01:34 DEBUG 07:01:34 DEBUG "pod": "ds-idrepo-1" 07:01:34 DEBUG }, 07:01:34 DEBUG "value": [ 07:01:34 DEBUG 1712728894, 07:01:34 DEBUG "0" 07:01:34 DEBUG ] 07:01:34 DEBUG } 07:01:34 DEBUG ] 07:01:34 DEBUG } 07:01:34 DEBUG } 07:01:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728894" 07:01:34 INFO [http_cmd]: http status code OK 07:01:34 DEBUG --- status code --- 07:01:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:34 DEBUG --- http response --- 07:01:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:01:34 DEBUG ----- output ----- 07:01:34 DEBUG { 07:01:34 DEBUG "status": "success", 07:01:34 DEBUG "data": { 07:01:34 DEBUG "resultType": "vector", 07:01:34 DEBUG "result": [ 07:01:34 DEBUG { 07:01:34 DEBUG "metric": { 07:01:34 DEBUG "pod": "am-654c4fcbcf-v5qb5" 07:01:34 DEBUG }, 07:01:34 DEBUG "value": [ 07:01:34 DEBUG 07:01:34 DEBUG 07:01:34 DEBUG 07:01:34 DEBUG [...] 07:01:34 DEBUG 07:01:34 DEBUG 07:01:34 DEBUG 07:01:34 DEBUG "pod": "am-654c4fcbcf-487cv" 07:01:34 DEBUG }, 07:01:34 DEBUG "value": [ 07:01:34 DEBUG 1712728894, 07:01:34 DEBUG "0" 07:01:34 DEBUG ] 07:01:34 DEBUG } 07:01:34 DEBUG ] 07:01:34 DEBUG } 07:01:34 DEBUG } 07:01:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712728894" 07:01:34 INFO [http_cmd]: http status code OK 07:01:34 DEBUG --- status code --- 07:01:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:34 DEBUG --- http response --- 07:01:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 07:01:34 DEBUG ----- output ----- 07:01:34 DEBUG { 07:01:34 DEBUG "status": "success", 07:01:34 DEBUG "data": { 07:01:34 DEBUG "resultType": "vector", 07:01:34 DEBUG "result": [ 07:01:34 DEBUG { 07:01:34 DEBUG "metric": { 07:01:34 DEBUG "pod": "admin-ui-7cf664f88d-pnt57" 07:01:34 DEBUG }, 07:01:34 DEBUG "value": [ 07:01:34 DEBUG 07:01:34 DEBUG 07:01:34 DEBUG 07:01:34 DEBUG [...] 07:01:34 DEBUG 07:01:34 DEBUG 07:01:34 DEBUG 07:01:34 DEBUG "pod": "am-654c4fcbcf-v5qb5" 07:01:34 DEBUG }, 07:01:34 DEBUG "value": [ 07:01:34 DEBUG 1712728894, 07:01:34 DEBUG "0" 07:01:34 DEBUG ] 07:01:34 DEBUG } 07:01:34 DEBUG ] 07:01:34 DEBUG } 07:01:34 DEBUG } 07:01:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728894" 07:01:34 INFO [http_cmd]: http status code OK 07:01:34 DEBUG --- status code --- 07:01:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:34 DEBUG --- http response --- 07:01:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:01:34 DEBUG ----- output ----- 07:01:34 DEBUG { 07:01:34 DEBUG "status": "success", 07:01:34 DEBUG "data": { 07:01:34 DEBUG "resultType": "vector", 07:01:34 DEBUG "result": [ 07:01:34 DEBUG { 07:01:34 DEBUG "metric": { 07:01:34 DEBUG "instance": "10.142.0.25:9100" 07:01:34 DEBUG }, 07:01:34 DEBUG "value": [ 07:01:34 DEBUG 07:01:34 DEBUG 07:01:34 DEBUG 07:01:34 DEBUG [...] 07:01:34 DEBUG 07:01:34 DEBUG 07:01:34 DEBUG 07:01:34 DEBUG "instance": "10.142.0.77:9100" 07:01:34 DEBUG }, 07:01:34 DEBUG "value": [ 07:01:34 DEBUG 1712728894, 07:01:34 DEBUG "25949.966666666667" 07:01:34 DEBUG ] 07:01:34 DEBUG } 07:01:34 DEBUG ] 07:01:34 DEBUG } 07:01:34 DEBUG } 07:01:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712728894" 07:01:34 INFO [http_cmd]: http status code OK 07:01:34 DEBUG --- status code --- 07:01:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:34 DEBUG --- http response --- 07:01:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:01:34 DEBUG ----- output ----- 07:01:34 DEBUG { 07:01:34 DEBUG "status": "success", 07:01:34 DEBUG "data": { 07:01:34 DEBUG "resultType": "vector", 07:01:34 DEBUG "result": [ 07:01:34 DEBUG { 07:01:34 DEBUG "metric": { 07:01:34 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-r9xm" 07:01:34 DEBUG }, 07:01:34 DEBUG "value": [ 07:01:34 DEBUG 07:01:34 DEBUG 07:01:34 DEBUG 07:01:34 DEBUG [...] 07:01:34 DEBUG 07:01:34 DEBUG 07:01:34 DEBUG 07:01:34 DEBUG "node": "gke-xlou-cdm-default-pool-f05840a3-csf7" 07:01:34 DEBUG }, 07:01:34 DEBUG "value": [ 07:01:34 DEBUG 1712728894, 07:01:34 DEBUG "2486239232" 07:01:34 DEBUG ] 07:01:34 DEBUG } 07:01:34 DEBUG ] 07:01:34 DEBUG } 07:01:34 DEBUG } 07:01:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712728894" 07:01:34 INFO [http_cmd]: http status code OK 07:01:34 DEBUG --- status code --- 07:01:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:34 DEBUG --- http response --- 07:01:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:01:34 DEBUG ----- output ----- 07:01:34 DEBUG { 07:01:34 DEBUG "status": "success", 07:01:34 DEBUG "data": { 07:01:34 DEBUG "resultType": "vector", 07:01:34 DEBUG "result": [ 07:01:34 DEBUG { 07:01:34 DEBUG "metric": { 07:01:34 DEBUG "instance": "10.142.0.25:9100" 07:01:34 DEBUG }, 07:01:34 DEBUG "value": [ 07:01:34 DEBUG 07:01:34 DEBUG 07:01:34 DEBUG 07:01:34 DEBUG [...] 07:01:34 DEBUG 07:01:34 DEBUG 07:01:34 DEBUG 07:01:34 DEBUG "instance": "10.142.0.77:9100" 07:01:34 DEBUG }, 07:01:34 DEBUG "value": [ 07:01:34 DEBUG 1712728894, 07:01:34 DEBUG "0" 07:01:34 DEBUG ] 07:01:34 DEBUG } 07:01:34 DEBUG ] 07:01:34 DEBUG } 07:01:34 DEBUG } 07:01:55 INFO 07:01:55 INFO [loop_until]: kubectl --namespace=xlou top pods 07:01:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:01:55 INFO [loop_until]: OK (rc = 0) 07:01:55 DEBUG --- stdout --- 07:01:55 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-7cf664f88d-pnt57 1m 9Mi am-654c4fcbcf-487cv 7m 8870Mi am-654c4fcbcf-9scrj 11m 8378Mi am-654c4fcbcf-v5qb5 9m 8096Mi ds-cts-0 13m 11767Mi ds-cts-1 13m 11965Mi ds-cts-2 12m 11473Mi ds-idrepo-0 14m 5620Mi ds-idrepo-1 10m 5677Mi ds-idrepo-2 13m 5703Mi end-user-ui-6d9fd5487b-2s47b 1m 5Mi idm-6bfb7c96cc-qklk9 4m 1198Mi idm-6bfb7c96cc-vsb58 4m 2213Mi lodemon-b776b7d88-thrqf 7m 73Mi login-ui-5f964fb894-5zpvq 1m 4Mi overseer-0-6f8784665d-zq5hb 1016m 7224Mi 07:01:55 DEBUG --- stderr --- 07:01:55 DEBUG 07:02:04 INFO 07:02:04 INFO [loop_until]: kubectl --namespace=xlou top node 07:02:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:02:04 INFO [loop_until]: OK (rc = 0) 07:02:04 DEBUG --- stdout --- 07:02:04 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-f05840a3-8ff6 93m 0% 2313Mi 3% gke-xlou-cdm-default-pool-f05840a3-8jnh 69m 0% 8990Mi 15% gke-xlou-cdm-default-pool-f05840a3-csf7 75m 0% 3257Mi 5% gke-xlou-cdm-default-pool-f05840a3-i7d7 215m 1% 6502Mi 11% gke-xlou-cdm-default-pool-f05840a3-lc4w 73m 0% 9283Mi 15% gke-xlou-cdm-default-pool-f05840a3-qh34 65m 0% 9734Mi 16% gke-xlou-cdm-default-pool-f05840a3-r9xm 75m 0% 865Mi 1% gke-xlou-cdm-ds-32e4dcb1-32mw 71m 0% 6320Mi 10% gke-xlou-cdm-ds-32e4dcb1-3pj9 69m 0% 12510Mi 21% gke-xlou-cdm-ds-32e4dcb1-4hls 65m 0% 12210Mi 20% gke-xlou-cdm-ds-32e4dcb1-gxzj 66m 0% 6400Mi 10% gke-xlou-cdm-ds-32e4dcb1-j2hk 67m 0% 12701Mi 21% gke-xlou-cdm-ds-32e4dcb1-rr7g 62m 0% 6376Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 1083m 6% 10744Mi 18% 07:02:04 DEBUG --- stderr --- 07:02:04 DEBUG 07:02:20 INFO Finished: True 07:02:20 INFO Waiting for threads to register finish flag 07:03:04 INFO Done. Have a nice day! :) 127.0.0.1 - - [10/Apr/2024 07:03:04] "GET /monitoring/stop HTTP/1.1" 200 - 07:03:08 WARNING File /tmp/lodemon_data-20240409_230625/node_disk_read_bytes_total.json does not exist. Skipping graph generation... WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p6471', ...) 127.0.0.1 - - [10/Apr/2024 07:03:13] "GET /monitoring/process HTTP/1.1" 200 -