==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-ccc964465-p7gdm Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-default-pool-1b0769c8-7vmj/10.142.0.111 Start Time: Thu, 11 Apr 2024 01:27:23 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=ccc964465 skaffold.dev/run-id=8e63acc6-045d-490f-a910-36ee0ab7634f Annotations: Status: Running IP: 10.106.42.155 IPs: IP: 10.106.42.155 Controlled By: ReplicaSet/lodemon-ccc964465 Containers: lodemon: Container ID: containerd://17c25e1563846a7eca5dda566128718c50053a443aaffdc123bd8f7ed0f907cb Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:134bb4712d11502a25e7ee31e6d23fdf7779d5261411aa7de9c6af8c021f4fc0 Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Thu, 11 Apr 2024 01:27:24 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /lodestar/config/deployments_to_monitor.yaml from deployments-config (rw,path="deployments_to_monitor.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kzdr4 (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false deployments-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-deployments Optional: false kube-api-access-kzdr4: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== 06:30:31 INFO 06:30:31 INFO [loop_until]: kubectl --namespace=xlou top pods 06:30:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:30:31 INFO [loop_until]: OK (rc = 0) 06:30:31 DEBUG --- stdout --- 06:30:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 8678m 8500Mi am-5b5bd8567b-8h2w4 9164m 8562Mi am-5b5bd8567b-qlkw6 7558m 8457Mi ds-cts-0 7848m 12868Mi ds-cts-1 8554m 13399Mi ds-cts-2 8238m 12842Mi ds-idrepo-0 11548m 5497Mi ds-idrepo-1 344m 5479Mi ds-idrepo-2 356m 5783Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 4m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 2m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4124m 1841Mi 06:30:31 DEBUG --- stderr --- 06:30:31 DEBUG 06:30:34 INFO 06:30:34 INFO [loop_until]: kubectl --namespace=xlou top node 06:30:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:30:34 INFO [loop_until]: OK (rc = 0) 06:30:34 DEBUG --- stdout --- 06:30:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 2725Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 66m 0% 2785Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7915m 26% 10108Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10793m 36% 2978Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9564m 32% 10090Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 151m 0% 3854Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9805m 32% 10188Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 424m 2% 6455Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11570m 72% 6189Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8399m 52% 13529Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 8090m 50% 13468Mi 22% gke-xlou-cdm-ds-32e4dcb1-j2hk 8868m 55% 14003Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 404m 2% 6154Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4313m 27% 5616Mi 9% 06:30:34 DEBUG --- stderr --- 06:30:34 DEBUG 06:31:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813480" 06:31:20 INFO [http_cmd]: http status code OK 06:31:20 DEBUG --- status code --- 06:31:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:20 DEBUG --- http response --- 06:31:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 06:31:20 DEBUG ----- output ----- 06:31:20 DEBUG { 06:31:20 DEBUG "status": "success", 06:31:20 DEBUG "data": { 06:31:20 DEBUG "resultType": "vector", 06:31:20 DEBUG "result": [ 06:31:20 DEBUG { 06:31:20 DEBUG "metric": { 06:31:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:31:20 DEBUG }, 06:31:20 DEBUG "value": [ 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG [...] 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:31:20 DEBUG }, 06:31:20 DEBUG "value": [ 06:31:20 DEBUG 1712813480, 06:31:20 DEBUG "0" 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG } 06:31:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813480" 06:31:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813480" 06:31:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712813480" 06:31:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813480" 06:31:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813480" 06:31:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813480" 06:31:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813480" 06:31:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712813480" 06:31:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813480" 06:31:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813480" 06:31:20 INFO [http_cmd]: http status code OK 06:31:20 DEBUG --- status code --- 06:31:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:20 DEBUG --- http response --- 06:31:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712813480" 06:31:20 INFO [http_cmd]: http status code OK 06:31:20 INFO [http_cmd]: http status code OK 06:31:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:31:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813480" 06:31:20 INFO [http_cmd]: http status code OK 06:31:20 INFO [http_cmd]: http status code OK 06:31:20 DEBUG --- status code --- 06:31:20 DEBUG --- status code --- 06:31:20 DEBUG ----- output ----- 06:31:20 DEBUG --- status code --- 06:31:20 INFO [http_cmd]: http status code OK 06:31:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813480" 06:31:20 INFO [http_cmd]: http status code OK 06:31:20 DEBUG --- status code --- 06:31:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813480" 06:31:20 ERROR [http_cmd]: ERROR 06:31:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:20 INFO [http_cmd]: http status code OK 06:31:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:20 INFO [http_cmd]: http status code OK 06:31:20 DEBUG { 06:31:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:20 DEBUG --- status code --- 06:31:20 DEBUG --- status code --- 06:31:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:20 ERROR --- status code --- 06:31:20 DEBUG --- http response --- 06:31:20 DEBUG --- status code --- 06:31:20 DEBUG --- http response --- 06:31:20 INFO [http_cmd]: http status code OK 06:31:20 DEBUG --- status code --- 06:31:20 DEBUG "status": "success", 06:31:20 DEBUG --- http response --- 06:31:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:20 DEBUG --- http response --- 06:31:20 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:31:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:31:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:20 INFO [http_cmd]: http status code OK 06:31:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813480" 06:31:20 DEBUG --- status code --- 06:31:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:31:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:20 DEBUG "data": { 06:31:20 DEBUG "resultType": "vector", 06:31:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:31:20 DEBUG --- http response --- 06:31:20 ERROR --- http response --- 06:31:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:31:20 DEBUG ----- output ----- 06:31:20 DEBUG --- http response --- 06:31:20 DEBUG --- status code --- 06:31:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:20 DEBUG ----- output ----- 06:31:20 DEBUG --- http response --- 06:31:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813480" 06:31:20 DEBUG --- http response --- 06:31:20 DEBUG "result": [ 06:31:20 DEBUG ----- output ----- 06:31:20 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:31:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 06:31:20 DEBUG ----- output ----- 06:31:20 DEBUG { 06:31:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:31:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:20 DEBUG --- http response --- 06:31:20 INFO [http_cmd]: http status code OK 06:31:20 INFO [http_cmd]: http status code OK 06:31:20 DEBUG { 06:31:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:31:20 DEBUG ----- output ----- 06:31:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:31:20 DEBUG { 06:31:20 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:31:20 DEBUG ----- output ----- 06:31:20 DEBUG { 06:31:20 DEBUG "status": "success", 06:31:20 DEBUG ----- output ----- 06:31:20 DEBUG --- http response --- 06:31:20 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:31:20 DEBUG --- status code --- 06:31:20 DEBUG --- status code --- 06:31:20 DEBUG "status": "success", 06:31:20 DEBUG { 06:31:20 DEBUG { 06:31:20 DEBUG ----- output ----- 06:31:20 DEBUG "status": "success", 06:31:20 DEBUG { 06:31:20 DEBUG "status": "success", 06:31:20 DEBUG "data": { 06:31:20 DEBUG { 06:31:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:31:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:20 DEBUG "data": { 06:31:20 DEBUG "metric": { 06:31:20 DEBUG "status": "success", 06:31:20 DEBUG { 06:31:20 DEBUG "data": { 06:31:20 DEBUG "status": "success", 06:31:20 DEBUG "data": { 06:31:20 DEBUG "resultType": "vector", 06:31:20 DEBUG "status": "success", 06:31:20 DEBUG ----- output ----- 06:31:20 DEBUG --- http response --- 06:31:20 DEBUG --- http response --- 06:31:20 DEBUG "resultType": "vector", 06:31:20 DEBUG "pod": "ds-idrepo-2" 06:31:20 DEBUG "data": { 06:31:20 DEBUG "status": "success", 06:31:20 DEBUG "resultType": "vector", 06:31:20 DEBUG "data": { 06:31:20 DEBUG "resultType": "vector", 06:31:20 DEBUG "result": [ 06:31:20 DEBUG "data": { 06:31:20 DEBUG { 06:31:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:31:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:31:20 DEBUG "result": [ 06:31:20 INFO [http_cmd]: http status code OK 06:31:20 DEBUG }, 06:31:20 DEBUG "resultType": "vector", 06:31:20 DEBUG "data": { 06:31:20 DEBUG "result": [ 06:31:20 DEBUG "resultType": "vector", 06:31:20 DEBUG "result": [ 06:31:20 DEBUG { 06:31:20 DEBUG "resultType": "vector", 06:31:20 DEBUG "status": "success", 06:31:20 DEBUG ----- output ----- 06:31:20 DEBUG ----- output ----- 06:31:20 DEBUG { 06:31:20 DEBUG --- status code --- 06:31:20 DEBUG "value": [ 06:31:20 DEBUG "result": [ 06:31:20 DEBUG "resultType": "vector", 06:31:20 DEBUG { 06:31:20 DEBUG "result": [ 06:31:20 DEBUG { 06:31:20 DEBUG "metric": { 06:31:20 DEBUG "result": [ 06:31:20 DEBUG "data": { 06:31:20 DEBUG { 06:31:20 DEBUG { 06:31:20 DEBUG "metric": { 06:31:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:20 DEBUG 06:31:20 DEBUG { 06:31:20 DEBUG "result": [ 06:31:20 DEBUG "metric": { 06:31:20 DEBUG { 06:31:20 DEBUG "metric": { 06:31:20 DEBUG "instance": "10.142.0.107:9100" 06:31:20 DEBUG { 06:31:20 DEBUG "metric": { 06:31:20 DEBUG "status": "success", 06:31:20 DEBUG "status": "success", 06:31:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:31:20 DEBUG --- http response --- 06:31:20 DEBUG 06:31:20 DEBUG "metric": { 06:31:20 DEBUG { 06:31:20 DEBUG "pod": "ds-idrepo-2" 06:31:20 DEBUG "metric": { 06:31:20 INFO [http_cmd]: http status code OK 06:31:20 DEBUG "pod": "ds-idrepo-2" 06:31:20 DEBUG }, 06:31:20 DEBUG "resultType": "vector", 06:31:20 DEBUG "pod": "ds-idrepo-2" 06:31:20 DEBUG "data": { 06:31:20 DEBUG "data": { 06:31:20 DEBUG }, 06:31:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:31:20 DEBUG 06:31:20 DEBUG "pod": "ds-idrepo-2" 06:31:20 DEBUG "metric": { 06:31:20 DEBUG }, 06:31:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:31:20 DEBUG --- status code --- 06:31:20 DEBUG }, 06:31:20 DEBUG "value": [ 06:31:20 DEBUG "result": [ 06:31:20 DEBUG }, 06:31:20 DEBUG "resultType": "vector", 06:31:20 DEBUG "resultType": "vector", 06:31:20 DEBUG "value": [ 06:31:20 DEBUG ----- output ----- 06:31:20 DEBUG [...] 06:31:20 DEBUG }, 06:31:20 DEBUG "pod": "ds-idrepo-2" 06:31:20 DEBUG "value": [ 06:31:20 DEBUG }, 06:31:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:20 DEBUG "value": [ 06:31:20 DEBUG 06:31:20 DEBUG { 06:31:20 DEBUG "value": [ 06:31:20 DEBUG "result": [ 06:31:20 DEBUG "result": [ 06:31:20 DEBUG 06:31:20 DEBUG { 06:31:20 DEBUG 06:31:20 DEBUG "value": [ 06:31:20 DEBUG }, 06:31:20 DEBUG 06:31:20 DEBUG "value": [ 06:31:20 DEBUG --- http response --- 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG "metric": { 06:31:20 DEBUG 06:31:20 DEBUG { 06:31:20 DEBUG { 06:31:20 DEBUG 06:31:20 DEBUG "status": "success", 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG "value": [ 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:31:20 DEBUG 06:31:20 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:31:20 DEBUG 06:31:20 DEBUG "metric": { 06:31:20 DEBUG "metric": { 06:31:20 DEBUG 06:31:20 DEBUG "data": { 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG ----- output ----- 06:31:20 DEBUG [...] 06:31:20 DEBUG }, 06:31:20 DEBUG 06:31:20 DEBUG "pod": "ds-idrepo-2" 06:31:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:31:20 DEBUG [...] 06:31:20 DEBUG "resultType": "vector", 06:31:20 DEBUG "pod": "ds-idrepo-0" 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG [...] 06:31:20 DEBUG 06:31:20 DEBUG [...] 06:31:20 DEBUG { 06:31:20 DEBUG 06:31:20 DEBUG "value": [ 06:31:20 DEBUG [...] 06:31:20 DEBUG }, 06:31:20 DEBUG }, 06:31:20 DEBUG 06:31:20 DEBUG "result": [ 06:31:20 DEBUG }, 06:31:20 DEBUG [...] 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG [...] 06:31:20 DEBUG 06:31:20 DEBUG "status": "success", 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG "value": [ 06:31:20 DEBUG "value": [ 06:31:20 DEBUG 06:31:20 DEBUG { 06:31:20 DEBUG "value": [ 06:31:20 DEBUG 06:31:20 DEBUG [...] 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG "data": { 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG "metric": { 06:31:20 DEBUG 1712813480, 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG "resultType": "vector", 06:31:20 DEBUG "instance": "10.142.0.77:9100" 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:31:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:31:20 DEBUG "0" 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG "pod": "ds-cts-2" 06:31:20 DEBUG 06:31:20 DEBUG "pod": "ds-idrepo-0" 06:31:20 DEBUG "result": [ 06:31:20 DEBUG }, 06:31:20 DEBUG [...] 06:31:20 DEBUG "pod": "ds-cts-2" 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG }, 06:31:20 DEBUG }, 06:31:20 DEBUG ] 06:31:20 DEBUG "pod": "ds-idrepo-0" 06:31:20 DEBUG 06:31:20 DEBUG }, 06:31:20 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:31:20 DEBUG }, 06:31:20 DEBUG { 06:31:20 DEBUG "value": [ 06:31:20 DEBUG 06:31:20 DEBUG }, 06:31:20 DEBUG [...] 06:31:20 DEBUG [...] 06:31:20 DEBUG "value": [ 06:31:20 DEBUG "value": [ 06:31:20 DEBUG } 06:31:20 DEBUG }, 06:31:20 DEBUG "pod": "ds-cts-2" 06:31:20 DEBUG "value": [ 06:31:20 DEBUG }, 06:31:20 DEBUG "value": [ 06:31:20 DEBUG "metric": { 06:31:20 DEBUG 1712813480, 06:31:20 DEBUG 06:31:20 DEBUG "value": [ 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG ] 06:31:20 DEBUG "value": [ 06:31:20 DEBUG }, 06:31:20 DEBUG 1712813480, 06:31:20 DEBUG "value": [ 06:31:20 DEBUG 1712813480, 06:31:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:31:20 DEBUG "52935048192" 06:31:20 DEBUG 06:31:20 DEBUG 1712813480, 06:31:20 DEBUG 06:31:20 DEBUG 1712813480, 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG } 06:31:20 DEBUG 1712813480, 06:31:20 DEBUG "value": [ 06:31:20 DEBUG "0" 06:31:20 DEBUG 1712813480, 06:31:20 DEBUG "0" 06:31:20 DEBUG }, 06:31:20 DEBUG ] 06:31:20 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 06:31:20 DEBUG "0.000029593888888888886" 06:31:20 DEBUG 06:31:20 DEBUG "1364.5" 06:31:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:31:20 DEBUG 06:31:20 DEBUG } 06:31:20 DEBUG "0" 06:31:20 DEBUG 1712813480, 06:31:20 DEBUG ] 06:31:20 DEBUG "0" 06:31:20 DEBUG ] 06:31:20 DEBUG "value": [ 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG }, 06:31:20 DEBUG "pod": "ds-cts-2" 06:31:20 DEBUG ] 06:31:20 DEBUG }, 06:31:20 DEBUG [...] 06:31:20 DEBUG ] 06:31:20 DEBUG "0" 06:31:20 DEBUG } 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG ] 06:31:20 DEBUG ] 06:31:20 DEBUG 06:31:20 DEBUG "value": [ 06:31:20 DEBUG }, 06:31:20 DEBUG } 06:31:20 DEBUG "value": [ 06:31:20 DEBUG 06:31:20 DEBUG } 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG } 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG } 06:31:20 DEBUG [...] 06:31:20 DEBUG 1712813480, 06:31:20 DEBUG "value": [ 06:31:20 DEBUG ] 06:31:20 DEBUG 1712813480, 06:31:20 DEBUG 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG } 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG ] 06:31:20 DEBUG ] 06:31:20 DEBUG 06:31:20 DEBUG "14009806848" 06:31:20 DEBUG 1712813480, 06:31:20 DEBUG } 06:31:20 DEBUG "0" 06:31:20 DEBUG 06:31:20 DEBUG } 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG } 06:31:20 DEBUG } 06:31:20 DEBUG } 06:31:20 DEBUG 06:31:20 DEBUG ] 06:31:20 DEBUG "29.966666666666665" 06:31:20 DEBUG } 06:31:20 DEBUG ] 06:31:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:31:20 DEBUG }, 06:31:20 DEBUG } 06:31:20 DEBUG } 06:31:20 DEBUG } 06:31:20 DEBUG } 06:31:20 DEBUG 06:31:20 DEBUG } 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG } 06:31:20 DEBUG "value": [ 06:31:20 DEBUG } 06:31:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:31:20 DEBUG }, 06:31:20 DEBUG "value": [ 06:31:20 DEBUG ] 06:31:20 DEBUG 1712813480, 06:31:20 DEBUG } 06:31:20 DEBUG ] 06:31:20 DEBUG 1712813480, 06:31:20 DEBUG } 06:31:20 DEBUG "1364.9666666666667" 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG } 06:31:20 DEBUG "1381.5333333333333" 06:31:20 DEBUG } 06:31:20 DEBUG ] 06:31:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813480" 06:31:20 DEBUG } 06:31:20 DEBUG } 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG } 06:31:20 DEBUG ] 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG } 06:31:20 DEBUG } 06:31:20 DEBUG } 06:31:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712813480" 06:31:20 INFO [http_cmd]: http status code OK 06:31:20 DEBUG --- status code --- 06:31:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:20 DEBUG --- http response --- 06:31:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 06:31:20 DEBUG ----- output ----- 06:31:20 DEBUG { 06:31:20 DEBUG "status": "success", 06:31:20 DEBUG "data": { 06:31:20 DEBUG "resultType": "vector", 06:31:20 DEBUG "result": [ 06:31:20 DEBUG { 06:31:20 DEBUG "metric": { 06:31:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:31:20 DEBUG }, 06:31:20 DEBUG "value": [ 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG [...] 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:31:20 DEBUG }, 06:31:20 DEBUG "value": [ 06:31:20 DEBUG 1712813480, 06:31:20 DEBUG "32819.23001758843" 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG } 06:31:20 INFO [http_cmd]: http status code OK 06:31:20 DEBUG --- status code --- 06:31:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:20 DEBUG --- http response --- 06:31:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:31:20 DEBUG ----- output ----- 06:31:20 DEBUG { 06:31:20 DEBUG "status": "success", 06:31:20 DEBUG "data": { 06:31:20 DEBUG "resultType": "vector", 06:31:20 DEBUG "result": [ 06:31:20 DEBUG { 06:31:20 DEBUG "metric": { 06:31:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:31:20 DEBUG }, 06:31:20 DEBUG "value": [ 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG [...] 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:31:20 DEBUG }, 06:31:20 DEBUG "value": [ 06:31:20 DEBUG 1712813480, 06:31:20 DEBUG "0.02" 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG } 06:31:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813480" 06:31:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813480" 06:31:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813480" 06:31:20 INFO [http_cmd]: http status code OK 06:31:20 DEBUG --- status code --- 06:31:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:20 DEBUG --- http response --- 06:31:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:31:20 DEBUG ----- output ----- 06:31:20 DEBUG { 06:31:20 DEBUG "status": "success", 06:31:20 DEBUG "data": { 06:31:20 DEBUG "resultType": "vector", 06:31:20 DEBUG "result": [ 06:31:20 DEBUG { 06:31:20 DEBUG "metric": { 06:31:20 DEBUG "instance": "10.142.0.107:9100" 06:31:20 DEBUG }, 06:31:20 DEBUG "value": [ 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG [...] 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG "instance": "10.142.0.77:9100" 06:31:20 DEBUG }, 06:31:20 DEBUG "value": [ 06:31:20 DEBUG 1712813480, 06:31:20 DEBUG "0.0006666666666662973" 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG } 06:31:20 INFO [http_cmd]: http status code OK 06:31:20 DEBUG --- status code --- 06:31:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:20 DEBUG --- http response --- 06:31:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:31:20 DEBUG ----- output ----- 06:31:20 DEBUG { 06:31:20 DEBUG "status": "success", 06:31:20 DEBUG "data": { 06:31:20 DEBUG "resultType": "vector", 06:31:20 DEBUG "result": [ 06:31:20 DEBUG { 06:31:20 DEBUG "metric": { 06:31:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:31:20 DEBUG }, 06:31:20 DEBUG "value": [ 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG [...] 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:31:20 DEBUG }, 06:31:20 DEBUG "value": [ 06:31:20 DEBUG 1712813480, 06:31:20 DEBUG "244725" 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG } 06:31:20 INFO [http_cmd]: http status code OK 06:31:20 DEBUG --- status code --- 06:31:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:20 DEBUG --- http response --- 06:31:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:31:20 DEBUG ----- output ----- 06:31:20 DEBUG { 06:31:20 DEBUG "status": "success", 06:31:20 DEBUG "data": { 06:31:20 DEBUG "resultType": "vector", 06:31:20 DEBUG "result": [ 06:31:20 DEBUG { 06:31:20 DEBUG "metric": { 06:31:20 DEBUG "instance": "10.142.0.107:9100" 06:31:20 DEBUG }, 06:31:20 DEBUG "value": [ 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG [...] 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG "instance": "10.142.0.77:9100" 06:31:20 DEBUG }, 06:31:20 DEBUG "value": [ 06:31:20 DEBUG 1712813480, 06:31:20 DEBUG "83175463.4" 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG } 06:31:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712813480" 06:31:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813480" 06:31:20 INFO [http_cmd]: http status code OK 06:31:20 DEBUG --- status code --- 06:31:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:20 DEBUG --- http response --- 06:31:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813480" 06:31:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:31:20 DEBUG ----- output ----- 06:31:20 DEBUG { 06:31:20 DEBUG "status": "success", 06:31:20 DEBUG "data": { 06:31:20 DEBUG "resultType": "vector", 06:31:20 DEBUG "result": [ 06:31:20 DEBUG { 06:31:20 DEBUG "metric": { 06:31:20 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:31:20 DEBUG }, 06:31:20 DEBUG "value": [ 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG [...] 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:31:20 DEBUG }, 06:31:20 DEBUG "value": [ 06:31:20 DEBUG 1712813480, 06:31:20 DEBUG "4.20990833347389" 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG } 06:31:20 INFO [http_cmd]: http status code OK 06:31:20 DEBUG --- status code --- 06:31:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:20 DEBUG --- http response --- 06:31:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 06:31:20 DEBUG ----- output ----- 06:31:20 DEBUG { 06:31:20 DEBUG "status": "success", 06:31:20 DEBUG "data": { 06:31:20 DEBUG "resultType": "vector", 06:31:20 DEBUG "result": [ 06:31:20 DEBUG { 06:31:20 DEBUG "metric": { 06:31:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:31:20 DEBUG }, 06:31:20 DEBUG "value": [ 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG [...] 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:31:20 DEBUG }, 06:31:20 DEBUG "value": [ 06:31:20 DEBUG 1712813480, 06:31:20 DEBUG "4.203429708618391" 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG } 06:31:20 INFO [http_cmd]: http status code OK 06:31:20 DEBUG --- status code --- 06:31:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:20 DEBUG --- http response --- 06:31:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:31:20 DEBUG ----- output ----- 06:31:20 DEBUG { 06:31:20 DEBUG "status": "success", 06:31:20 DEBUG "data": { 06:31:20 DEBUG "resultType": "vector", 06:31:20 DEBUG "result": [ 06:31:20 DEBUG { 06:31:20 DEBUG "metric": { 06:31:20 DEBUG "instance": "10.142.0.107:9100" 06:31:20 DEBUG }, 06:31:20 DEBUG "value": [ 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG [...] 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG "instance": "10.142.0.77:9100" 06:31:20 DEBUG }, 06:31:20 DEBUG "value": [ 06:31:20 DEBUG 1712813480, 06:31:20 DEBUG "0.0022999999999998503" 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG } 06:31:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813480" 06:31:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813480" 06:31:20 INFO [http_cmd]: http status code OK 06:31:20 DEBUG --- status code --- 06:31:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:20 DEBUG --- http response --- 06:31:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:31:20 DEBUG ----- output ----- 06:31:20 DEBUG { 06:31:20 DEBUG "status": "success", 06:31:20 DEBUG "data": { 06:31:20 DEBUG "resultType": "vector", 06:31:20 DEBUG "result": [ 06:31:20 DEBUG { 06:31:20 DEBUG "metric": { 06:31:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:31:20 DEBUG }, 06:31:20 DEBUG "value": [ 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG [...] 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:31:20 DEBUG }, 06:31:20 DEBUG "value": [ 06:31:20 DEBUG 1712813480, 06:31:20 DEBUG "1362.3" 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG } 06:31:20 INFO [http_cmd]: http status code OK 06:31:20 DEBUG --- status code --- 06:31:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:31:20 DEBUG --- http response --- 06:31:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:31:20 DEBUG ----- output ----- 06:31:20 DEBUG { 06:31:20 DEBUG "status": "success", 06:31:20 DEBUG "data": { 06:31:20 DEBUG "resultType": "vector", 06:31:20 DEBUG "result": [ 06:31:20 DEBUG { 06:31:20 DEBUG "metric": { 06:31:20 DEBUG "instance": "10.142.0.107:9100" 06:31:20 DEBUG }, 06:31:20 DEBUG "value": [ 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG [...] 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG 06:31:20 DEBUG "instance": "10.142.0.77:9100" 06:31:20 DEBUG }, 06:31:20 DEBUG "value": [ 06:31:20 DEBUG 1712813480, 06:31:20 DEBUG "83173043.5" 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG ] 06:31:20 DEBUG } 06:31:20 DEBUG } 06:31:31 INFO 06:31:31 INFO [loop_until]: kubectl --namespace=xlou top pods 06:31:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:31:31 INFO [loop_until]: OK (rc = 0) 06:31:31 DEBUG --- stdout --- 06:31:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 8802m 8501Mi am-5b5bd8567b-8h2w4 8748m 8563Mi am-5b5bd8567b-qlkw6 7290m 8462Mi ds-cts-0 8233m 13388Mi ds-cts-1 8541m 12955Mi ds-cts-2 8481m 13391Mi ds-idrepo-0 11745m 5497Mi ds-idrepo-1 437m 5479Mi ds-idrepo-2 409m 5783Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 4m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 2m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4278m 1841Mi 06:31:31 DEBUG --- stderr --- 06:31:31 DEBUG 06:31:34 INFO 06:31:34 INFO [loop_until]: kubectl --namespace=xlou top node 06:31:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:31:34 INFO [loop_until]: OK (rc = 0) 06:31:34 DEBUG --- stdout --- 06:31:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 2731Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 64m 0% 2775Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7888m 26% 10112Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10807m 36% 2980Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9671m 32% 10093Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 146m 0% 3857Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9607m 32% 10178Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 460m 2% 6458Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11579m 72% 6185Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8701m 54% 13958Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 8289m 52% 13980Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 8668m 54% 13562Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 457m 2% 6147Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4342m 27% 5618Mi 9% 06:31:34 DEBUG --- stderr --- 06:31:34 DEBUG 06:32:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813540" 06:32:20 INFO [http_cmd]: http status code OK 06:32:20 DEBUG --- status code --- 06:32:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:20 DEBUG --- http response --- 06:32:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 06:32:20 DEBUG ----- output ----- 06:32:20 DEBUG { 06:32:20 DEBUG "status": "success", 06:32:20 DEBUG "data": { 06:32:20 DEBUG "resultType": "vector", 06:32:20 DEBUG "result": [ 06:32:20 DEBUG { 06:32:20 DEBUG "metric": { 06:32:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG [...] 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 1712813540, 06:32:20 DEBUG "5422348.251293423" 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG } 06:32:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813540" 06:32:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813540" 06:32:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712813540" 06:32:20 INFO [http_cmd]: http status code OK 06:32:20 DEBUG --- status code --- 06:32:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:20 DEBUG --- http response --- 06:32:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:32:20 DEBUG ----- output ----- 06:32:20 DEBUG { 06:32:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813540" 06:32:20 DEBUG "status": "success", 06:32:20 DEBUG "data": { 06:32:20 DEBUG "resultType": "vector", 06:32:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813540" 06:32:20 DEBUG "result": [ 06:32:20 DEBUG { 06:32:20 DEBUG "metric": { 06:32:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG [...] 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 1712813540, 06:32:20 DEBUG "1357.3333333333333" 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG } 06:32:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813540" 06:32:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712813540" 06:32:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813540" 06:32:20 INFO [http_cmd]: http status code OK 06:32:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712813540" 06:32:20 DEBUG --- status code --- 06:32:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:20 DEBUG --- http response --- 06:32:20 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:32:20 INFO [http_cmd]: http status code OK 06:32:20 INFO [http_cmd]: http status code OK 06:32:20 DEBUG --- status code --- 06:32:20 DEBUG --- status code --- 06:32:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:20 DEBUG --- http response --- 06:32:20 DEBUG --- http response --- 06:32:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:32:20 DEBUG ----- output ----- 06:32:20 DEBUG { 06:32:20 DEBUG "status": "success", 06:32:20 DEBUG "data": { 06:32:20 DEBUG "resultType": "vector", 06:32:20 DEBUG "result": [ 06:32:20 DEBUG { 06:32:20 DEBUG "metric": { 06:32:20 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG [...] 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 1712813540, 06:32:20 DEBUG "14129455104" 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG } 06:32:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:32:20 DEBUG ----- output ----- 06:32:20 DEBUG { 06:32:20 DEBUG "status": "success", 06:32:20 DEBUG "data": { 06:32:20 DEBUG "resultType": "vector", 06:32:20 DEBUG "result": [ 06:32:20 DEBUG { 06:32:20 DEBUG "metric": { 06:32:20 DEBUG "pod": "ds-idrepo-2" 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG [...] 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG "pod": "ds-cts-2" 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 1712813540, 06:32:20 DEBUG "29.366666666666667" 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG } 06:32:20 INFO [http_cmd]: http status code OK 06:32:20 DEBUG --- status code --- 06:32:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:20 DEBUG --- http response --- 06:32:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:32:20 DEBUG ----- output ----- 06:32:20 DEBUG { 06:32:20 DEBUG "status": "success", 06:32:20 DEBUG "data": { 06:32:20 DEBUG "resultType": "vector", 06:32:20 DEBUG "result": [ 06:32:20 DEBUG { 06:32:20 DEBUG "metric": { 06:32:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG [...] 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 1712813540, 06:32:20 DEBUG "0.021" 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG ] 06:32:20 INFO [http_cmd]: http status code OK 06:32:20 DEBUG } 06:32:20 INFO [http_cmd]: http status code OK 06:32:20 DEBUG --- status code --- 06:32:20 DEBUG } 06:32:20 DEBUG --- status code --- 06:32:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:20 INFO [http_cmd]: http status code OK 06:32:20 ERROR [http_cmd]: ERROR 06:32:20 DEBUG --- http response --- 06:32:20 DEBUG --- http response --- 06:32:20 DEBUG --- status code --- 06:32:20 ERROR --- status code --- 06:32:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:32:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 06:32:20 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:32:20 DEBUG ----- output ----- 06:32:20 DEBUG --- http response --- 06:32:20 DEBUG ----- output ----- 06:32:20 ERROR --- http response --- 06:32:20 DEBUG { 06:32:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:32:20 DEBUG { 06:32:20 DEBUG "status": "success", 06:32:20 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:32:20 DEBUG ----- output ----- 06:32:20 DEBUG "status": "success", 06:32:20 DEBUG "data": { 06:32:20 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:32:20 DEBUG { 06:32:20 DEBUG "data": { 06:32:20 DEBUG "resultType": "vector", 06:32:20 DEBUG "status": "success", 06:32:20 DEBUG "resultType": "vector", 06:32:20 DEBUG "result": [ 06:32:20 DEBUG "data": { 06:32:20 DEBUG "result": [ 06:32:20 DEBUG { 06:32:20 DEBUG "resultType": "vector", 06:32:20 DEBUG { 06:32:20 DEBUG "metric": { 06:32:20 DEBUG "result": [ 06:32:20 DEBUG "metric": { 06:32:20 DEBUG "pod": "ds-idrepo-2" 06:32:20 DEBUG { 06:32:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:32:20 DEBUG }, 06:32:20 DEBUG "metric": { 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG "pod": "ds-idrepo-2" 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 06:32:20 DEBUG }, 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG [...] 06:32:20 DEBUG 06:32:20 DEBUG [...] 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG [...] 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG "pod": "ds-cts-2" 06:32:20 DEBUG 06:32:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:32:20 DEBUG }, 06:32:20 DEBUG 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG "pod": "ds-idrepo-0" 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 1712813540, 06:32:20 DEBUG }, 06:32:20 DEBUG 1712813540, 06:32:20 DEBUG "0" 06:32:20 DEBUG "value": [ 06:32:20 DEBUG "0" 06:32:20 DEBUG ] 06:32:20 DEBUG 1712813540, 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG "0" 06:32:20 DEBUG } 06:32:20 DEBUG ] 06:32:20 DEBUG ] 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG } 06:32:20 DEBUG } 06:32:20 DEBUG } 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG } 06:32:20 DEBUG } 06:32:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813540" 06:32:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712813540" 06:32:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813540" 06:32:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813540" 06:32:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813540" 06:32:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813540" 06:32:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813540" 06:32:20 INFO [http_cmd]: http status code OK 06:32:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813540" 06:32:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813540" 06:32:20 DEBUG --- status code --- 06:32:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:20 INFO [http_cmd]: http status code OK 06:32:20 INFO [http_cmd]: http status code OK 06:32:20 INFO [http_cmd]: http status code OK 06:32:20 DEBUG --- http response --- 06:32:20 INFO [http_cmd]: http status code OK 06:32:20 DEBUG --- status code --- 06:32:20 DEBUG --- status code --- 06:32:20 DEBUG --- status code --- 06:32:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:32:20 DEBUG --- status code --- 06:32:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813540" 06:32:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:20 INFO [http_cmd]: http status code OK 06:32:20 INFO [http_cmd]: http status code OK 06:32:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:20 DEBUG ----- output ----- 06:32:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:20 DEBUG --- http response --- 06:32:20 DEBUG --- status code --- 06:32:20 INFO [http_cmd]: http status code OK 06:32:20 DEBUG --- status code --- 06:32:20 DEBUG --- http response --- 06:32:20 DEBUG --- http response --- 06:32:20 DEBUG { 06:32:20 DEBUG --- http response --- 06:32:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:32:20 DEBUG --- status code --- 06:32:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:32:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:32:20 DEBUG "status": "success", 06:32:20 DEBUG --- http response --- 06:32:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:32:20 DEBUG ----- output ----- 06:32:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:20 DEBUG --- http response --- 06:32:20 DEBUG ----- output ----- 06:32:20 DEBUG ----- output ----- 06:32:20 DEBUG "data": { 06:32:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:32:20 DEBUG ----- output ----- 06:32:20 DEBUG { 06:32:20 DEBUG --- http response --- 06:32:20 DEBUG { 06:32:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:32:20 DEBUG { 06:32:20 DEBUG "resultType": "vector", 06:32:20 DEBUG ----- output ----- 06:32:20 DEBUG { 06:32:20 DEBUG "status": "success", 06:32:20 INFO [http_cmd]: http status code OK 06:32:20 DEBUG "status": "success", 06:32:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:32:20 DEBUG ----- output ----- 06:32:20 DEBUG "status": "success", 06:32:20 DEBUG "result": [ 06:32:20 DEBUG { 06:32:20 DEBUG "status": "success", 06:32:20 DEBUG "data": { 06:32:20 DEBUG --- status code --- 06:32:20 DEBUG "data": { 06:32:20 DEBUG ----- output ----- 06:32:20 DEBUG { 06:32:20 DEBUG "data": { 06:32:20 DEBUG { 06:32:20 DEBUG "status": "success", 06:32:20 DEBUG "data": { 06:32:20 DEBUG "resultType": "vector", 06:32:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:20 DEBUG "resultType": "vector", 06:32:20 DEBUG { 06:32:20 DEBUG "status": "success", 06:32:20 DEBUG "resultType": "vector", 06:32:20 DEBUG "metric": { 06:32:20 DEBUG "data": { 06:32:20 DEBUG "resultType": "vector", 06:32:20 DEBUG "result": [ 06:32:20 DEBUG --- http response --- 06:32:20 DEBUG "result": [ 06:32:20 DEBUG "status": "success", 06:32:20 DEBUG "data": { 06:32:20 DEBUG "result": [ 06:32:20 DEBUG "pod": "ds-idrepo-2" 06:32:20 DEBUG "resultType": "vector", 06:32:20 DEBUG "result": [ 06:32:20 DEBUG { 06:32:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:32:20 DEBUG { 06:32:20 DEBUG "data": { 06:32:20 DEBUG "resultType": "vector", 06:32:20 DEBUG { 06:32:20 DEBUG }, 06:32:20 DEBUG "result": [ 06:32:20 DEBUG { 06:32:20 DEBUG "metric": { 06:32:20 DEBUG ----- output ----- 06:32:20 DEBUG "metric": { 06:32:20 DEBUG "resultType": "vector", 06:32:20 DEBUG "result": [ 06:32:20 DEBUG "metric": { 06:32:20 DEBUG "value": [ 06:32:20 DEBUG { 06:32:20 DEBUG "metric": { 06:32:20 DEBUG "instance": "10.142.0.107:9100" 06:32:20 DEBUG { 06:32:20 DEBUG "pod": "ds-idrepo-2" 06:32:20 DEBUG "result": [ 06:32:20 DEBUG { 06:32:20 DEBUG "pod": "ds-idrepo-2" 06:32:20 INFO [http_cmd]: http status code OK 06:32:20 DEBUG 06:32:20 DEBUG "metric": { 06:32:20 DEBUG "pod": "ds-idrepo-2" 06:32:20 DEBUG }, 06:32:20 DEBUG "status": "success", 06:32:20 DEBUG }, 06:32:20 DEBUG { 06:32:20 DEBUG "metric": { 06:32:20 DEBUG }, 06:32:20 DEBUG --- status code --- 06:32:20 DEBUG 06:32:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG "data": { 06:32:20 DEBUG "value": [ 06:32:20 DEBUG "metric": { 06:32:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:32:20 DEBUG "value": [ 06:32:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:20 DEBUG 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 06:32:20 DEBUG "resultType": "vector", 06:32:20 DEBUG 06:32:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:32:20 DEBUG }, 06:32:20 DEBUG 06:32:20 DEBUG --- http response --- 06:32:20 DEBUG [...] 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG "result": [ 06:32:20 DEBUG 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG { 06:32:20 DEBUG 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG ----- output ----- 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG [...] 06:32:20 DEBUG "metric": { 06:32:20 DEBUG [...] 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG [...] 06:32:20 DEBUG 06:32:20 DEBUG { 06:32:20 DEBUG 06:32:20 DEBUG [...] 06:32:20 DEBUG 06:32:20 DEBUG "instance": "10.142.0.107:9100" 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG "pod": "ds-idrepo-0" 06:32:20 DEBUG "status": "success", 06:32:20 DEBUG [...] 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG }, 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG [...] 06:32:20 DEBUG 06:32:20 DEBUG }, 06:32:20 DEBUG "data": { 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 06:32:20 DEBUG [...] 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG "value": [ 06:32:20 DEBUG "resultType": "vector", 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG "instance": "10.142.0.77:9100" 06:32:20 DEBUG 06:32:20 DEBUG "pod": "ds-cts-2" 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG "pod": "ds-cts-2" 06:32:20 DEBUG 1712813540, 06:32:20 DEBUG "result": [ 06:32:20 DEBUG 06:32:20 DEBUG "pod": "ds-idrepo-0" 06:32:20 DEBUG }, 06:32:20 DEBUG 06:32:20 DEBUG }, 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG }, 06:32:20 DEBUG "0" 06:32:20 DEBUG { 06:32:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 06:32:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:32:20 DEBUG "value": [ 06:32:20 DEBUG ] 06:32:20 DEBUG "metric": { 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 1712813540, 06:32:20 DEBUG [...] 06:32:20 DEBUG 1712813540, 06:32:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:32:20 DEBUG }, 06:32:20 DEBUG 1712813540, 06:32:20 DEBUG } 06:32:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 1712813540, 06:32:20 DEBUG "52936330240" 06:32:20 DEBUG 06:32:20 DEBUG "0" 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG "0" 06:32:20 DEBUG ] 06:32:20 DEBUG }, 06:32:20 DEBUG 1712813540, 06:32:20 DEBUG "0" 06:32:20 DEBUG ] 06:32:20 DEBUG 06:32:20 DEBUG ] 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 1712813540, 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG "value": [ 06:32:20 DEBUG "0" 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG 06:32:20 DEBUG } 06:32:20 DEBUG 1712813540, 06:32:20 DEBUG "1361" 06:32:20 DEBUG } 06:32:20 DEBUG } 06:32:20 DEBUG 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG ] 06:32:20 DEBUG "instance": "10.142.0.77:9100" 06:32:20 DEBUG ] 06:32:20 DEBUG "1356.8333333333333" 06:32:20 DEBUG ] 06:32:20 DEBUG ] 06:32:20 DEBUG 06:32:20 DEBUG } 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG } 06:32:20 DEBUG 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG } 06:32:20 DEBUG } 06:32:20 DEBUG 1712813540, 06:32:20 DEBUG } 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG [...] 06:32:20 DEBUG } 06:32:20 DEBUG } 06:32:20 DEBUG } 06:32:20 DEBUG "82809137.5" 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG 06:32:20 DEBUG } 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG } 06:32:20 DEBUG 06:32:20 DEBUG } 06:32:20 DEBUG } 06:32:20 DEBUG 06:32:20 DEBUG ] 06:32:20 DEBUG "pod": "ds-cts-0" 06:32:20 DEBUG } 06:32:20 DEBUG }, 06:32:20 DEBUG } 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 1712813540, 06:32:20 DEBUG "0" 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG } 06:32:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813540" 06:32:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813540" 06:32:20 INFO [http_cmd]: http status code OK 06:32:20 DEBUG --- status code --- 06:32:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:20 DEBUG --- http response --- 06:32:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:32:20 DEBUG ----- output ----- 06:32:20 DEBUG { 06:32:20 DEBUG "status": "success", 06:32:20 DEBUG "data": { 06:32:20 DEBUG "resultType": "vector", 06:32:20 DEBUG "result": [ 06:32:20 DEBUG { 06:32:20 DEBUG "metric": { 06:32:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG [...] 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 1712813540, 06:32:20 DEBUG "245023.5" 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG } 06:32:20 INFO [http_cmd]: http status code OK 06:32:20 DEBUG --- status code --- 06:32:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:20 DEBUG --- http response --- 06:32:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:32:20 DEBUG ----- output ----- 06:32:20 DEBUG { 06:32:20 DEBUG "status": "success", 06:32:20 DEBUG "data": { 06:32:20 DEBUG "resultType": "vector", 06:32:20 DEBUG "result": [ 06:32:20 DEBUG { 06:32:20 DEBUG "metric": { 06:32:20 DEBUG "instance": "10.142.0.107:9100" 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG [...] 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG "instance": "10.142.0.77:9100" 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 1712813540, 06:32:20 DEBUG "0.00033333333333350387" 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG } 06:32:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813540" 06:32:20 INFO [http_cmd]: http status code OK 06:32:20 DEBUG --- status code --- 06:32:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:20 DEBUG --- http response --- 06:32:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:32:20 DEBUG ----- output ----- 06:32:20 DEBUG { 06:32:20 DEBUG "status": "success", 06:32:20 DEBUG "data": { 06:32:20 DEBUG "resultType": "vector", 06:32:20 DEBUG "result": [ 06:32:20 DEBUG { 06:32:20 DEBUG "metric": { 06:32:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG [...] 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 1712813540, 06:32:20 DEBUG "1361.7" 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG } 06:32:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712813540" 06:32:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813540" 06:32:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813540" 06:32:20 INFO [http_cmd]: http status code OK 06:32:20 DEBUG --- status code --- 06:32:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:20 DEBUG --- http response --- 06:32:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:32:20 DEBUG ----- output ----- 06:32:20 DEBUG { 06:32:20 DEBUG "status": "success", 06:32:20 DEBUG "data": { 06:32:20 DEBUG "resultType": "vector", 06:32:20 DEBUG "result": [ 06:32:20 DEBUG { 06:32:20 DEBUG "metric": { 06:32:20 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG [...] 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 1712813540, 06:32:20 DEBUG "4.177734779580815" 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG } 06:32:20 INFO [http_cmd]: http status code OK 06:32:20 DEBUG --- status code --- 06:32:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:20 DEBUG --- http response --- 06:32:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:32:20 DEBUG ----- output ----- 06:32:20 DEBUG { 06:32:20 DEBUG "status": "success", 06:32:20 DEBUG "data": { 06:32:20 DEBUG "resultType": "vector", 06:32:20 DEBUG "result": [ 06:32:20 DEBUG { 06:32:20 DEBUG "metric": { 06:32:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG [...] 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG "pod": "ds-cts-0" 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 1712813540, 06:32:20 DEBUG "0" 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG } 06:32:20 INFO [http_cmd]: http status code OK 06:32:20 DEBUG --- status code --- 06:32:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:20 DEBUG --- http response --- 06:32:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:32:20 DEBUG ----- output ----- 06:32:20 DEBUG { 06:32:20 DEBUG "status": "success", 06:32:20 DEBUG "data": { 06:32:20 DEBUG "resultType": "vector", 06:32:20 DEBUG "result": [ 06:32:20 DEBUG { 06:32:20 DEBUG "metric": { 06:32:20 DEBUG "instance": "10.142.0.107:9100" 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG [...] 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG "instance": "10.142.0.77:9100" 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 1712813540, 06:32:20 DEBUG "0.0023333333333335794" 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG } 06:32:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813540" 06:32:20 INFO [http_cmd]: http status code OK 06:32:20 DEBUG --- status code --- 06:32:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:32:20 DEBUG --- http response --- 06:32:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:32:20 DEBUG ----- output ----- 06:32:20 DEBUG { 06:32:20 DEBUG "status": "success", 06:32:20 DEBUG "data": { 06:32:20 DEBUG "resultType": "vector", 06:32:20 DEBUG "result": [ 06:32:20 DEBUG { 06:32:20 DEBUG "metric": { 06:32:20 DEBUG "instance": "10.142.0.107:9100" 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG [...] 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG 06:32:20 DEBUG "instance": "10.142.0.77:9100" 06:32:20 DEBUG }, 06:32:20 DEBUG "value": [ 06:32:20 DEBUG 1712813540, 06:32:20 DEBUG "82806639.93333334" 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG ] 06:32:20 DEBUG } 06:32:20 DEBUG } 06:32:31 INFO 06:32:31 INFO [loop_until]: kubectl --namespace=xlou top pods 06:32:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:32:31 INFO [loop_until]: OK (rc = 0) 06:32:31 DEBUG --- stdout --- 06:32:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 8980m 8504Mi am-5b5bd8567b-8h2w4 8772m 8562Mi am-5b5bd8567b-qlkw6 7380m 8458Mi ds-cts-0 8222m 13387Mi ds-cts-1 8651m 12546Mi ds-cts-2 8245m 13390Mi ds-idrepo-0 11623m 5497Mi ds-idrepo-1 349m 5480Mi ds-idrepo-2 363m 5783Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 4m 1010Mi idm-668d898cfd-st2t7 3m 1135Mi lodemon-ccc964465-p7gdm 2m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4113m 1844Mi 06:32:31 DEBUG --- stderr --- 06:32:31 DEBUG 06:32:34 INFO 06:32:34 INFO [loop_until]: kubectl --namespace=xlou top node 06:32:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:32:34 INFO [loop_until]: OK (rc = 0) 06:32:34 DEBUG --- stdout --- 06:32:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 2731Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 61m 0% 2773Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7832m 26% 10113Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10700m 35% 2971Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9553m 31% 10089Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 142m 0% 3846Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9678m 32% 10187Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 424m 2% 6454Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11557m 72% 6189Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8463m 53% 13969Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 7915m 49% 13988Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 8836m 55% 13137Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 398m 2% 6153Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4264m 26% 5619Mi 9% 06:32:34 DEBUG --- stderr --- 06:32:34 DEBUG 06:33:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813600" 06:33:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813600" 06:33:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712813600" 06:33:20 INFO [http_cmd]: http status code OK 06:33:20 INFO [http_cmd]: http status code OK 06:33:20 DEBUG --- status code --- 06:33:20 DEBUG --- status code --- 06:33:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:20 DEBUG --- http response --- 06:33:20 DEBUG --- http response --- 06:33:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:33:20 DEBUG ----- output ----- 06:33:20 DEBUG { 06:33:20 DEBUG "status": "success", 06:33:20 DEBUG "data": { 06:33:20 DEBUG "resultType": "vector", 06:33:20 DEBUG "result": [ 06:33:20 DEBUG { 06:33:20 DEBUG "metric": { 06:33:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:33:20 DEBUG }, 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG [...] 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:33:20 DEBUG }, 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 1712813600, 06:33:20 DEBUG "1373.6333333333332" 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG } 06:33:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:33:20 DEBUG ----- output ----- 06:33:20 DEBUG { 06:33:20 DEBUG "status": "success", 06:33:20 DEBUG "data": { 06:33:20 DEBUG "resultType": "vector", 06:33:20 DEBUG "result": [ 06:33:20 DEBUG { 06:33:20 DEBUG "metric": { 06:33:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:33:20 DEBUG }, 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG [...] 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:33:20 DEBUG }, 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 1712813600, 06:33:20 DEBUG "0" 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG } 06:33:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813600" 06:33:20 INFO [http_cmd]: http status code OK 06:33:20 DEBUG --- status code --- 06:33:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:20 DEBUG --- http response --- 06:33:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:33:20 DEBUG ----- output ----- 06:33:20 DEBUG { 06:33:20 DEBUG "status": "success", 06:33:20 DEBUG "data": { 06:33:20 DEBUG "resultType": "vector", 06:33:20 DEBUG "result": [ 06:33:20 DEBUG { 06:33:20 DEBUG "metric": { 06:33:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:33:20 DEBUG }, 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG [...] 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:33:20 DEBUG }, 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 1712813600, 06:33:20 DEBUG "0.021" 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG } 06:33:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813600" 06:33:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813600" 06:33:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813600" 06:33:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813600" 06:33:20 INFO [http_cmd]: http status code OK 06:33:20 DEBUG --- status code --- 06:33:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:20 DEBUG --- http response --- 06:33:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:33:20 DEBUG ----- output ----- 06:33:20 DEBUG { 06:33:20 DEBUG "status": "success", 06:33:20 DEBUG "data": { 06:33:20 DEBUG "resultType": "vector", 06:33:20 DEBUG "result": [ 06:33:20 DEBUG { 06:33:20 DEBUG "metric": { 06:33:20 DEBUG "instance": "10.142.0.107:9100" 06:33:20 DEBUG }, 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG [...] 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG "instance": "10.142.0.77:9100" 06:33:20 DEBUG }, 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 1712813600, 06:33:20 DEBUG "52937743360" 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG } 06:33:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712813600" 06:33:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813600" 06:33:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712813600" 06:33:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813600" 06:33:20 INFO [http_cmd]: http status code OK 06:33:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712813600" 06:33:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813600" 06:33:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813600" 06:33:20 DEBUG --- status code --- 06:33:20 INFO [http_cmd]: http status code OK 06:33:20 INFO [http_cmd]: http status code OK 06:33:20 INFO [http_cmd]: http status code OK 06:33:20 INFO [http_cmd]: http status code OK 06:33:20 DEBUG --- status code --- 06:33:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813600" 06:33:20 DEBUG --- status code --- 06:33:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813600" 06:33:20 DEBUG --- status code --- 06:33:20 DEBUG --- status code --- 06:33:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:20 INFO [http_cmd]: http status code OK 06:33:20 INFO [http_cmd]: http status code OK 06:33:20 INFO [http_cmd]: http status code OK 06:33:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:20 ERROR [http_cmd]: ERROR 06:33:20 DEBUG --- http response --- 06:33:20 DEBUG --- http response --- 06:33:20 DEBUG --- status code --- 06:33:20 DEBUG --- status code --- 06:33:20 DEBUG --- status code --- 06:33:20 DEBUG --- http response --- 06:33:20 DEBUG --- http response --- 06:33:20 DEBUG --- http response --- 06:33:20 ERROR --- status code --- 06:33:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:33:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:33:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:20 INFO [http_cmd]: http status code OK 06:33:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:20 INFO [http_cmd]: http status code OK 06:33:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:33:20 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:33:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:33:20 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:33:20 DEBUG ----- output ----- 06:33:20 DEBUG --- http response --- 06:33:20 DEBUG ----- output ----- 06:33:20 DEBUG --- http response --- 06:33:20 DEBUG --- status code --- 06:33:20 DEBUG --- http response --- 06:33:20 DEBUG --- status code --- 06:33:20 DEBUG ----- output ----- 06:33:20 DEBUG ----- output ----- 06:33:20 INFO [http_cmd]: http status code OK 06:33:20 ERROR --- http response --- 06:33:20 DEBUG { 06:33:20 DEBUG { 06:33:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:33:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:33:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:33:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:20 DEBUG { 06:33:20 DEBUG { 06:33:20 DEBUG --- status code --- 06:33:20 DEBUG "status": "success", 06:33:20 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:33:20 DEBUG "status": "success", 06:33:20 DEBUG ----- output ----- 06:33:20 DEBUG --- http response --- 06:33:20 DEBUG ----- output ----- 06:33:20 DEBUG ----- output ----- 06:33:20 DEBUG --- http response --- 06:33:20 DEBUG "status": "success", 06:33:20 DEBUG "status": "success", 06:33:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:20 DEBUG "data": { 06:33:20 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:33:20 DEBUG "data": { 06:33:20 DEBUG { 06:33:20 DEBUG { 06:33:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:33:20 DEBUG { 06:33:20 DEBUG "data": { 06:33:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:33:20 DEBUG "data": { 06:33:20 INFO [http_cmd]: http status code OK 06:33:20 DEBUG --- http response --- 06:33:20 DEBUG "resultType": "vector", 06:33:20 DEBUG "resultType": "vector", 06:33:20 DEBUG "status": "success", 06:33:20 DEBUG "status": "success", 06:33:20 DEBUG ----- output ----- 06:33:20 DEBUG "status": "success", 06:33:20 DEBUG "resultType": "vector", 06:33:20 DEBUG ----- output ----- 06:33:20 DEBUG "resultType": "vector", 06:33:20 DEBUG --- status code --- 06:33:20 DEBUG "result": [ 06:33:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:33:20 DEBUG "result": [ 06:33:20 DEBUG "data": { 06:33:20 DEBUG "data": { 06:33:20 DEBUG { 06:33:20 DEBUG "data": { 06:33:20 DEBUG "result": [ 06:33:20 DEBUG { 06:33:20 DEBUG "result": [ 06:33:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:20 DEBUG { 06:33:20 DEBUG ----- output ----- 06:33:20 DEBUG { 06:33:20 DEBUG "resultType": "vector", 06:33:20 DEBUG "resultType": "vector", 06:33:20 DEBUG "status": "success", 06:33:20 DEBUG "resultType": "vector", 06:33:20 DEBUG { 06:33:20 DEBUG "status": "success", 06:33:20 DEBUG { 06:33:20 DEBUG --- http response --- 06:33:20 DEBUG "metric": { 06:33:20 DEBUG { 06:33:20 DEBUG "metric": { 06:33:20 DEBUG "result": [ 06:33:20 DEBUG "result": [ 06:33:20 DEBUG "data": { 06:33:20 DEBUG "result": [ 06:33:20 DEBUG "metric": { 06:33:20 DEBUG "data": { 06:33:20 DEBUG "metric": { 06:33:20 DEBUG "pod": "ds-idrepo-2" 06:33:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:33:20 DEBUG "status": "success", 06:33:20 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:33:20 DEBUG { 06:33:20 DEBUG { 06:33:20 DEBUG "resultType": "vector", 06:33:20 DEBUG { 06:33:20 DEBUG "pod": "ds-idrepo-2" 06:33:20 DEBUG "resultType": "vector", 06:33:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:33:20 DEBUG }, 06:33:20 DEBUG ----- output ----- 06:33:20 DEBUG "data": { 06:33:20 DEBUG }, 06:33:20 DEBUG "metric": { 06:33:20 DEBUG "metric": { 06:33:20 DEBUG "result": [ 06:33:20 DEBUG "metric": { 06:33:20 DEBUG }, 06:33:20 DEBUG "result": [ 06:33:20 DEBUG }, 06:33:20 DEBUG "value": [ 06:33:20 DEBUG { 06:33:20 DEBUG "resultType": "vector", 06:33:20 DEBUG "value": [ 06:33:20 DEBUG "pod": "ds-idrepo-2" 06:33:20 DEBUG "pod": "ds-idrepo-2" 06:33:20 DEBUG { 06:33:20 DEBUG "pod": "ds-idrepo-2" 06:33:20 DEBUG "value": [ 06:33:20 DEBUG { 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 06:33:20 DEBUG "status": "success", 06:33:20 DEBUG "result": [ 06:33:20 DEBUG 06:33:20 DEBUG }, 06:33:20 DEBUG }, 06:33:20 DEBUG "metric": { 06:33:20 DEBUG }, 06:33:20 DEBUG 06:33:20 DEBUG "metric": { 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG "data": { 06:33:20 DEBUG { 06:33:20 DEBUG 06:33:20 DEBUG "value": [ 06:33:20 DEBUG "value": [ 06:33:20 DEBUG "instance": "10.142.0.107:9100" 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 06:33:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG "resultType": "vector", 06:33:20 DEBUG "metric": { 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG }, 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG }, 06:33:20 DEBUG 06:33:20 DEBUG [...] 06:33:20 DEBUG "result": [ 06:33:20 DEBUG "pod": "ds-idrepo-2" 06:33:20 DEBUG [...] 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 06:33:20 DEBUG [...] 06:33:20 DEBUG "value": [ 06:33:20 DEBUG [...] 06:33:20 DEBUG 06:33:20 DEBUG { 06:33:20 DEBUG }, 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG "metric": { 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 06:33:20 DEBUG [...] 06:33:20 DEBUG [...] 06:33:20 DEBUG 06:33:20 DEBUG [...] 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG "pod": "ds-idrepo-0" 06:33:20 DEBUG }, 06:33:20 DEBUG 06:33:20 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG [...] 06:33:20 DEBUG 06:33:20 DEBUG "pod": "ds-cts-2" 06:33:20 DEBUG [...] 06:33:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:33:20 DEBUG }, 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 06:33:20 DEBUG }, 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG }, 06:33:20 DEBUG 06:33:20 DEBUG }, 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 06:33:20 DEBUG [...] 06:33:20 DEBUG "value": [ 06:33:20 DEBUG "pod": "ds-idrepo-0" 06:33:20 DEBUG "pod": "ds-cts-2" 06:33:20 DEBUG 06:33:20 DEBUG "pod": "ds-cts-2" 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 1712813600, 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 1712813600, 06:33:20 DEBUG }, 06:33:20 DEBUG }, 06:33:20 DEBUG 06:33:20 DEBUG }, 06:33:20 DEBUG 1712813600, 06:33:20 DEBUG 06:33:20 DEBUG 1712813600, 06:33:20 DEBUG "0" 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG "13479874560" 06:33:20 DEBUG "value": [ 06:33:20 DEBUG "value": [ 06:33:20 DEBUG "instance": "10.142.0.77:9100" 06:33:20 DEBUG "value": [ 06:33:20 DEBUG "0.00029593888888888886" 06:33:20 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:33:20 DEBUG "0" 06:33:20 DEBUG ] 06:33:20 DEBUG [...] 06:33:20 DEBUG 06:33:20 DEBUG ] 06:33:20 DEBUG 1712813600, 06:33:20 DEBUG 1712813600, 06:33:20 DEBUG }, 06:33:20 DEBUG 1712813600, 06:33:20 DEBUG ] 06:33:20 DEBUG }, 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG 06:33:20 DEBUG "pod": "ds-idrepo-0" 06:33:20 DEBUG } 06:33:20 DEBUG "0" 06:33:20 DEBUG "29.133333333333333" 06:33:20 DEBUG "value": [ 06:33:20 DEBUG "0" 06:33:20 DEBUG } 06:33:20 DEBUG "value": [ 06:33:20 DEBUG } 06:33:20 DEBUG ] 06:33:20 DEBUG 06:33:20 DEBUG }, 06:33:20 DEBUG ] 06:33:20 DEBUG ] 06:33:20 DEBUG ] 06:33:20 DEBUG 1712813600, 06:33:20 DEBUG ] 06:33:20 DEBUG ] 06:33:20 DEBUG 1712813600, 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG 06:33:20 DEBUG "value": [ 06:33:20 DEBUG } 06:33:20 DEBUG } 06:33:20 DEBUG } 06:33:20 DEBUG "83128112.63333334" 06:33:20 DEBUG } 06:33:20 DEBUG } 06:33:20 DEBUG "0" 06:33:20 DEBUG } 06:33:20 DEBUG } 06:33:20 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:33:20 DEBUG 1712813600, 06:33:20 DEBUG } 06:33:20 DEBUG ] 06:33:20 DEBUG ] 06:33:20 DEBUG ] 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG }, 06:33:20 DEBUG "0" 06:33:20 DEBUG } 06:33:20 DEBUG } 06:33:20 DEBUG } 06:33:20 DEBUG } 06:33:20 DEBUG } 06:33:20 DEBUG "value": [ 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG } 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG ] 06:33:20 DEBUG 1712813600, 06:33:20 DEBUG } 06:33:20 DEBUG } 06:33:20 DEBUG } 06:33:20 DEBUG "43523.44710497044" 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG } 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG } 06:33:20 DEBUG } 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG } 06:33:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813600" 06:33:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813600" 06:33:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813600" 06:33:20 INFO [http_cmd]: http status code OK 06:33:20 DEBUG --- status code --- 06:33:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:20 DEBUG --- http response --- 06:33:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:33:20 DEBUG ----- output ----- 06:33:20 DEBUG { 06:33:20 DEBUG "status": "success", 06:33:20 DEBUG "data": { 06:33:20 DEBUG "resultType": "vector", 06:33:20 DEBUG "result": [ 06:33:20 DEBUG { 06:33:20 DEBUG "metric": { 06:33:20 DEBUG "pod": "ds-idrepo-2" 06:33:20 DEBUG }, 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG [...] 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG "pod": "ds-cts-2" 06:33:20 DEBUG }, 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 1712813600, 06:33:20 DEBUG "0" 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG } 06:33:20 INFO [http_cmd]: http status code OK 06:33:20 INFO [http_cmd]: http status code OK 06:33:20 DEBUG --- status code --- 06:33:20 DEBUG --- status code --- 06:33:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:20 DEBUG --- http response --- 06:33:20 DEBUG --- http response --- 06:33:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:33:20 DEBUG ----- output ----- 06:33:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:33:20 DEBUG { 06:33:20 DEBUG ----- output ----- 06:33:20 DEBUG "status": "success", 06:33:20 DEBUG { 06:33:20 DEBUG "data": { 06:33:20 DEBUG "status": "success", 06:33:20 DEBUG "resultType": "vector", 06:33:20 DEBUG "data": { 06:33:20 DEBUG "result": [ 06:33:20 DEBUG "resultType": "vector", 06:33:20 DEBUG { 06:33:20 DEBUG "result": [ 06:33:20 DEBUG "metric": { 06:33:20 DEBUG { 06:33:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:33:20 DEBUG "metric": { 06:33:20 DEBUG }, 06:33:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:33:20 DEBUG "value": [ 06:33:20 DEBUG }, 06:33:20 DEBUG 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG [...] 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG [...] 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:33:20 DEBUG 06:33:20 DEBUG }, 06:33:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:33:20 DEBUG "value": [ 06:33:20 DEBUG }, 06:33:20 DEBUG 1712813600, 06:33:20 DEBUG "value": [ 06:33:20 DEBUG "1373.6" 06:33:20 DEBUG 1712813600, 06:33:20 DEBUG ] 06:33:20 DEBUG "1369.1" 06:33:20 DEBUG } 06:33:20 DEBUG ] 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG } 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG } 06:33:20 DEBUG } 06:33:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813600" 06:33:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813600" 06:33:20 INFO [http_cmd]: http status code OK 06:33:20 DEBUG --- status code --- 06:33:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:20 DEBUG --- http response --- 06:33:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:33:20 DEBUG ----- output ----- 06:33:20 DEBUG { 06:33:20 DEBUG "status": "success", 06:33:20 DEBUG "data": { 06:33:20 DEBUG "resultType": "vector", 06:33:20 DEBUG "result": [ 06:33:20 DEBUG { 06:33:20 DEBUG "metric": { 06:33:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:33:20 DEBUG }, 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG [...] 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:33:20 DEBUG }, 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 1712813600, 06:33:20 DEBUG "244945" 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG } 06:33:20 INFO [http_cmd]: http status code OK 06:33:20 DEBUG --- status code --- 06:33:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:20 DEBUG --- http response --- 06:33:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813600" 06:33:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:33:20 DEBUG ----- output ----- 06:33:20 DEBUG { 06:33:20 DEBUG "status": "success", 06:33:20 DEBUG "data": { 06:33:20 DEBUG "resultType": "vector", 06:33:20 DEBUG "result": [ 06:33:20 DEBUG { 06:33:20 DEBUG "metric": { 06:33:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:33:20 DEBUG }, 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG [...] 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:33:20 DEBUG }, 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 1712813600, 06:33:20 DEBUG "4.126632419823161" 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG } 06:33:20 INFO [http_cmd]: http status code OK 06:33:20 DEBUG --- status code --- 06:33:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:20 DEBUG --- http response --- 06:33:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:33:20 DEBUG ----- output ----- 06:33:20 DEBUG { 06:33:20 DEBUG "status": "success", 06:33:20 DEBUG "data": { 06:33:20 DEBUG "resultType": "vector", 06:33:20 DEBUG "result": [ 06:33:20 DEBUG { 06:33:20 DEBUG "metric": { 06:33:20 DEBUG "instance": "10.142.0.107:9100" 06:33:20 DEBUG }, 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG [...] 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG "instance": "10.142.0.77:9100" 06:33:20 DEBUG }, 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 1712813600, 06:33:20 DEBUG "0.0006666666666666524" 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG } 06:33:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813600" 06:33:20 INFO [http_cmd]: http status code OK 06:33:20 DEBUG --- status code --- 06:33:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:20 DEBUG --- http response --- 06:33:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:33:20 DEBUG ----- output ----- 06:33:20 DEBUG { 06:33:20 DEBUG "status": "success", 06:33:20 DEBUG "data": { 06:33:20 DEBUG "resultType": "vector", 06:33:20 DEBUG "result": [ 06:33:20 DEBUG { 06:33:20 DEBUG "metric": { 06:33:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:33:20 DEBUG }, 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG [...] 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:33:20 DEBUG }, 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 1712813600, 06:33:20 DEBUG "1373.7" 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG } 06:33:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712813600" 06:33:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813600" 06:33:20 INFO [http_cmd]: http status code OK 06:33:20 DEBUG --- status code --- 06:33:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:20 DEBUG --- http response --- 06:33:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:33:20 DEBUG ----- output ----- 06:33:20 DEBUG { 06:33:20 DEBUG "status": "success", 06:33:20 DEBUG "data": { 06:33:20 DEBUG "resultType": "vector", 06:33:20 DEBUG "result": [ 06:33:20 DEBUG { 06:33:20 DEBUG "metric": { 06:33:20 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:33:20 DEBUG }, 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG [...] 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:33:20 DEBUG }, 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 1712813600, 06:33:20 DEBUG "4.175340397770528" 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG } 06:33:20 INFO [http_cmd]: http status code OK 06:33:20 DEBUG --- status code --- 06:33:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:20 DEBUG --- http response --- 06:33:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:33:20 DEBUG ----- output ----- 06:33:20 DEBUG { 06:33:20 DEBUG "status": "success", 06:33:20 DEBUG "data": { 06:33:20 DEBUG "resultType": "vector", 06:33:20 DEBUG "result": [ 06:33:20 DEBUG { 06:33:20 DEBUG "metric": { 06:33:20 DEBUG "instance": "10.142.0.107:9100" 06:33:20 DEBUG }, 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG [...] 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG "instance": "10.142.0.77:9100" 06:33:20 DEBUG }, 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 1712813600, 06:33:20 DEBUG "0.002133333333333335" 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG } 06:33:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813600" 06:33:20 INFO [http_cmd]: http status code OK 06:33:20 DEBUG --- status code --- 06:33:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:33:20 DEBUG --- http response --- 06:33:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:33:20 DEBUG ----- output ----- 06:33:20 DEBUG { 06:33:20 DEBUG "status": "success", 06:33:20 DEBUG "data": { 06:33:20 DEBUG "resultType": "vector", 06:33:20 DEBUG "result": [ 06:33:20 DEBUG { 06:33:20 DEBUG "metric": { 06:33:20 DEBUG "instance": "10.142.0.107:9100" 06:33:20 DEBUG }, 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG [...] 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG 06:33:20 DEBUG "instance": "10.142.0.77:9100" 06:33:20 DEBUG }, 06:33:20 DEBUG "value": [ 06:33:20 DEBUG 1712813600, 06:33:20 DEBUG "83126004.5" 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG ] 06:33:20 DEBUG } 06:33:20 DEBUG } 06:33:31 INFO 06:33:31 INFO [loop_until]: kubectl --namespace=xlou top pods 06:33:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:33:31 INFO [loop_until]: OK (rc = 0) 06:33:31 DEBUG --- stdout --- 06:33:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 8996m 8504Mi am-5b5bd8567b-8h2w4 8910m 8563Mi am-5b5bd8567b-qlkw6 7595m 8459Mi ds-cts-0 8354m 12903Mi ds-cts-1 8661m 13375Mi ds-cts-2 8787m 13017Mi ds-idrepo-0 11268m 5497Mi ds-idrepo-1 343m 5479Mi ds-idrepo-2 348m 5783Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 4m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 3m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4183m 1844Mi 06:33:31 DEBUG --- stderr --- 06:33:31 DEBUG 06:33:34 INFO 06:33:34 INFO [loop_until]: kubectl --namespace=xlou top node 06:33:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:33:34 INFO [loop_until]: OK (rc = 0) 06:33:34 DEBUG --- stdout --- 06:33:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 2723Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 63m 0% 2773Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7983m 26% 10120Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10421m 34% 2980Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9504m 31% 10079Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 126m 0% 3858Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9644m 32% 10187Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 396m 2% 6458Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11170m 70% 6188Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 9015m 56% 13619Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 8151m 51% 13576Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 8856m 55% 13887Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 400m 2% 6151Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4263m 26% 5616Mi 9% 06:33:34 DEBUG --- stderr --- 06:33:34 DEBUG 06:34:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813660" 06:34:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813660" 06:34:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813660" 06:34:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712813660" 06:34:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813660" 06:34:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813660" 06:34:20 INFO [http_cmd]: http status code OK 06:34:20 INFO [http_cmd]: http status code OK 06:34:20 DEBUG --- status code --- 06:34:20 DEBUG --- status code --- 06:34:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:20 DEBUG --- http response --- 06:34:20 DEBUG --- http response --- 06:34:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:34:20 DEBUG ----- output ----- 06:34:20 DEBUG { 06:34:20 DEBUG "status": "success", 06:34:20 DEBUG "data": { 06:34:20 DEBUG "resultType": "vector", 06:34:20 DEBUG "result": [ 06:34:20 DEBUG { 06:34:20 DEBUG "metric": { 06:34:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:34:20 DEBUG }, 06:34:20 DEBUG "value": [ 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG [...] 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:34:20 DEBUG }, 06:34:20 DEBUG "value": [ 06:34:20 DEBUG 1712813660, 06:34:20 DEBUG "1349.2" 06:34:20 DEBUG ] 06:34:20 DEBUG } 06:34:20 DEBUG ] 06:34:20 DEBUG } 06:34:20 DEBUG } 06:34:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 06:34:20 DEBUG ----- output ----- 06:34:20 DEBUG { 06:34:20 DEBUG "status": "success", 06:34:20 DEBUG "data": { 06:34:20 DEBUG "resultType": "vector", 06:34:20 DEBUG "result": [ 06:34:20 DEBUG { 06:34:20 DEBUG "metric": { 06:34:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:34:20 DEBUG }, 06:34:20 DEBUG "value": [ 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG [...] 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG "pod": "ds-cts-1" 06:34:20 DEBUG }, 06:34:20 DEBUG "value": [ 06:34:20 DEBUG 1712813660, 06:34:20 DEBUG "20314403.30513476" 06:34:20 DEBUG ] 06:34:20 DEBUG } 06:34:20 DEBUG ] 06:34:20 DEBUG } 06:34:20 DEBUG } 06:34:20 INFO [http_cmd]: http status code OK 06:34:20 DEBUG --- status code --- 06:34:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:20 DEBUG --- http response --- 06:34:20 INFO [http_cmd]: http status code OK 06:34:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:34:20 DEBUG --- status code --- 06:34:20 INFO [http_cmd]: http status code OK 06:34:20 DEBUG ----- output ----- 06:34:20 INFO [http_cmd]: http status code OK 06:34:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:20 DEBUG --- status code --- 06:34:20 DEBUG { 06:34:20 DEBUG --- status code --- 06:34:20 DEBUG --- http response --- 06:34:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:20 DEBUG "status": "success", 06:34:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:20 DEBUG --- http response --- 06:34:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:34:20 DEBUG "data": { 06:34:20 DEBUG --- http response --- 06:34:20 DEBUG ----- output ----- 06:34:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:34:20 DEBUG "resultType": "vector", 06:34:20 DEBUG { 06:34:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:34:20 DEBUG ----- output ----- 06:34:20 DEBUG "result": [ 06:34:20 DEBUG "status": "success", 06:34:20 DEBUG ----- output ----- 06:34:20 DEBUG { 06:34:20 DEBUG { 06:34:20 DEBUG "data": { 06:34:20 DEBUG { 06:34:20 DEBUG "status": "success", 06:34:20 DEBUG "metric": { 06:34:20 DEBUG "resultType": "vector", 06:34:20 DEBUG "status": "success", 06:34:20 DEBUG "data": { 06:34:20 DEBUG "pod": "ds-idrepo-2" 06:34:20 DEBUG "result": [ 06:34:20 DEBUG "data": { 06:34:20 DEBUG "resultType": "vector", 06:34:20 DEBUG }, 06:34:20 DEBUG { 06:34:20 DEBUG "resultType": "vector", 06:34:20 DEBUG "result": [ 06:34:20 DEBUG "value": [ 06:34:20 DEBUG "metric": { 06:34:20 DEBUG "result": [ 06:34:20 DEBUG { 06:34:20 DEBUG 06:34:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:34:20 DEBUG { 06:34:20 DEBUG "metric": { 06:34:20 DEBUG 06:34:20 DEBUG }, 06:34:20 DEBUG "metric": { 06:34:20 DEBUG "instance": "10.142.0.107:9100" 06:34:20 DEBUG 06:34:20 DEBUG "value": [ 06:34:20 DEBUG "instance": "10.142.0.107:9100" 06:34:20 DEBUG }, 06:34:20 DEBUG [...] 06:34:20 DEBUG 06:34:20 DEBUG }, 06:34:20 DEBUG "value": [ 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG "value": [ 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG [...] 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG "pod": "ds-idrepo-0" 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG [...] 06:34:20 DEBUG }, 06:34:20 DEBUG 06:34:20 DEBUG [...] 06:34:20 DEBUG 06:34:20 DEBUG "value": [ 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG 1712813660, 06:34:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG "0" 06:34:20 DEBUG }, 06:34:20 DEBUG 06:34:20 DEBUG "instance": "10.142.0.77:9100" 06:34:20 DEBUG ] 06:34:20 DEBUG "value": [ 06:34:20 DEBUG "instance": "10.142.0.77:9100" 06:34:20 DEBUG }, 06:34:20 DEBUG } 06:34:20 DEBUG 1712813660, 06:34:20 DEBUG }, 06:34:20 DEBUG "value": [ 06:34:20 DEBUG ] 06:34:20 DEBUG "0.021" 06:34:20 DEBUG "value": [ 06:34:20 DEBUG 1712813660, 06:34:20 DEBUG } 06:34:20 DEBUG ] 06:34:20 DEBUG 1712813660, 06:34:20 DEBUG "82621424.69999999" 06:34:20 DEBUG } 06:34:20 DEBUG } 06:34:20 DEBUG "52939023360" 06:34:20 DEBUG ] 06:34:20 DEBUG ] 06:34:20 DEBUG ] 06:34:20 DEBUG } 06:34:20 DEBUG } 06:34:20 DEBUG } 06:34:20 DEBUG ] 06:34:20 DEBUG } 06:34:20 DEBUG ] 06:34:20 DEBUG } 06:34:20 DEBUG } 06:34:20 DEBUG } 06:34:20 DEBUG } 06:34:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813660" 06:34:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712813660" 06:34:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813660" 06:34:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813660" 06:34:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813660" 06:34:20 INFO [http_cmd]: http status code OK 06:34:20 DEBUG --- status code --- 06:34:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:20 DEBUG --- http response --- 06:34:20 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:34:20 ERROR [http_cmd]: ERROR 06:34:20 INFO [http_cmd]: http status code OK 06:34:20 ERROR --- status code --- 06:34:20 DEBUG --- status code --- 06:34:20 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:34:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:20 ERROR --- http response --- 06:34:20 DEBUG --- http response --- 06:34:20 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:34:20 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:34:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:34:20 DEBUG ----- output ----- 06:34:20 DEBUG { 06:34:20 DEBUG "status": "success", 06:34:20 DEBUG "data": { 06:34:20 DEBUG "resultType": "vector", 06:34:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813660" 06:34:20 DEBUG "result": [ 06:34:20 DEBUG { 06:34:20 DEBUG "metric": { 06:34:20 DEBUG "pod": "ds-idrepo-2" 06:34:20 DEBUG }, 06:34:20 DEBUG "value": [ 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG [...] 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG "pod": "ds-cts-2" 06:34:20 DEBUG }, 06:34:20 DEBUG "value": [ 06:34:20 DEBUG 1712813660, 06:34:20 DEBUG "0" 06:34:20 DEBUG ] 06:34:20 DEBUG } 06:34:20 DEBUG ] 06:34:20 DEBUG } 06:34:20 DEBUG } 06:34:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712813660" 06:34:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813660" 06:34:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712813660" 06:34:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813660" 06:34:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813660" 06:34:20 INFO [http_cmd]: http status code OK 06:34:20 INFO [http_cmd]: http status code OK 06:34:20 DEBUG --- status code --- 06:34:20 DEBUG --- status code --- 06:34:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:20 DEBUG --- http response --- 06:34:20 DEBUG --- http response --- 06:34:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:34:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 06:34:20 INFO [http_cmd]: http status code OK 06:34:20 DEBUG ----- output ----- 06:34:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813660" 06:34:20 DEBUG ----- output ----- 06:34:20 DEBUG --- status code --- 06:34:20 DEBUG { 06:34:20 DEBUG { 06:34:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:20 INFO [http_cmd]: http status code OK 06:34:20 DEBUG "status": "success", 06:34:20 INFO [http_cmd]: http status code OK 06:34:20 DEBUG "status": "success", 06:34:20 DEBUG --- http response --- 06:34:20 DEBUG --- status code --- 06:34:20 DEBUG "data": { 06:34:20 DEBUG --- status code --- 06:34:20 DEBUG "data": { 06:34:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:34:20 DEBUG "resultType": "vector", 06:34:20 INFO [http_cmd]: http status code OK 06:34:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:20 INFO [http_cmd]: http status code OK 06:34:20 DEBUG "resultType": "vector", 06:34:20 DEBUG --- http response --- 06:34:20 DEBUG ----- output ----- 06:34:20 DEBUG "result": [ 06:34:20 DEBUG --- status code --- 06:34:20 DEBUG --- http response --- 06:34:20 DEBUG --- status code --- 06:34:20 DEBUG "result": [ 06:34:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813660" 06:34:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:34:20 DEBUG { 06:34:20 DEBUG { 06:34:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813660" 06:34:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:20 INFO [http_cmd]: http status code OK 06:34:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:34:20 DEBUG { 06:34:20 DEBUG ----- output ----- 06:34:20 DEBUG "status": "success", 06:34:20 DEBUG "metric": { 06:34:20 DEBUG --- http response --- 06:34:20 DEBUG --- status code --- 06:34:20 INFO [http_cmd]: http status code OK 06:34:20 DEBUG --- http response --- 06:34:20 DEBUG ----- output ----- 06:34:20 DEBUG "metric": { 06:34:20 DEBUG { 06:34:20 DEBUG "data": { 06:34:20 DEBUG "resultType": "vector", 06:34:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:34:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:20 DEBUG --- status code --- 06:34:20 DEBUG { 06:34:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:34:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:34:20 DEBUG "status": "success", 06:34:20 DEBUG "pod": "ds-idrepo-2" 06:34:20 DEBUG "result": [ 06:34:20 DEBUG ----- output ----- 06:34:20 DEBUG --- http response --- 06:34:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:20 DEBUG "status": "success", 06:34:20 DEBUG ----- output ----- 06:34:20 DEBUG }, 06:34:20 DEBUG "data": { 06:34:20 DEBUG }, 06:34:20 DEBUG { 06:34:20 DEBUG { 06:34:20 DEBUG --- http response --- 06:34:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:34:20 DEBUG "data": { 06:34:20 DEBUG { 06:34:20 DEBUG "value": [ 06:34:20 DEBUG "resultType": "vector", 06:34:20 DEBUG "value": [ 06:34:20 DEBUG "metric": { 06:34:20 DEBUG "status": "success", 06:34:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:34:20 DEBUG ----- output ----- 06:34:20 DEBUG "resultType": "vector", 06:34:20 DEBUG "status": "success", 06:34:20 DEBUG 06:34:20 DEBUG "result": [ 06:34:20 DEBUG 06:34:20 DEBUG "pod": "ds-idrepo-2" 06:34:20 DEBUG "data": { 06:34:20 DEBUG ----- output ----- 06:34:20 DEBUG { 06:34:20 DEBUG "result": [ 06:34:20 DEBUG "data": { 06:34:20 DEBUG 06:34:20 DEBUG { 06:34:20 DEBUG 06:34:20 DEBUG }, 06:34:20 DEBUG "resultType": "vector", 06:34:20 DEBUG { 06:34:20 DEBUG "status": "success", 06:34:20 DEBUG { 06:34:20 DEBUG "resultType": "vector", 06:34:20 DEBUG 06:34:20 DEBUG "metric": { 06:34:20 DEBUG 06:34:20 DEBUG "value": [ 06:34:20 DEBUG "result": [ 06:34:20 DEBUG "status": "success", 06:34:20 DEBUG "data": { 06:34:20 DEBUG "metric": { 06:34:20 DEBUG "result": [ 06:34:20 DEBUG [...] 06:34:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:34:20 DEBUG [...] 06:34:20 DEBUG 06:34:20 DEBUG { 06:34:20 DEBUG "data": { 06:34:20 INFO [http_cmd]: http status code OK 06:34:20 DEBUG "resultType": "vector", 06:34:20 INFO [http_cmd]: http status code OK 06:34:20 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:34:20 DEBUG { 06:34:20 DEBUG 06:34:20 DEBUG }, 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG "metric": { 06:34:20 DEBUG "resultType": "vector", 06:34:20 DEBUG --- status code --- 06:34:20 DEBUG "result": [ 06:34:20 DEBUG --- status code --- 06:34:20 DEBUG }, 06:34:20 DEBUG "metric": { 06:34:20 DEBUG 06:34:20 DEBUG "value": [ 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG "pod": "ds-idrepo-2" 06:34:20 DEBUG "result": [ 06:34:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:20 DEBUG { 06:34:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:20 DEBUG "value": [ 06:34:20 DEBUG "pod": "ds-idrepo-2" 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG [...] 06:34:20 DEBUG }, 06:34:20 DEBUG { 06:34:20 DEBUG --- http response --- 06:34:20 DEBUG "metric": { 06:34:20 DEBUG --- http response --- 06:34:20 DEBUG 06:34:20 DEBUG }, 06:34:20 DEBUG "value": [ 06:34:20 DEBUG 06:34:20 DEBUG "pod": "ds-idrepo-0" 06:34:20 DEBUG 06:34:20 DEBUG "value": [ 06:34:20 DEBUG "metric": { 06:34:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:34:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:34:20 DEBUG 06:34:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:34:20 DEBUG "pod": "ds-cts-1" 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG }, 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG "pod": "ds-idrepo-2" 06:34:20 DEBUG }, 06:34:20 DEBUG ----- output ----- 06:34:20 DEBUG 06:34:20 DEBUG ----- output ----- 06:34:20 DEBUG }, 06:34:20 DEBUG 06:34:20 DEBUG [...] 06:34:20 DEBUG "value": [ 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG }, 06:34:20 DEBUG "value": [ 06:34:20 DEBUG { 06:34:20 DEBUG [...] 06:34:20 DEBUG { 06:34:20 DEBUG "value": [ 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG 1712813660, 06:34:20 DEBUG "pod": "ds-idrepo-0" 06:34:20 DEBUG 06:34:20 DEBUG "value": [ 06:34:20 DEBUG 06:34:20 DEBUG "status": "success", 06:34:20 DEBUG 06:34:20 DEBUG "status": "success", 06:34:20 DEBUG 1712813660, 06:34:20 DEBUG [...] 06:34:20 DEBUG 06:34:20 DEBUG "0" 06:34:20 DEBUG }, 06:34:20 DEBUG [...] 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG "data": { 06:34:20 DEBUG 06:34:20 DEBUG "data": { 06:34:20 DEBUG "32795666.60108859" 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG ] 06:34:20 DEBUG "value": [ 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG "resultType": "vector", 06:34:20 DEBUG 06:34:20 DEBUG "resultType": "vector", 06:34:20 DEBUG ] 06:34:20 DEBUG 06:34:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:34:20 DEBUG } 06:34:20 DEBUG 1712813660, 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG [...] 06:34:20 DEBUG "result": [ 06:34:20 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 06:34:20 DEBUG "result": [ 06:34:20 DEBUG } 06:34:20 DEBUG 06:34:20 DEBUG }, 06:34:20 DEBUG ] 06:34:20 DEBUG "0" 06:34:20 DEBUG 06:34:20 DEBUG [...] 06:34:20 DEBUG 06:34:20 DEBUG { 06:34:20 DEBUG }, 06:34:20 DEBUG { 06:34:20 DEBUG ] 06:34:20 DEBUG "pod": "ds-cts-2" 06:34:20 DEBUG "value": [ 06:34:20 DEBUG } 06:34:20 DEBUG ] 06:34:20 DEBUG "pod": "ds-cts-2" 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG "metric": { 06:34:20 DEBUG "value": [ 06:34:20 DEBUG "metric": { 06:34:20 DEBUG } 06:34:20 DEBUG }, 06:34:20 DEBUG 1712813660, 06:34:20 DEBUG } 06:34:20 DEBUG } 06:34:20 DEBUG }, 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:34:20 DEBUG 1712813660, 06:34:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:34:20 DEBUG } 06:34:20 DEBUG "value": [ 06:34:20 DEBUG "0" 06:34:20 DEBUG ] 06:34:20 DEBUG "value": [ 06:34:20 DEBUG 06:34:20 DEBUG "pod": "ds-cts-1" 06:34:20 DEBUG }, 06:34:20 DEBUG "14054195200" 06:34:20 DEBUG }, 06:34:20 DEBUG 1712813660, 06:34:20 DEBUG ] 06:34:20 DEBUG } 06:34:20 DEBUG 1712813660, 06:34:20 DEBUG "pod": "ds-cts-2" 06:34:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813660" 06:34:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813660" 06:34:20 DEBUG }, 06:34:20 DEBUG "value": [ 06:34:20 DEBUG ] 06:34:20 DEBUG "value": [ 06:34:20 DEBUG "0" 06:34:20 DEBUG } 06:34:20 DEBUG } 06:34:20 DEBUG "29.666666666666668" 06:34:20 DEBUG }, 06:34:20 DEBUG "value": [ 06:34:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813660" 06:34:20 DEBUG 06:34:20 DEBUG } 06:34:20 DEBUG 06:34:20 DEBUG ] 06:34:20 DEBUG } 06:34:20 DEBUG ] 06:34:20 DEBUG } 06:34:20 DEBUG 1712813660, 06:34:20 DEBUG 06:34:20 DEBUG ] 06:34:20 DEBUG 06:34:20 DEBUG ] 06:34:20 DEBUG ] 06:34:20 DEBUG "value": [ 06:34:20 DEBUG ] 06:34:20 DEBUG "431731534.9203226" 06:34:20 DEBUG 06:34:20 DEBUG } 06:34:20 DEBUG 06:34:20 DEBUG } 06:34:20 DEBUG } 06:34:20 DEBUG 1712813660, 06:34:20 DEBUG } 06:34:20 DEBUG ] 06:34:20 DEBUG [...] 06:34:20 DEBUG } 06:34:20 DEBUG [...] 06:34:20 DEBUG } 06:34:20 DEBUG } 06:34:20 DEBUG "0" 06:34:20 DEBUG } 06:34:20 DEBUG } 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG ] 06:34:20 DEBUG ] 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG } 06:34:20 DEBUG } 06:34:20 INFO [http_cmd]: http status code OK 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG ] 06:34:20 DEBUG } 06:34:20 DEBUG --- status code --- 06:34:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:34:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:34:20 DEBUG } 06:34:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:20 DEBUG }, 06:34:20 DEBUG }, 06:34:20 INFO [http_cmd]: http status code OK 06:34:20 DEBUG } 06:34:20 DEBUG --- http response --- 06:34:20 DEBUG "value": [ 06:34:20 DEBUG "value": [ 06:34:20 DEBUG --- status code --- 06:34:20 DEBUG 1712813660, 06:34:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:34:20 DEBUG 1712813660, 06:34:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:20 DEBUG "1363.8666666666666" 06:34:20 DEBUG ----- output ----- 06:34:20 DEBUG "1348.5333333333333" 06:34:20 DEBUG --- http response --- 06:34:20 DEBUG ] 06:34:20 DEBUG { 06:34:20 DEBUG ] 06:34:20 DEBUG } 06:34:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:34:20 DEBUG "status": "success", 06:34:20 DEBUG } 06:34:20 DEBUG ] 06:34:20 DEBUG ----- output ----- 06:34:20 INFO [http_cmd]: http status code OK 06:34:20 DEBUG "data": { 06:34:20 DEBUG ] 06:34:20 DEBUG } 06:34:20 DEBUG { 06:34:20 DEBUG --- status code --- 06:34:20 DEBUG "resultType": "vector", 06:34:20 DEBUG } 06:34:20 DEBUG } 06:34:20 DEBUG "status": "success", 06:34:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:20 DEBUG "result": [ 06:34:20 DEBUG } 06:34:20 DEBUG "data": { 06:34:20 DEBUG --- http response --- 06:34:20 DEBUG { 06:34:20 DEBUG "resultType": "vector", 06:34:20 DEBUG "metric": { 06:34:20 DEBUG "result": [ 06:34:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:34:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:34:20 DEBUG { 06:34:20 DEBUG ----- output ----- 06:34:20 DEBUG }, 06:34:20 DEBUG "metric": { 06:34:20 DEBUG { 06:34:20 DEBUG "value": [ 06:34:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:34:20 DEBUG "status": "success", 06:34:20 DEBUG 06:34:20 DEBUG }, 06:34:20 DEBUG "data": { 06:34:20 DEBUG 06:34:20 DEBUG "value": [ 06:34:20 DEBUG "resultType": "vector", 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG "result": [ 06:34:20 DEBUG [...] 06:34:20 DEBUG 06:34:20 DEBUG { 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG "metric": { 06:34:20 DEBUG 06:34:20 DEBUG [...] 06:34:20 DEBUG "instance": "10.142.0.107:9100" 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG }, 06:34:20 DEBUG "pod": "ds-cts-1" 06:34:20 DEBUG 06:34:20 DEBUG "value": [ 06:34:20 DEBUG }, 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG "value": [ 06:34:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:34:20 DEBUG 06:34:20 DEBUG 1712813660, 06:34:20 DEBUG }, 06:34:20 DEBUG 06:34:20 DEBUG "8.726823908190772" 06:34:20 DEBUG "value": [ 06:34:20 DEBUG [...] 06:34:20 DEBUG ] 06:34:20 DEBUG 1712813660, 06:34:20 DEBUG 06:34:20 DEBUG } 06:34:20 DEBUG "244894" 06:34:20 DEBUG 06:34:20 DEBUG ] 06:34:20 DEBUG ] 06:34:20 DEBUG 06:34:20 DEBUG } 06:34:20 DEBUG } 06:34:20 DEBUG "instance": "10.142.0.77:9100" 06:34:20 DEBUG } 06:34:20 DEBUG ] 06:34:20 DEBUG }, 06:34:20 DEBUG } 06:34:20 DEBUG "value": [ 06:34:20 DEBUG } 06:34:20 DEBUG 1712813660, 06:34:20 DEBUG "0.0006666666666666524" 06:34:20 DEBUG ] 06:34:20 DEBUG } 06:34:20 DEBUG ] 06:34:20 DEBUG } 06:34:20 DEBUG } 06:34:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813660" 06:34:20 INFO [http_cmd]: http status code OK 06:34:20 DEBUG --- status code --- 06:34:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:20 DEBUG --- http response --- 06:34:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:34:20 DEBUG ----- output ----- 06:34:20 DEBUG { 06:34:20 DEBUG "status": "success", 06:34:20 DEBUG "data": { 06:34:20 DEBUG "resultType": "vector", 06:34:20 DEBUG "result": [ 06:34:20 DEBUG { 06:34:20 DEBUG "metric": { 06:34:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:34:20 DEBUG }, 06:34:20 DEBUG "value": [ 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG [...] 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:34:20 DEBUG }, 06:34:20 DEBUG "value": [ 06:34:20 DEBUG 1712813660, 06:34:20 DEBUG "1353.6666666666667" 06:34:20 DEBUG ] 06:34:20 DEBUG } 06:34:20 DEBUG ] 06:34:20 DEBUG } 06:34:20 DEBUG } 06:34:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712813660" 06:34:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813660" 06:34:20 INFO [http_cmd]: http status code OK 06:34:20 DEBUG --- status code --- 06:34:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:20 DEBUG --- http response --- 06:34:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:34:20 DEBUG ----- output ----- 06:34:20 DEBUG { 06:34:20 DEBUG "status": "success", 06:34:20 DEBUG "data": { 06:34:20 DEBUG "resultType": "vector", 06:34:20 DEBUG "result": [ 06:34:20 DEBUG { 06:34:20 DEBUG "metric": { 06:34:20 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:34:20 DEBUG }, 06:34:20 DEBUG "value": [ 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG [...] 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:34:20 DEBUG }, 06:34:20 DEBUG "value": [ 06:34:20 DEBUG 1712813660, 06:34:20 DEBUG "4.214943857045643" 06:34:20 DEBUG ] 06:34:20 DEBUG } 06:34:20 DEBUG ] 06:34:20 DEBUG } 06:34:20 DEBUG } 06:34:20 INFO [http_cmd]: http status code OK 06:34:20 DEBUG --- status code --- 06:34:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:20 DEBUG --- http response --- 06:34:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:34:20 DEBUG ----- output ----- 06:34:20 DEBUG { 06:34:20 DEBUG "status": "success", 06:34:20 DEBUG "data": { 06:34:20 DEBUG "resultType": "vector", 06:34:20 DEBUG "result": [ 06:34:20 DEBUG { 06:34:20 DEBUG "metric": { 06:34:20 DEBUG "instance": "10.142.0.107:9100" 06:34:20 DEBUG }, 06:34:20 DEBUG "value": [ 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG [...] 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG "instance": "10.142.0.77:9100" 06:34:20 DEBUG }, 06:34:20 DEBUG "value": [ 06:34:20 DEBUG 1712813660, 06:34:20 DEBUG "0.002133333333333572" 06:34:20 DEBUG ] 06:34:20 DEBUG } 06:34:20 DEBUG ] 06:34:20 DEBUG } 06:34:20 DEBUG } 06:34:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813660" 06:34:20 INFO [http_cmd]: http status code OK 06:34:20 DEBUG --- status code --- 06:34:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:34:20 DEBUG --- http response --- 06:34:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:34:20 DEBUG ----- output ----- 06:34:20 DEBUG { 06:34:20 DEBUG "status": "success", 06:34:20 DEBUG "data": { 06:34:20 DEBUG "resultType": "vector", 06:34:20 DEBUG "result": [ 06:34:20 DEBUG { 06:34:20 DEBUG "metric": { 06:34:20 DEBUG "instance": "10.142.0.107:9100" 06:34:20 DEBUG }, 06:34:20 DEBUG "value": [ 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG [...] 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG 06:34:20 DEBUG "instance": "10.142.0.77:9100" 06:34:20 DEBUG }, 06:34:20 DEBUG "value": [ 06:34:20 DEBUG 1712813660, 06:34:20 DEBUG "82618472.5" 06:34:20 DEBUG ] 06:34:20 DEBUG } 06:34:20 DEBUG ] 06:34:20 DEBUG } 06:34:20 DEBUG } 06:34:31 INFO 06:34:31 INFO [loop_until]: kubectl --namespace=xlou top pods 06:34:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:34:31 INFO [loop_until]: OK (rc = 0) 06:34:31 DEBUG --- stdout --- 06:34:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 8980m 8500Mi am-5b5bd8567b-8h2w4 8812m 8562Mi am-5b5bd8567b-qlkw6 7297m 8456Mi ds-cts-0 8217m 13387Mi ds-cts-1 8575m 12717Mi ds-cts-2 8647m 13391Mi ds-idrepo-0 11434m 5498Mi ds-idrepo-1 358m 5479Mi ds-idrepo-2 348m 5783Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 4m 1011Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 2m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4119m 1845Mi 06:34:31 DEBUG --- stderr --- 06:34:31 DEBUG 06:34:34 INFO 06:34:34 INFO [loop_until]: kubectl --namespace=xlou top node 06:34:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:34:34 INFO [loop_until]: OK (rc = 0) 06:34:34 DEBUG --- stdout --- 06:34:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 2726Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 64m 0% 2777Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7875m 26% 10123Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10711m 35% 2984Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9678m 32% 10078Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 146m 0% 3858Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9625m 32% 10187Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 410m 2% 6460Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11583m 72% 6190Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8832m 55% 13959Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 8406m 52% 13991Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 8851m 55% 13320Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 400m 2% 6155Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4249m 26% 5618Mi 9% 06:34:34 DEBUG --- stderr --- 06:34:34 DEBUG 06:35:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813720" 06:35:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813720" 06:35:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813720" 06:35:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712813720" 06:35:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813720" 06:35:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813720" 06:35:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712813720" 06:35:20 INFO [http_cmd]: http status code OK 06:35:20 INFO [http_cmd]: http status code OK 06:35:20 INFO [http_cmd]: http status code OK 06:35:20 DEBUG --- status code --- 06:35:20 DEBUG --- status code --- 06:35:20 DEBUG --- status code --- 06:35:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:20 DEBUG --- http response --- 06:35:20 DEBUG --- http response --- 06:35:20 DEBUG --- http response --- 06:35:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:35:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:35:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813720" 06:35:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:35:20 DEBUG ----- output ----- 06:35:20 DEBUG ----- output ----- 06:35:20 DEBUG ----- output ----- 06:35:20 DEBUG { 06:35:20 DEBUG { 06:35:20 DEBUG { 06:35:20 DEBUG "status": "success", 06:35:20 DEBUG "status": "success", 06:35:20 DEBUG "status": "success", 06:35:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813720" 06:35:20 DEBUG "data": { 06:35:20 DEBUG "data": { 06:35:20 DEBUG "data": { 06:35:20 DEBUG "resultType": "vector", 06:35:20 DEBUG "resultType": "vector", 06:35:20 DEBUG "resultType": "vector", 06:35:20 DEBUG "result": [ 06:35:20 DEBUG "result": [ 06:35:20 DEBUG "result": [ 06:35:20 DEBUG { 06:35:20 DEBUG { 06:35:20 DEBUG { 06:35:20 DEBUG "metric": { 06:35:20 DEBUG "metric": { 06:35:20 DEBUG "metric": { 06:35:20 DEBUG "pod": "ds-idrepo-2" 06:35:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:35:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:35:20 ERROR [http_cmd]: ERROR 06:35:20 DEBUG }, 06:35:20 DEBUG }, 06:35:20 INFO [http_cmd]: http status code OK 06:35:20 INFO [http_cmd]: http status code OK 06:35:20 DEBUG }, 06:35:20 ERROR --- status code --- 06:35:20 DEBUG "value": [ 06:35:20 DEBUG "value": [ 06:35:20 DEBUG --- status code --- 06:35:20 DEBUG --- status code --- 06:35:20 DEBUG "value": [ 06:35:20 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:20 DEBUG 06:35:20 ERROR --- http response --- 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG --- http response --- 06:35:20 DEBUG --- http response --- 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:35:20 DEBUG 06:35:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:35:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:35:20 DEBUG 06:35:20 DEBUG [...] 06:35:20 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:35:20 DEBUG [...] 06:35:20 DEBUG ----- output ----- 06:35:20 DEBUG ----- output ----- 06:35:20 DEBUG [...] 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG { 06:35:20 DEBUG { 06:35:20 INFO [http_cmd]: http status code OK 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG "status": "success", 06:35:20 DEBUG "status": "success", 06:35:20 DEBUG --- status code --- 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG "data": { 06:35:20 DEBUG "data": { 06:35:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:20 DEBUG 06:35:20 DEBUG "pod": "ds-idrepo-0" 06:35:20 DEBUG "pod": "am-5b5bd8567b-qlkw6" 06:35:20 DEBUG "resultType": "vector", 06:35:20 DEBUG "resultType": "vector", 06:35:20 DEBUG --- http response --- 06:35:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:35:20 DEBUG }, 06:35:20 DEBUG }, 06:35:20 DEBUG "result": [ 06:35:20 DEBUG "result": [ 06:35:20 DEBUG }, 06:35:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:35:20 DEBUG "value": [ 06:35:20 DEBUG "value": [ 06:35:20 DEBUG { 06:35:20 DEBUG { 06:35:20 DEBUG "value": [ 06:35:20 DEBUG ----- output ----- 06:35:20 DEBUG 1712813720, 06:35:20 DEBUG 1712813720, 06:35:20 DEBUG "metric": { 06:35:20 DEBUG "metric": { 06:35:20 DEBUG 1712813720, 06:35:20 DEBUG { 06:35:20 DEBUG "0" 06:35:20 DEBUG "4501380.219780219" 06:35:20 DEBUG "instance": "10.142.0.107:9100" 06:35:20 DEBUG "instance": "10.142.0.107:9100" 06:35:20 DEBUG "1351.5333333333333" 06:35:20 DEBUG "status": "success", 06:35:20 DEBUG ] 06:35:20 DEBUG ] 06:35:20 DEBUG }, 06:35:20 DEBUG }, 06:35:20 DEBUG ] 06:35:20 DEBUG "data": { 06:35:20 DEBUG } 06:35:20 DEBUG } 06:35:20 DEBUG "value": [ 06:35:20 DEBUG "value": [ 06:35:20 DEBUG } 06:35:20 DEBUG "resultType": "vector", 06:35:20 DEBUG ] 06:35:20 DEBUG ] 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG ] 06:35:20 DEBUG "result": [ 06:35:20 DEBUG } 06:35:20 DEBUG } 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG } 06:35:20 DEBUG { 06:35:20 DEBUG } 06:35:20 DEBUG } 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG } 06:35:20 INFO [http_cmd]: http status code OK 06:35:20 INFO [http_cmd]: http status code OK 06:35:20 DEBUG "metric": { 06:35:20 DEBUG [...] 06:35:20 DEBUG [...] 06:35:20 DEBUG --- status code --- 06:35:20 DEBUG --- status code --- 06:35:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:20 DEBUG }, 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG --- http response --- 06:35:20 DEBUG --- http response --- 06:35:20 DEBUG "value": [ 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:35:20 DEBUG 06:35:20 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:35:20 DEBUG "instance": "10.142.0.77:9100" 06:35:20 DEBUG "instance": "10.142.0.77:9100" 06:35:20 DEBUG ----- output ----- 06:35:20 DEBUG 06:35:20 DEBUG }, 06:35:20 DEBUG }, 06:35:20 DEBUG { 06:35:20 DEBUG 06:35:20 DEBUG "value": [ 06:35:20 DEBUG "value": [ 06:35:20 DEBUG "status": "success", 06:35:20 DEBUG [...] 06:35:20 DEBUG 1712813720, 06:35:20 DEBUG 1712813720, 06:35:20 DEBUG "data": { 06:35:20 DEBUG 06:35:20 DEBUG "52940252160" 06:35:20 DEBUG "82923012.83333333" 06:35:20 DEBUG "resultType": "vector", 06:35:20 DEBUG 06:35:20 DEBUG ] 06:35:20 DEBUG ] 06:35:20 DEBUG "result": [ 06:35:20 DEBUG 06:35:20 DEBUG } 06:35:20 DEBUG } 06:35:20 DEBUG { 06:35:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:35:20 DEBUG ] 06:35:20 DEBUG ] 06:35:20 DEBUG "metric": { 06:35:20 DEBUG }, 06:35:20 DEBUG } 06:35:20 DEBUG } 06:35:20 DEBUG "pod": "ds-idrepo-2" 06:35:20 DEBUG "value": [ 06:35:20 DEBUG } 06:35:20 DEBUG } 06:35:20 DEBUG }, 06:35:20 DEBUG 1712813720, 06:35:20 DEBUG "value": [ 06:35:20 DEBUG "0.021" 06:35:20 DEBUG 06:35:20 DEBUG ] 06:35:20 DEBUG 06:35:20 DEBUG } 06:35:20 DEBUG 06:35:20 DEBUG ] 06:35:20 DEBUG [...] 06:35:20 DEBUG } 06:35:20 DEBUG 06:35:20 DEBUG } 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG "pod": "ds-cts-2" 06:35:20 DEBUG }, 06:35:20 DEBUG "value": [ 06:35:20 DEBUG 1712813720, 06:35:20 DEBUG "0" 06:35:20 DEBUG ] 06:35:20 DEBUG } 06:35:20 DEBUG ] 06:35:20 DEBUG } 06:35:20 DEBUG } 06:35:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813720" 06:35:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813720" 06:35:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813720" 06:35:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813720" 06:35:20 INFO [http_cmd]: http status code OK 06:35:20 DEBUG --- status code --- 06:35:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:20 DEBUG --- http response --- 06:35:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:35:20 DEBUG ----- output ----- 06:35:20 DEBUG { 06:35:20 DEBUG "status": "success", 06:35:20 DEBUG "data": { 06:35:20 DEBUG "resultType": "vector", 06:35:20 DEBUG "result": [ 06:35:20 DEBUG { 06:35:20 DEBUG "metric": { 06:35:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:35:20 DEBUG }, 06:35:20 DEBUG "value": [ 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG [...] 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:35:20 DEBUG }, 06:35:20 DEBUG "value": [ 06:35:20 DEBUG 1712813720, 06:35:20 DEBUG "244485" 06:35:20 DEBUG ] 06:35:20 DEBUG } 06:35:20 DEBUG ] 06:35:20 DEBUG } 06:35:20 DEBUG } 06:35:20 INFO [http_cmd]: http status code OK 06:35:20 DEBUG --- status code --- 06:35:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:20 DEBUG --- http response --- 06:35:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:35:20 DEBUG ----- output ----- 06:35:20 DEBUG { 06:35:20 DEBUG "status": "success", 06:35:20 DEBUG "data": { 06:35:20 DEBUG "resultType": "vector", 06:35:20 DEBUG "result": [ 06:35:20 DEBUG { 06:35:20 DEBUG "metric": { 06:35:20 DEBUG "pod": "ds-idrepo-2" 06:35:20 DEBUG }, 06:35:20 DEBUG "value": [ 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG [...] 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG "pod": "ds-cts-2" 06:35:20 DEBUG }, 06:35:20 DEBUG "value": [ 06:35:20 DEBUG 1712813720, 06:35:20 DEBUG "28.9" 06:35:20 DEBUG ] 06:35:20 DEBUG } 06:35:20 DEBUG ] 06:35:20 DEBUG } 06:35:20 DEBUG } 06:35:20 INFO [http_cmd]: http status code OK 06:35:20 DEBUG --- status code --- 06:35:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:20 DEBUG --- http response --- 06:35:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:35:20 DEBUG ----- output ----- 06:35:20 DEBUG { 06:35:20 DEBUG "status": "success", 06:35:20 DEBUG "data": { 06:35:20 DEBUG "resultType": "vector", 06:35:20 DEBUG "result": [ 06:35:20 DEBUG { 06:35:20 DEBUG "metric": { 06:35:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:35:20 DEBUG }, 06:35:20 DEBUG "value": [ 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG [...] 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:35:20 DEBUG }, 06:35:20 DEBUG "value": [ 06:35:20 DEBUG 1712813720, 06:35:20 DEBUG "1357.3333333333333" 06:35:20 DEBUG ] 06:35:20 DEBUG } 06:35:20 DEBUG ] 06:35:20 DEBUG } 06:35:20 DEBUG } 06:35:20 INFO [http_cmd]: http status code OK 06:35:20 DEBUG --- status code --- 06:35:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:20 DEBUG --- http response --- 06:35:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:35:20 DEBUG ----- output ----- 06:35:20 DEBUG { 06:35:20 DEBUG "status": "success", 06:35:20 DEBUG "data": { 06:35:20 DEBUG "resultType": "vector", 06:35:20 DEBUG "result": [ 06:35:20 DEBUG { 06:35:20 DEBUG "metric": { 06:35:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:35:20 DEBUG }, 06:35:20 DEBUG "value": [ 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG [...] 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:35:20 DEBUG }, 06:35:20 DEBUG "value": [ 06:35:20 DEBUG 1712813720, 06:35:20 DEBUG "0" 06:35:20 DEBUG ] 06:35:20 DEBUG } 06:35:20 DEBUG ] 06:35:20 DEBUG } 06:35:20 DEBUG } 06:35:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813720" 06:35:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813720" 06:35:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813720" 06:35:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712813720" 06:35:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813720" 06:35:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712813720" 06:35:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813720" 06:35:20 INFO [http_cmd]: http status code OK 06:35:20 DEBUG --- status code --- 06:35:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:20 DEBUG --- http response --- 06:35:20 INFO [http_cmd]: http status code OK 06:35:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:35:20 DEBUG --- status code --- 06:35:20 DEBUG ----- output ----- 06:35:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:20 DEBUG { 06:35:20 DEBUG --- http response --- 06:35:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813720" 06:35:20 DEBUG "status": "success", 06:35:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:35:20 DEBUG "data": { 06:35:20 INFO [http_cmd]: http status code OK 06:35:20 DEBUG ----- output ----- 06:35:20 INFO [http_cmd]: http status code OK 06:35:20 DEBUG "resultType": "vector", 06:35:20 DEBUG --- status code --- 06:35:20 DEBUG { 06:35:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813720" 06:35:20 DEBUG --- status code --- 06:35:20 DEBUG "result": [ 06:35:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813720" 06:35:20 DEBUG "status": "success", 06:35:20 INFO [http_cmd]: http status code OK 06:35:20 INFO [http_cmd]: http status code OK 06:35:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:20 DEBUG { 06:35:20 DEBUG --- http response --- 06:35:20 DEBUG "data": { 06:35:20 INFO [http_cmd]: http status code OK 06:35:20 DEBUG --- status code --- 06:35:20 DEBUG --- status code --- 06:35:20 DEBUG --- http response --- 06:35:20 DEBUG "metric": { 06:35:20 DEBUG "resultType": "vector", 06:35:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:35:20 DEBUG --- status code --- 06:35:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:20 INFO [http_cmd]: http status code OK 06:35:20 DEBUG "pod": "ds-idrepo-2" 06:35:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:35:20 DEBUG "result": [ 06:35:20 DEBUG ----- output ----- 06:35:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:20 DEBUG --- http response --- 06:35:20 DEBUG --- http response --- 06:35:20 DEBUG --- status code --- 06:35:20 DEBUG }, 06:35:20 DEBUG ----- output ----- 06:35:20 DEBUG { 06:35:20 DEBUG { 06:35:20 DEBUG "status": "success", 06:35:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:35:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:35:20 DEBUG "value": [ 06:35:20 DEBUG { 06:35:20 DEBUG "metric": { 06:35:20 DEBUG --- http response --- 06:35:20 DEBUG "data": { 06:35:20 DEBUG ----- output ----- 06:35:20 DEBUG --- http response --- 06:35:20 DEBUG ----- output ----- 06:35:20 DEBUG 06:35:20 DEBUG "status": "success", 06:35:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:35:20 DEBUG "resultType": "vector", 06:35:20 DEBUG "result": [ 06:35:20 DEBUG { 06:35:20 DEBUG { 06:35:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:35:20 DEBUG 06:35:20 DEBUG "data": { 06:35:20 DEBUG }, 06:35:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:35:20 DEBUG { 06:35:20 DEBUG "status": "success", 06:35:20 DEBUG "status": "success", 06:35:20 DEBUG ----- output ----- 06:35:20 DEBUG 06:35:20 DEBUG "resultType": "vector", 06:35:20 DEBUG "value": [ 06:35:20 DEBUG ----- output ----- 06:35:20 DEBUG "metric": { 06:35:20 DEBUG "data": { 06:35:20 DEBUG "data": { 06:35:20 DEBUG { 06:35:20 DEBUG [...] 06:35:20 DEBUG "result": [ 06:35:20 DEBUG 06:35:20 DEBUG { 06:35:20 DEBUG "pod": "ds-idrepo-2" 06:35:20 DEBUG "resultType": "vector", 06:35:20 DEBUG "resultType": "vector", 06:35:20 DEBUG "status": "success", 06:35:20 DEBUG 06:35:20 DEBUG { 06:35:20 DEBUG 06:35:20 DEBUG "status": "success", 06:35:20 DEBUG }, 06:35:20 DEBUG "result": [ 06:35:20 DEBUG "result": [ 06:35:20 DEBUG "data": { 06:35:20 DEBUG 06:35:20 DEBUG "metric": { 06:35:20 DEBUG 06:35:20 DEBUG "data": { 06:35:20 DEBUG "value": [ 06:35:20 DEBUG { 06:35:20 DEBUG { 06:35:20 DEBUG "resultType": "vector", 06:35:20 DEBUG 06:35:20 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:35:20 DEBUG [...] 06:35:20 INFO [http_cmd]: http status code OK 06:35:20 DEBUG "resultType": "vector", 06:35:20 DEBUG 06:35:20 DEBUG "metric": { 06:35:20 DEBUG "metric": { 06:35:20 DEBUG "result": [ 06:35:20 DEBUG "pod": "ds-idrepo-0" 06:35:20 DEBUG }, 06:35:20 DEBUG 06:35:20 DEBUG --- status code --- 06:35:20 DEBUG "result": [ 06:35:20 DEBUG 06:35:20 DEBUG "pod": "ds-idrepo-2" 06:35:20 DEBUG "pod": "ds-idrepo-2" 06:35:20 DEBUG { 06:35:20 DEBUG }, 06:35:20 DEBUG "value": [ 06:35:20 DEBUG 06:35:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:20 DEBUG { 06:35:20 DEBUG 06:35:20 DEBUG }, 06:35:20 DEBUG }, 06:35:20 DEBUG "metric": { 06:35:20 DEBUG "value": [ 06:35:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813720" 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG --- http response --- 06:35:20 DEBUG "metric": { 06:35:20 DEBUG [...] 06:35:20 DEBUG "value": [ 06:35:20 DEBUG "value": [ 06:35:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:35:20 DEBUG 1712813720, 06:35:20 DEBUG 06:35:20 DEBUG "pod": "am-5b5bd8567b-qlkw6" 06:35:20 INFO [http_cmd]: http status code OK 06:35:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:35:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG }, 06:35:20 DEBUG "0" 06:35:20 DEBUG 06:35:20 DEBUG }, 06:35:20 DEBUG --- status code --- 06:35:20 DEBUG }, 06:35:20 DEBUG ----- output ----- 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG "value": [ 06:35:20 DEBUG ] 06:35:20 DEBUG [...] 06:35:20 DEBUG "value": [ 06:35:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:20 DEBUG "value": [ 06:35:20 DEBUG { 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG } 06:35:20 DEBUG 06:35:20 DEBUG 1712813720, 06:35:20 DEBUG --- http response --- 06:35:20 DEBUG 06:35:20 DEBUG "status": "success", 06:35:20 DEBUG "pod": "ds-idrepo-0" 06:35:20 DEBUG [...] 06:35:20 DEBUG [...] 06:35:20 DEBUG 06:35:20 DEBUG ] 06:35:20 DEBUG 06:35:20 DEBUG "0" 06:35:20 DEBUG 06:35:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:35:20 DEBUG "data": { 06:35:20 DEBUG }, 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG } 06:35:20 DEBUG 06:35:20 DEBUG ] 06:35:20 DEBUG 06:35:20 DEBUG ----- output ----- 06:35:20 DEBUG "resultType": "vector", 06:35:20 DEBUG "value": [ 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG [...] 06:35:20 DEBUG } 06:35:20 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 06:35:20 DEBUG } 06:35:20 DEBUG [...] 06:35:20 DEBUG { 06:35:20 DEBUG "result": [ 06:35:20 DEBUG 1712813720, 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG }, 06:35:20 DEBUG ] 06:35:20 DEBUG 06:35:20 DEBUG "status": "success", 06:35:20 DEBUG { 06:35:20 DEBUG "0" 06:35:20 DEBUG "pod": "ds-cts-2" 06:35:20 DEBUG "pod": "ds-cts-2" 06:35:20 DEBUG 06:35:20 DEBUG "value": [ 06:35:20 DEBUG } 06:35:20 DEBUG 06:35:20 DEBUG "data": { 06:35:20 DEBUG "metric": { 06:35:20 DEBUG ] 06:35:20 DEBUG }, 06:35:20 DEBUG }, 06:35:20 DEBUG 06:35:20 DEBUG 1712813720, 06:35:20 DEBUG } 06:35:20 DEBUG 06:35:20 DEBUG "resultType": "vector", 06:35:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:35:20 DEBUG } 06:35:20 DEBUG "value": [ 06:35:20 DEBUG "value": [ 06:35:20 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:35:20 DEBUG "14128316416" 06:35:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:35:20 DEBUG "result": [ 06:35:20 DEBUG }, 06:35:20 DEBUG ] 06:35:20 DEBUG 1712813720, 06:35:20 DEBUG 1712813720, 06:35:20 DEBUG }, 06:35:20 DEBUG ] 06:35:20 DEBUG }, 06:35:20 DEBUG { 06:35:20 DEBUG "value": [ 06:35:20 DEBUG } 06:35:20 DEBUG "0" 06:35:20 DEBUG "0" 06:35:20 DEBUG "value": [ 06:35:20 DEBUG } 06:35:20 DEBUG "value": [ 06:35:20 INFO [http_cmd]: http status code OK 06:35:20 DEBUG "metric": { 06:35:20 DEBUG 06:35:20 DEBUG } 06:35:20 DEBUG ] 06:35:20 DEBUG ] 06:35:20 DEBUG 1712813720, 06:35:20 DEBUG ] 06:35:20 DEBUG 1712813720, 06:35:20 DEBUG --- status code --- 06:35:20 DEBUG "instance": "10.142.0.107:9100" 06:35:20 DEBUG 06:35:20 DEBUG } 06:35:20 DEBUG } 06:35:20 DEBUG "0" 06:35:20 DEBUG } 06:35:20 DEBUG "1351.7666666666667" 06:35:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:20 DEBUG }, 06:35:20 DEBUG 06:35:20 DEBUG ] 06:35:20 DEBUG ] 06:35:20 DEBUG ] 06:35:20 DEBUG } 06:35:20 DEBUG ] 06:35:20 DEBUG --- http response --- 06:35:20 DEBUG "value": [ 06:35:20 DEBUG [...] 06:35:20 DEBUG } 06:35:20 DEBUG } 06:35:20 DEBUG } 06:35:20 DEBUG } 06:35:20 DEBUG 06:35:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:35:20 DEBUG 06:35:20 DEBUG } 06:35:20 DEBUG } 06:35:20 DEBUG ] 06:35:20 DEBUG ] 06:35:20 DEBUG 06:35:20 DEBUG ----- output ----- 06:35:20 DEBUG 06:35:20 DEBUG } 06:35:20 DEBUG } 06:35:20 DEBUG 06:35:20 DEBUG { 06:35:20 DEBUG 06:35:20 DEBUG } 06:35:20 DEBUG } 06:35:20 DEBUG [...] 06:35:20 DEBUG "status": "success", 06:35:20 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:35:20 DEBUG 06:35:20 DEBUG "data": { 06:35:20 DEBUG }, 06:35:20 DEBUG 06:35:20 DEBUG "resultType": "vector", 06:35:20 DEBUG "value": [ 06:35:20 DEBUG 06:35:20 DEBUG "result": [ 06:35:20 DEBUG 1712813720, 06:35:20 DEBUG "instance": "10.142.0.77:9100" 06:35:20 DEBUG { 06:35:20 DEBUG "0" 06:35:20 DEBUG }, 06:35:20 DEBUG "metric": { 06:35:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813720" 06:35:20 DEBUG ] 06:35:20 DEBUG "value": [ 06:35:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:35:20 DEBUG } 06:35:20 DEBUG 1712813720, 06:35:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712813720" 06:35:20 DEBUG }, 06:35:20 DEBUG ] 06:35:20 DEBUG "0.00033333333333350387" 06:35:20 DEBUG "value": [ 06:35:20 DEBUG } 06:35:20 DEBUG ] 06:35:20 DEBUG 06:35:20 DEBUG } 06:35:20 DEBUG } 06:35:20 DEBUG 06:35:20 DEBUG ] 06:35:20 DEBUG 06:35:20 DEBUG } 06:35:20 DEBUG [...] 06:35:20 DEBUG } 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:35:20 DEBUG }, 06:35:20 DEBUG "value": [ 06:35:20 DEBUG 1712813720, 06:35:20 DEBUG "1353.8333333333333" 06:35:20 DEBUG ] 06:35:20 DEBUG } 06:35:20 DEBUG ] 06:35:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813720" 06:35:20 DEBUG } 06:35:20 DEBUG } 06:35:20 INFO [http_cmd]: http status code OK 06:35:20 DEBUG --- status code --- 06:35:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:20 DEBUG --- http response --- 06:35:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:35:20 DEBUG ----- output ----- 06:35:20 DEBUG { 06:35:20 DEBUG "status": "success", 06:35:20 DEBUG "data": { 06:35:20 DEBUG "resultType": "vector", 06:35:20 DEBUG "result": [ 06:35:20 DEBUG { 06:35:20 DEBUG "metric": { 06:35:20 DEBUG "instance": "10.142.0.107:9100" 06:35:20 DEBUG }, 06:35:20 DEBUG "value": [ 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG [...] 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG 06:35:20 DEBUG "instance": "10.142.0.77:9100" 06:35:20 DEBUG }, 06:35:20 DEBUG "value": [ 06:35:20 DEBUG 1712813720, 06:35:20 DEBUG "82920430.86666666" 06:35:20 DEBUG ] 06:35:20 DEBUG } 06:35:20 DEBUG ] 06:35:20 DEBUG } 06:35:20 DEBUG } 06:35:20 INFO [http_cmd]: http status code OK 06:35:20 DEBUG --- status code --- 06:35:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:20 DEBUG --- http response --- 06:35:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:35:20 INFO [http_cmd]: http status code OK 06:35:20 DEBUG ----- output ----- 06:35:20 DEBUG --- status code --- 06:35:20 DEBUG { 06:35:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:35:20 DEBUG "status": "success", 06:35:20 DEBUG --- http response --- 06:35:20 DEBUG "data": { 06:35:20 DEBUG "resultType": "vector", 06:35:20 DEBUG "result": [ 06:35:20 DEBUG { 06:35:20 DEBUG "metric": { 06:35:20 DEBUG "instance": "10.142.0.107:9100" 06:35:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:35:20 DEBUG }, 06:35:20 DEBUG ----- output ----- 06:35:20 DEBUG "value": [ 06:35:20 DEBUG { 06:35:20 DEBUG 06:35:20 DEBUG "status": "success", 06:35:20 DEBUG 06:35:20 DEBUG "data": { 06:35:20 DEBUG 06:35:20 DEBUG "resultType": "vector", 06:35:20 DEBUG [...] 06:35:20 DEBUG "result": [ 06:35:20 DEBUG 06:35:20 DEBUG { 06:35:20 DEBUG 06:35:20 DEBUG "metric": { 06:35:20 DEBUG 06:35:20 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:35:20 DEBUG "instance": "10.142.0.77:9100" 06:35:20 DEBUG }, 06:35:20 DEBUG }, 06:35:20 DEBUG "value": [ 06:35:20 DEBUG "value": [ 06:35:20 DEBUG 06:35:20 DEBUG 1712813720, 06:35:20 DEBUG 06:35:20 DEBUG "0.00220000000000032" 06:35:20 DEBUG 06:35:20 DEBUG ] 06:35:20 DEBUG [...] 06:35:20 DEBUG } 06:35:20 DEBUG 06:35:20 DEBUG ] 06:35:20 DEBUG 06:35:20 DEBUG } 06:35:20 DEBUG 06:35:20 DEBUG } 06:35:20 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:35:20 DEBUG }, 06:35:20 DEBUG "value": [ 06:35:20 DEBUG 1712813720, 06:35:20 DEBUG "4.140838628124822" 06:35:20 DEBUG ] 06:35:20 DEBUG } 06:35:20 DEBUG ] 06:35:20 DEBUG } 06:35:20 DEBUG } 06:35:31 INFO 06:35:31 INFO [loop_until]: kubectl --namespace=xlou top pods 06:35:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:35:31 INFO [loop_until]: OK (rc = 0) 06:35:31 DEBUG --- stdout --- 06:35:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 8776m 8505Mi am-5b5bd8567b-8h2w4 8937m 8563Mi am-5b5bd8567b-qlkw6 7431m 8459Mi ds-cts-0 8150m 13386Mi ds-cts-1 8803m 13399Mi ds-cts-2 8541m 12495Mi ds-idrepo-0 11329m 5498Mi ds-idrepo-1 348m 5479Mi ds-idrepo-2 351m 5783Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 3m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 2m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4087m 1845Mi 06:35:31 DEBUG --- stderr --- 06:35:31 DEBUG 06:35:34 INFO 06:35:34 INFO [loop_until]: kubectl --namespace=xlou top node 06:35:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:35:34 INFO [loop_until]: OK (rc = 0) 06:35:34 DEBUG --- stdout --- 06:35:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 2737Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 65m 0% 2772Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7944m 26% 10122Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10766m 36% 2996Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9630m 32% 10094Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 146m 0% 3859Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9571m 32% 10190Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 431m 2% 6458Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11581m 72% 6188Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8640m 54% 13121Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 8110m 51% 14005Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 8622m 54% 14011Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 419m 2% 6154Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4250m 26% 5618Mi 9% 06:35:34 DEBUG --- stderr --- 06:35:34 DEBUG 06:36:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813780" 06:36:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712813780" 06:36:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813780" 06:36:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813780" 06:36:20 INFO [http_cmd]: http status code OK 06:36:20 DEBUG --- status code --- 06:36:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:20 DEBUG --- http response --- 06:36:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:36:20 DEBUG ----- output ----- 06:36:20 DEBUG { 06:36:20 DEBUG "status": "success", 06:36:20 DEBUG "data": { 06:36:20 DEBUG "resultType": "vector", 06:36:20 DEBUG "result": [ 06:36:20 DEBUG { 06:36:20 DEBUG "metric": { 06:36:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:36:20 DEBUG }, 06:36:20 DEBUG "value": [ 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG [...] 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:36:20 DEBUG }, 06:36:20 DEBUG "value": [ 06:36:20 DEBUG 1712813780, 06:36:20 DEBUG "0.021" 06:36:20 DEBUG ] 06:36:20 DEBUG } 06:36:20 DEBUG ] 06:36:20 DEBUG } 06:36:20 DEBUG } 06:36:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813780" 06:36:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813780" 06:36:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813780" 06:36:20 INFO [http_cmd]: http status code OK 06:36:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813780" 06:36:20 DEBUG --- status code --- 06:36:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:20 DEBUG --- http response --- 06:36:20 INFO [http_cmd]: http status code OK 06:36:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:36:20 DEBUG --- status code --- 06:36:20 DEBUG ----- output ----- 06:36:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:20 DEBUG { 06:36:20 DEBUG --- http response --- 06:36:20 DEBUG "status": "success", 06:36:20 DEBUG "data": { 06:36:20 DEBUG "resultType": "vector", 06:36:20 DEBUG "result": [ 06:36:20 DEBUG { 06:36:20 DEBUG "metric": { 06:36:20 DEBUG "instance": "10.142.0.107:9100" 06:36:20 DEBUG }, 06:36:20 DEBUG "value": [ 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG [...] 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG "instance": "10.142.0.77:9100" 06:36:20 DEBUG }, 06:36:20 DEBUG "value": [ 06:36:20 DEBUG 1712813780, 06:36:20 DEBUG "52941589504" 06:36:20 DEBUG ] 06:36:20 DEBUG } 06:36:20 DEBUG ] 06:36:20 DEBUG } 06:36:20 DEBUG } 06:36:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:36:20 DEBUG ----- output ----- 06:36:20 DEBUG { 06:36:20 DEBUG "status": "success", 06:36:20 DEBUG "data": { 06:36:20 DEBUG "resultType": "vector", 06:36:20 DEBUG "result": [ 06:36:20 DEBUG { 06:36:20 DEBUG "metric": { 06:36:20 DEBUG "pod": "ds-idrepo-2" 06:36:20 DEBUG }, 06:36:20 DEBUG "value": [ 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG [...] 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG "pod": "ds-idrepo-0" 06:36:20 DEBUG }, 06:36:20 DEBUG "value": [ 06:36:20 DEBUG 1712813780, 06:36:20 DEBUG "0" 06:36:20 DEBUG ] 06:36:20 INFO [http_cmd]: http status code OK 06:36:20 DEBUG } 06:36:20 DEBUG --- status code --- 06:36:20 DEBUG ] 06:36:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:20 DEBUG } 06:36:20 DEBUG --- http response --- 06:36:20 DEBUG } 06:36:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:36:20 DEBUG ----- output ----- 06:36:20 DEBUG { 06:36:20 DEBUG "status": "success", 06:36:20 DEBUG "data": { 06:36:20 DEBUG "resultType": "vector", 06:36:20 DEBUG "result": [ 06:36:20 DEBUG { 06:36:20 DEBUG "metric": { 06:36:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:36:20 DEBUG }, 06:36:20 DEBUG "value": [ 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG [...] 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:36:20 DEBUG }, 06:36:20 DEBUG "value": [ 06:36:20 DEBUG 1712813780, 06:36:20 DEBUG "244290" 06:36:20 DEBUG ] 06:36:20 DEBUG } 06:36:20 DEBUG ] 06:36:20 DEBUG } 06:36:20 DEBUG } 06:36:20 INFO [http_cmd]: http status code OK 06:36:20 DEBUG --- status code --- 06:36:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:20 DEBUG --- http response --- 06:36:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:36:20 DEBUG ----- output ----- 06:36:20 DEBUG { 06:36:20 DEBUG "status": "success", 06:36:20 DEBUG "data": { 06:36:20 DEBUG "resultType": "vector", 06:36:20 DEBUG "result": [ 06:36:20 DEBUG { 06:36:20 DEBUG "metric": { 06:36:20 DEBUG "pod": "ds-idrepo-2" 06:36:20 DEBUG }, 06:36:20 DEBUG "value": [ 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG [...] 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG "pod": "ds-cts-2" 06:36:20 DEBUG }, 06:36:20 DEBUG "value": [ 06:36:20 DEBUG 1712813780, 06:36:20 DEBUG "27.866666666666667" 06:36:20 DEBUG ] 06:36:20 DEBUG } 06:36:20 DEBUG ] 06:36:20 DEBUG } 06:36:20 DEBUG } 06:36:20 INFO [http_cmd]: http status code OK 06:36:20 DEBUG --- status code --- 06:36:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:20 DEBUG --- http response --- 06:36:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:36:20 DEBUG ----- output ----- 06:36:20 DEBUG { 06:36:20 DEBUG "status": "success", 06:36:20 DEBUG "data": { 06:36:20 DEBUG "resultType": "vector", 06:36:20 DEBUG "result": [ 06:36:20 DEBUG { 06:36:20 DEBUG "metric": { 06:36:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:36:20 DEBUG }, 06:36:20 DEBUG "value": [ 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG [...] 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:36:20 DEBUG }, 06:36:20 DEBUG "value": [ 06:36:20 DEBUG 1712813780, 06:36:20 DEBUG "0" 06:36:20 DEBUG ] 06:36:20 DEBUG } 06:36:20 DEBUG ] 06:36:20 DEBUG } 06:36:20 DEBUG } 06:36:20 INFO [http_cmd]: http status code OK 06:36:20 DEBUG --- status code --- 06:36:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:20 DEBUG --- http response --- 06:36:20 INFO [http_cmd]: http status code OK 06:36:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712813780" 06:36:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:36:20 DEBUG --- status code --- 06:36:20 DEBUG ----- output ----- 06:36:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:20 DEBUG { 06:36:20 DEBUG --- http response --- 06:36:20 DEBUG "status": "success", 06:36:20 DEBUG "data": { 06:36:20 DEBUG "resultType": "vector", 06:36:20 DEBUG "result": [ 06:36:20 DEBUG { 06:36:20 DEBUG "metric": { 06:36:20 DEBUG "pod": "ds-idrepo-2" 06:36:20 DEBUG }, 06:36:20 DEBUG "value": [ 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG [...] 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 06:36:20 DEBUG "pod": "ds-cts-2" 06:36:20 DEBUG ----- output ----- 06:36:20 DEBUG }, 06:36:20 DEBUG { 06:36:20 DEBUG "value": [ 06:36:20 DEBUG "status": "success", 06:36:20 DEBUG 1712813780, 06:36:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813780" 06:36:20 DEBUG "data": { 06:36:20 DEBUG "0.00006292722222222222" 06:36:20 DEBUG "resultType": "vector", 06:36:20 DEBUG ] 06:36:20 DEBUG "result": [ 06:36:20 DEBUG } 06:36:20 DEBUG { 06:36:20 DEBUG ] 06:36:20 DEBUG "metric": { 06:36:20 DEBUG } 06:36:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:36:20 DEBUG } 06:36:20 DEBUG }, 06:36:20 DEBUG "value": [ 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG [...] 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:36:20 DEBUG }, 06:36:20 DEBUG "value": [ 06:36:20 DEBUG 1712813780, 06:36:20 DEBUG "0" 06:36:20 DEBUG ] 06:36:20 DEBUG } 06:36:20 DEBUG ] 06:36:20 DEBUG } 06:36:20 DEBUG } 06:36:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813780" 06:36:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813780" 06:36:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813780" 06:36:20 ERROR [http_cmd]: ERROR 06:36:20 INFO [http_cmd]: http status code OK 06:36:20 ERROR --- status code --- 06:36:20 DEBUG --- status code --- 06:36:20 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:36:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:20 ERROR --- http response --- 06:36:20 DEBUG --- http response --- 06:36:20 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:36:20 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:36:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:36:20 DEBUG ----- output ----- 06:36:20 DEBUG { 06:36:20 DEBUG "status": "success", 06:36:20 DEBUG "data": { 06:36:20 DEBUG "resultType": "vector", 06:36:20 DEBUG "result": [ 06:36:20 DEBUG { 06:36:20 DEBUG "metric": { 06:36:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:36:20 DEBUG }, 06:36:20 DEBUG "value": [ 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG [...] 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:36:20 DEBUG }, 06:36:20 DEBUG "value": [ 06:36:20 DEBUG 1712813780, 06:36:20 DEBUG "1373.6" 06:36:20 DEBUG ] 06:36:20 DEBUG } 06:36:20 DEBUG ] 06:36:20 DEBUG } 06:36:20 DEBUG } 06:36:20 INFO [http_cmd]: http status code OK 06:36:20 INFO [http_cmd]: http status code OK 06:36:20 DEBUG --- status code --- 06:36:20 DEBUG --- status code --- 06:36:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:20 DEBUG --- http response --- 06:36:20 DEBUG --- http response --- 06:36:20 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:36:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:36:20 DEBUG ----- output ----- 06:36:20 DEBUG { 06:36:20 DEBUG "status": "success", 06:36:20 DEBUG "data": { 06:36:20 DEBUG "resultType": "vector", 06:36:20 DEBUG "result": [ 06:36:20 DEBUG { 06:36:20 DEBUG "metric": { 06:36:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:36:20 DEBUG }, 06:36:20 DEBUG "value": [ 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG [...] 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:36:20 DEBUG }, 06:36:20 DEBUG "value": [ 06:36:20 DEBUG 1712813780, 06:36:20 DEBUG "1358.1666666666667" 06:36:20 DEBUG ] 06:36:20 DEBUG } 06:36:20 DEBUG ] 06:36:20 DEBUG } 06:36:20 DEBUG } 06:36:20 INFO [http_cmd]: http status code OK 06:36:20 DEBUG --- status code --- 06:36:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:20 DEBUG --- http response --- 06:36:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:36:20 DEBUG ----- output ----- 06:36:20 DEBUG { 06:36:20 DEBUG "status": "success", 06:36:20 DEBUG "data": { 06:36:20 DEBUG "resultType": "vector", 06:36:20 DEBUG "result": [ 06:36:20 DEBUG { 06:36:20 DEBUG "metric": { 06:36:20 DEBUG "instance": "10.142.0.107:9100" 06:36:20 DEBUG }, 06:36:20 DEBUG "value": [ 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG [...] 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG "instance": "10.142.0.77:9100" 06:36:20 DEBUG }, 06:36:20 DEBUG "value": [ 06:36:20 DEBUG 1712813780, 06:36:20 DEBUG "82359146.3" 06:36:20 DEBUG ] 06:36:20 DEBUG } 06:36:20 DEBUG ] 06:36:20 DEBUG } 06:36:20 DEBUG } 06:36:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813780" 06:36:20 INFO [http_cmd]: http status code OK 06:36:20 DEBUG --- status code --- 06:36:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:20 DEBUG --- http response --- 06:36:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:36:20 DEBUG ----- output ----- 06:36:20 DEBUG { 06:36:20 DEBUG "status": "success", 06:36:20 DEBUG "data": { 06:36:20 DEBUG "resultType": "vector", 06:36:20 DEBUG "result": [ 06:36:20 DEBUG { 06:36:20 DEBUG "metric": { 06:36:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:36:20 DEBUG }, 06:36:20 DEBUG "value": [ 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG [...] 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:36:20 DEBUG }, 06:36:20 DEBUG "value": [ 06:36:20 DEBUG 1712813780, 06:36:20 DEBUG "2.905372876311897" 06:36:20 DEBUG ] 06:36:20 DEBUG } 06:36:20 DEBUG ] 06:36:20 DEBUG } 06:36:20 DEBUG } 06:36:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813780" 06:36:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813780" 06:36:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813780" 06:36:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813780" 06:36:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712813780" 06:36:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712813780" 06:36:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813780" 06:36:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813780" 06:36:20 INFO [http_cmd]: http status code OK 06:36:20 INFO [http_cmd]: http status code OK 06:36:20 DEBUG --- status code --- 06:36:20 INFO [http_cmd]: http status code OK 06:36:20 INFO [http_cmd]: http status code OK 06:36:20 DEBUG --- status code --- 06:36:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:20 DEBUG --- status code --- 06:36:20 INFO [http_cmd]: http status code OK 06:36:20 DEBUG --- status code --- 06:36:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:20 DEBUG --- http response --- 06:36:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:20 DEBUG --- status code --- 06:36:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:20 DEBUG --- http response --- 06:36:20 DEBUG --- http response --- 06:36:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813780" 06:36:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813780" 06:36:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:36:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:20 DEBUG --- http response --- 06:36:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:36:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:36:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813780" 06:36:20 INFO [http_cmd]: http status code OK 06:36:20 INFO [http_cmd]: http status code OK 06:36:20 INFO [http_cmd]: http status code OK 06:36:20 DEBUG ----- output ----- 06:36:20 DEBUG --- http response --- 06:36:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 06:36:20 DEBUG ----- output ----- 06:36:20 DEBUG ----- output ----- 06:36:20 DEBUG --- status code --- 06:36:20 DEBUG --- status code --- 06:36:20 DEBUG --- status code --- 06:36:20 DEBUG { 06:36:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:36:20 DEBUG ----- output ----- 06:36:20 DEBUG { 06:36:20 DEBUG { 06:36:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:20 DEBUG "status": "success", 06:36:20 DEBUG ----- output ----- 06:36:20 DEBUG { 06:36:20 DEBUG "status": "success", 06:36:20 DEBUG "status": "success", 06:36:20 DEBUG --- http response --- 06:36:20 DEBUG --- http response --- 06:36:20 DEBUG --- http response --- 06:36:20 DEBUG "data": { 06:36:20 DEBUG { 06:36:20 DEBUG "status": "success", 06:36:20 DEBUG "data": { 06:36:20 DEBUG "data": { 06:36:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:36:20 DEBUG "resultType": "vector", 06:36:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:36:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:36:20 DEBUG "status": "success", 06:36:20 DEBUG "data": { 06:36:20 DEBUG "resultType": "vector", 06:36:20 INFO [http_cmd]: http status code OK 06:36:20 DEBUG "resultType": "vector", 06:36:20 DEBUG ----- output ----- 06:36:20 DEBUG "result": [ 06:36:20 DEBUG ----- output ----- 06:36:20 DEBUG ----- output ----- 06:36:20 DEBUG "data": { 06:36:20 DEBUG "resultType": "vector", 06:36:20 DEBUG "result": [ 06:36:20 DEBUG --- status code --- 06:36:20 DEBUG "result": [ 06:36:20 DEBUG { 06:36:20 DEBUG { 06:36:20 DEBUG { 06:36:20 DEBUG { 06:36:20 DEBUG "resultType": "vector", 06:36:20 DEBUG "result": [ 06:36:20 DEBUG { 06:36:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813780" 06:36:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:20 DEBUG { 06:36:20 DEBUG "status": "success", 06:36:20 DEBUG "metric": { 06:36:20 DEBUG "status": "success", 06:36:20 DEBUG "status": "success", 06:36:20 DEBUG "result": [ 06:36:20 DEBUG { 06:36:20 DEBUG "metric": { 06:36:20 DEBUG --- http response --- 06:36:20 DEBUG "metric": { 06:36:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712813780" 06:36:20 INFO [http_cmd]: http status code OK 06:36:20 DEBUG "data": { 06:36:20 DEBUG "pod": "ds-idrepo-2" 06:36:20 DEBUG "data": { 06:36:20 DEBUG "data": { 06:36:20 DEBUG { 06:36:20 DEBUG "metric": { 06:36:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:36:20 DEBUG "pod": "ds-idrepo-2" 06:36:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:36:20 DEBUG --- status code --- 06:36:20 DEBUG "resultType": "vector", 06:36:20 INFO [http_cmd]: http status code OK 06:36:20 DEBUG }, 06:36:20 DEBUG "resultType": "vector", 06:36:20 DEBUG "resultType": "vector", 06:36:20 DEBUG "metric": { 06:36:20 DEBUG "pod": "ds-idrepo-2" 06:36:20 DEBUG }, 06:36:20 DEBUG }, 06:36:20 DEBUG ----- output ----- 06:36:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:20 DEBUG "result": [ 06:36:20 DEBUG --- status code --- 06:36:20 DEBUG "value": [ 06:36:20 DEBUG "result": [ 06:36:20 DEBUG "result": [ 06:36:20 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:36:20 DEBUG }, 06:36:20 DEBUG "value": [ 06:36:20 DEBUG "value": [ 06:36:20 DEBUG { 06:36:20 DEBUG --- http response --- 06:36:20 DEBUG { 06:36:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:20 DEBUG 06:36:20 DEBUG { 06:36:20 DEBUG { 06:36:20 DEBUG }, 06:36:20 DEBUG "value": [ 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG "status": "success", 06:36:20 DEBUG "metric": { 06:36:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:36:20 DEBUG --- http response --- 06:36:20 DEBUG 06:36:20 DEBUG "metric": { 06:36:20 DEBUG "metric": { 06:36:20 DEBUG "value": [ 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG "data": { 06:36:20 DEBUG "pod": "ds-idrepo-2" 06:36:20 DEBUG ----- output ----- 06:36:20 DEBUG 06:36:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:36:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:36:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG "resultType": "vector", 06:36:20 DEBUG }, 06:36:20 DEBUG { 06:36:20 DEBUG [...] 06:36:20 DEBUG ----- output ----- 06:36:20 DEBUG }, 06:36:20 DEBUG }, 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG [...] 06:36:20 DEBUG "result": [ 06:36:20 DEBUG "value": [ 06:36:20 DEBUG "status": "success", 06:36:20 DEBUG 06:36:20 DEBUG { 06:36:20 DEBUG "value": [ 06:36:20 DEBUG "value": [ 06:36:20 DEBUG [...] 06:36:20 DEBUG [...] 06:36:20 DEBUG [...] 06:36:20 DEBUG 06:36:20 DEBUG { 06:36:20 DEBUG 06:36:20 DEBUG "data": { 06:36:20 DEBUG 06:36:20 DEBUG "status": "success", 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG "metric": { 06:36:20 DEBUG 06:36:20 DEBUG "resultType": "vector", 06:36:20 DEBUG 06:36:20 INFO [http_cmd]: http status code OK 06:36:20 DEBUG "data": { 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:36:20 DEBUG 06:36:20 DEBUG "result": [ 06:36:20 DEBUG "pod": "ds-idrepo-0" 06:36:20 DEBUG --- status code --- 06:36:20 DEBUG "resultType": "vector", 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG "pod": "ds-idrepo-0" 06:36:20 DEBUG }, 06:36:20 DEBUG [...] 06:36:20 DEBUG { 06:36:20 DEBUG }, 06:36:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:20 DEBUG "result": [ 06:36:20 DEBUG [...] 06:36:20 DEBUG [...] 06:36:20 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:36:20 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 06:36:20 DEBUG "pod": "ds-cts-2" 06:36:20 DEBUG }, 06:36:20 DEBUG "value": [ 06:36:20 DEBUG 06:36:20 DEBUG "metric": { 06:36:20 DEBUG "instance": "10.142.0.107:9100" 06:36:20 DEBUG "value": [ 06:36:20 DEBUG { 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG }, 06:36:20 DEBUG }, 06:36:20 DEBUG }, 06:36:20 DEBUG "value": [ 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG --- http response --- 06:36:20 DEBUG }, 06:36:20 DEBUG "value": [ 06:36:20 DEBUG "metric": { 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG "value": [ 06:36:20 DEBUG "value": [ 06:36:20 DEBUG "value": [ 06:36:20 INFO [http_cmd]: http status code OK 06:36:20 DEBUG 1712813780, 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 1712813780, 06:36:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:36:20 DEBUG "instance": "10.142.0.107:9100" 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 1712813780, 06:36:20 DEBUG 1712813780, 06:36:20 DEBUG 1712813780, 06:36:20 DEBUG --- status code --- 06:36:20 DEBUG "0" 06:36:20 DEBUG [...] 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG "0" 06:36:20 DEBUG ----- output ----- 06:36:20 DEBUG }, 06:36:20 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:36:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:36:20 DEBUG "0" 06:36:20 DEBUG "13582405632" 06:36:20 DEBUG "0" 06:36:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:36:20 DEBUG ] 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG ] 06:36:20 DEBUG { 06:36:20 DEBUG "value": [ 06:36:20 DEBUG }, 06:36:20 DEBUG }, 06:36:20 DEBUG ] 06:36:20 DEBUG ] 06:36:20 DEBUG ] 06:36:20 DEBUG --- http response --- 06:36:20 DEBUG } 06:36:20 DEBUG ] 06:36:20 DEBUG "pod": "ds-cts-2" 06:36:20 DEBUG [...] 06:36:20 DEBUG 06:36:20 DEBUG "status": "success", 06:36:20 DEBUG 06:36:20 DEBUG "value": [ 06:36:20 DEBUG "value": [ 06:36:20 DEBUG } 06:36:20 DEBUG } 06:36:20 DEBUG } 06:36:20 DEBUG 06:36:20 DEBUG } 06:36:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:36:20 DEBUG }, 06:36:20 DEBUG } 06:36:20 DEBUG 06:36:20 DEBUG "data": { 06:36:20 DEBUG 06:36:20 DEBUG 1712813780, 06:36:20 DEBUG "60449879.26557837" 06:36:20 DEBUG ] 06:36:20 DEBUG ] 06:36:20 DEBUG ] 06:36:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:36:20 DEBUG } 06:36:20 DEBUG ----- output ----- 06:36:20 DEBUG "value": [ 06:36:20 DEBUG ] 06:36:20 DEBUG 06:36:20 DEBUG "resultType": "vector", 06:36:20 DEBUG 06:36:20 DEBUG 1712813780, 06:36:20 DEBUG ] 06:36:20 DEBUG } 06:36:20 DEBUG } 06:36:20 DEBUG } 06:36:20 DEBUG }, 06:36:20 DEBUG { 06:36:20 DEBUG 1712813780, 06:36:20 DEBUG } 06:36:20 DEBUG "instance": "10.142.0.77:9100" 06:36:20 DEBUG "result": [ 06:36:20 DEBUG [...] 06:36:20 DEBUG "1358.9333333333334" 06:36:20 DEBUG } 06:36:20 DEBUG } 06:36:20 DEBUG } 06:36:20 DEBUG } 06:36:20 DEBUG "value": [ 06:36:20 DEBUG "status": "success", 06:36:20 DEBUG "0" 06:36:20 DEBUG } 06:36:20 DEBUG }, 06:36:20 DEBUG { 06:36:20 DEBUG "metric": { 06:36:20 DEBUG ] 06:36:20 DEBUG ] 06:36:20 DEBUG } 06:36:20 DEBUG "data": { 06:36:20 DEBUG ] 06:36:20 DEBUG "value": [ 06:36:20 DEBUG "instance": "10.142.0.107:9100" 06:36:20 DEBUG 06:36:20 DEBUG } 06:36:20 DEBUG 1712813780, 06:36:20 DEBUG } 06:36:20 DEBUG "resultType": "vector", 06:36:20 DEBUG } 06:36:20 DEBUG 1712813780, 06:36:20 DEBUG }, 06:36:20 DEBUG 06:36:20 DEBUG ] 06:36:20 DEBUG "1352.4333333333334" 06:36:20 DEBUG "result": [ 06:36:20 DEBUG ] 06:36:20 DEBUG "0" 06:36:20 DEBUG "value": [ 06:36:20 DEBUG 06:36:20 DEBUG } 06:36:20 DEBUG ] 06:36:20 DEBUG { 06:36:20 DEBUG } 06:36:20 DEBUG ] 06:36:20 DEBUG 06:36:20 DEBUG "instance": "10.142.0.77:9100" 06:36:20 DEBUG } 06:36:20 DEBUG } 06:36:20 DEBUG "metric": { 06:36:20 DEBUG } 06:36:20 DEBUG } 06:36:20 DEBUG 06:36:20 DEBUG }, 06:36:20 DEBUG ] 06:36:20 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:36:20 DEBUG ] 06:36:20 DEBUG 06:36:20 DEBUG "value": [ 06:36:20 DEBUG } 06:36:20 DEBUG }, 06:36:20 DEBUG } 06:36:20 DEBUG [...] 06:36:20 DEBUG 1712813780, 06:36:20 DEBUG } 06:36:20 DEBUG "value": [ 06:36:20 DEBUG } 06:36:20 DEBUG 06:36:20 DEBUG "82356474.36666667" 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG ] 06:36:20 DEBUG 06:36:20 DEBUG 06:36:20 DEBUG } 06:36:20 DEBUG 06:36:20 DEBUG "instance": "10.142.0.77:9100" 06:36:20 DEBUG ] 06:36:20 DEBUG [...] 06:36:20 DEBUG }, 06:36:20 DEBUG } 06:36:20 DEBUG 06:36:20 DEBUG "value": [ 06:36:20 DEBUG } 06:36:20 DEBUG 06:36:20 DEBUG 1712813780, 06:36:20 DEBUG 06:36:20 DEBUG "0.0022666666666668316" 06:36:20 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:36:20 DEBUG ] 06:36:20 DEBUG }, 06:36:20 DEBUG } 06:36:20 DEBUG "value": [ 06:36:20 DEBUG ] 06:36:20 DEBUG 1712813780, 06:36:20 DEBUG } 06:36:20 DEBUG "4.1021196499867125" 06:36:20 DEBUG } 06:36:20 DEBUG ] 06:36:20 DEBUG } 06:36:20 DEBUG ] 06:36:20 DEBUG } 06:36:20 DEBUG } 06:36:31 INFO 06:36:31 INFO [loop_until]: kubectl --namespace=xlou top pods 06:36:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:36:31 INFO [loop_until]: OK (rc = 0) 06:36:31 DEBUG --- stdout --- 06:36:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 9061m 8503Mi am-5b5bd8567b-8h2w4 8878m 8562Mi am-5b5bd8567b-qlkw6 7595m 8466Mi ds-cts-0 8219m 13121Mi ds-cts-1 8650m 12849Mi ds-cts-2 8609m 13006Mi ds-idrepo-0 11698m 5498Mi ds-idrepo-1 394m 5480Mi ds-idrepo-2 413m 5783Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 4m 1010Mi idm-668d898cfd-st2t7 4m 1136Mi lodemon-ccc964465-p7gdm 2m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4129m 1847Mi 06:36:31 DEBUG --- stderr --- 06:36:31 DEBUG 06:36:35 INFO 06:36:35 INFO [loop_until]: kubectl --namespace=xlou top node 06:36:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:36:35 INFO [loop_until]: OK (rc = 0) 06:36:35 DEBUG --- stdout --- 06:36:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 2738Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 62m 0% 2780Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7875m 26% 10119Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10667m 35% 2985Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9438m 31% 10081Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 145m 0% 3853Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9597m 32% 10188Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 484m 3% 6459Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11558m 72% 6185Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8753m 55% 13729Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 7929m 49% 13641Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 8618m 54% 13504Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 399m 2% 6156Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4223m 26% 5618Mi 9% 06:36:35 DEBUG --- stderr --- 06:36:35 DEBUG 06:37:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813840" 06:37:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813840" 06:37:20 INFO [http_cmd]: http status code OK 06:37:20 DEBUG --- status code --- 06:37:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:20 DEBUG --- http response --- 06:37:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:37:20 DEBUG ----- output ----- 06:37:20 DEBUG { 06:37:20 DEBUG "status": "success", 06:37:20 DEBUG "data": { 06:37:20 DEBUG "resultType": "vector", 06:37:20 DEBUG "result": [ 06:37:20 DEBUG { 06:37:20 DEBUG "metric": { 06:37:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:37:20 DEBUG }, 06:37:20 DEBUG "value": [ 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG [...] 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:37:20 DEBUG }, 06:37:20 DEBUG "value": [ 06:37:20 DEBUG 1712813840, 06:37:20 DEBUG "244529" 06:37:20 DEBUG ] 06:37:20 DEBUG } 06:37:20 DEBUG ] 06:37:20 DEBUG } 06:37:20 DEBUG } 06:37:20 INFO [http_cmd]: http status code OK 06:37:20 DEBUG --- status code --- 06:37:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:20 DEBUG --- http response --- 06:37:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:37:20 DEBUG ----- output ----- 06:37:20 DEBUG { 06:37:20 DEBUG "status": "success", 06:37:20 DEBUG "data": { 06:37:20 DEBUG "resultType": "vector", 06:37:20 DEBUG "result": [ 06:37:20 DEBUG { 06:37:20 DEBUG "metric": { 06:37:20 DEBUG "pod": "ds-idrepo-2" 06:37:20 DEBUG }, 06:37:20 DEBUG "value": [ 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG [...] 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG "pod": "ds-cts-2" 06:37:20 DEBUG }, 06:37:20 DEBUG "value": [ 06:37:20 DEBUG 1712813840, 06:37:20 DEBUG "29.866666666666667" 06:37:20 DEBUG ] 06:37:20 DEBUG } 06:37:20 DEBUG ] 06:37:20 DEBUG } 06:37:20 DEBUG } 06:37:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712813840" 06:37:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813840" 06:37:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712813840" 06:37:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813840" 06:37:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813840" 06:37:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813840" 06:37:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813840" 06:37:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813840" 06:37:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813840" 06:37:20 ERROR [http_cmd]: ERROR 06:37:20 INFO [http_cmd]: http status code OK 06:37:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813840" 06:37:20 INFO [http_cmd]: http status code OK 06:37:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813840" 06:37:20 ERROR --- status code --- 06:37:20 DEBUG --- status code --- 06:37:20 DEBUG --- status code --- 06:37:20 INFO [http_cmd]: http status code OK 06:37:20 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:37:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:20 DEBUG --- status code --- 06:37:20 ERROR --- http response --- 06:37:20 DEBUG --- http response --- 06:37:20 DEBUG --- http response --- 06:37:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:20 INFO [http_cmd]: http status code OK 06:37:20 INFO [http_cmd]: http status code OK 06:37:20 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:37:20 INFO [http_cmd]: http status code OK 06:37:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:37:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:37:20 DEBUG --- http response --- 06:37:20 DEBUG --- status code --- 06:37:20 DEBUG --- status code --- 06:37:20 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:37:20 DEBUG --- status code --- 06:37:20 DEBUG ----- output ----- 06:37:20 DEBUG ----- output ----- 06:37:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:37:20 INFO [http_cmd]: http status code OK 06:37:20 INFO [http_cmd]: http status code OK 06:37:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:20 INFO [http_cmd]: http status code OK 06:37:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813840" 06:37:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:20 DEBUG { 06:37:20 DEBUG { 06:37:20 DEBUG ----- output ----- 06:37:20 DEBUG --- status code --- 06:37:20 DEBUG --- status code --- 06:37:20 DEBUG --- http response --- 06:37:20 DEBUG --- status code --- 06:37:20 DEBUG --- http response --- 06:37:20 DEBUG --- http response --- 06:37:20 INFO [http_cmd]: http status code OK 06:37:20 DEBUG "status": "success", 06:37:20 DEBUG "status": "success", 06:37:20 DEBUG { 06:37:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:37:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:37:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:37:20 DEBUG --- status code --- 06:37:20 DEBUG "data": { 06:37:20 DEBUG "data": { 06:37:20 DEBUG "status": "success", 06:37:20 DEBUG --- http response --- 06:37:20 DEBUG --- http response --- 06:37:20 DEBUG ----- output ----- 06:37:20 DEBUG --- http response --- 06:37:20 DEBUG ----- output ----- 06:37:20 DEBUG ----- output ----- 06:37:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:20 DEBUG "resultType": "vector", 06:37:20 DEBUG "resultType": "vector", 06:37:20 DEBUG "data": { 06:37:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:37:20 DEBUG { 06:37:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:37:20 DEBUG { 06:37:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:37:20 DEBUG { 06:37:20 DEBUG --- http response --- 06:37:20 DEBUG "result": [ 06:37:20 DEBUG "result": [ 06:37:20 DEBUG "resultType": "vector", 06:37:20 DEBUG ----- output ----- 06:37:20 DEBUG "status": "success", 06:37:20 DEBUG ----- output ----- 06:37:20 DEBUG "status": "success", 06:37:20 DEBUG ----- output ----- 06:37:20 DEBUG "status": "success", 06:37:20 DEBUG { 06:37:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:37:20 DEBUG { 06:37:20 DEBUG "result": [ 06:37:20 DEBUG { 06:37:20 DEBUG "data": { 06:37:20 DEBUG { 06:37:20 DEBUG "data": { 06:37:20 DEBUG { 06:37:20 DEBUG "data": { 06:37:20 DEBUG "metric": { 06:37:20 DEBUG ----- output ----- 06:37:20 DEBUG "metric": { 06:37:20 DEBUG { 06:37:20 DEBUG "status": "success", 06:37:20 DEBUG "resultType": "vector", 06:37:20 DEBUG "status": "success", 06:37:20 DEBUG "resultType": "vector", 06:37:20 DEBUG "status": "success", 06:37:20 DEBUG "resultType": "vector", 06:37:20 DEBUG "pod": "ds-idrepo-2" 06:37:20 DEBUG { 06:37:20 DEBUG "instance": "10.142.0.107:9100" 06:37:20 DEBUG "metric": { 06:37:20 DEBUG "data": { 06:37:20 DEBUG "result": [ 06:37:20 DEBUG "data": { 06:37:20 DEBUG "result": [ 06:37:20 DEBUG "data": { 06:37:20 DEBUG "result": [ 06:37:20 DEBUG }, 06:37:20 DEBUG "status": "success", 06:37:20 DEBUG }, 06:37:20 DEBUG "instance": "10.142.0.107:9100" 06:37:20 DEBUG "resultType": "vector", 06:37:20 DEBUG { 06:37:20 DEBUG "resultType": "vector", 06:37:20 DEBUG { 06:37:20 DEBUG "resultType": "vector", 06:37:20 DEBUG { 06:37:20 DEBUG "value": [ 06:37:20 DEBUG "data": { 06:37:20 DEBUG "value": [ 06:37:20 DEBUG }, 06:37:20 DEBUG "result": [ 06:37:20 DEBUG "metric": { 06:37:20 DEBUG "result": [ 06:37:20 DEBUG "metric": { 06:37:20 DEBUG "result": [ 06:37:20 DEBUG "metric": { 06:37:20 DEBUG 06:37:20 DEBUG "resultType": "vector", 06:37:20 DEBUG 06:37:20 DEBUG "value": [ 06:37:20 DEBUG { 06:37:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:37:20 DEBUG { 06:37:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:37:20 DEBUG { 06:37:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:37:20 DEBUG 06:37:20 DEBUG "result": [ 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG }, 06:37:20 DEBUG "metric": { 06:37:20 DEBUG }, 06:37:20 DEBUG "metric": { 06:37:20 DEBUG }, 06:37:20 DEBUG 06:37:20 DEBUG { 06:37:20 DEBUG "metric": { 06:37:20 DEBUG [...] 06:37:20 DEBUG 06:37:20 DEBUG "value": [ 06:37:20 INFO [http_cmd]: http status code OK 06:37:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:37:20 DEBUG "value": [ 06:37:20 DEBUG "pod": "ds-idrepo-2" 06:37:20 DEBUG "value": [ 06:37:20 DEBUG [...] 06:37:20 DEBUG "metric": { 06:37:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG --- status code --- 06:37:20 DEBUG }, 06:37:20 DEBUG 06:37:20 DEBUG }, 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:37:20 DEBUG }, 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:20 DEBUG "value": [ 06:37:20 DEBUG 06:37:20 DEBUG "value": [ 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG }, 06:37:20 DEBUG "value": [ 06:37:20 DEBUG [...] 06:37:20 DEBUG "instance": "10.142.0.77:9100" 06:37:20 DEBUG 06:37:20 DEBUG --- http response --- 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG "value": [ 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG }, 06:37:20 DEBUG [...] 06:37:20 DEBUG 06:37:20 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:37:20 DEBUG [...] 06:37:20 DEBUG 06:37:20 DEBUG [...] 06:37:20 DEBUG "pod": "ds-idrepo-0" 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG "value": [ 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG }, 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 1712813840, 06:37:20 DEBUG 06:37:20 DEBUG [...] 06:37:20 DEBUG 06:37:20 DEBUG [...] 06:37:20 DEBUG 06:37:20 DEBUG "value": [ 06:37:20 DEBUG 06:37:20 DEBUG [...] 06:37:20 DEBUG "instance": "10.142.0.77:9100" 06:37:20 DEBUG "52943039488" 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 1712813840, 06:37:20 DEBUG [...] 06:37:20 DEBUG 06:37:20 DEBUG }, 06:37:20 DEBUG ] 06:37:20 DEBUG } 06:37:20 DEBUG 06:37:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:37:20 DEBUG 06:37:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:37:20 DEBUG "0" 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG "value": [ 06:37:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:37:20 DEBUG ] 06:37:20 DEBUG 06:37:20 DEBUG }, 06:37:20 DEBUG 06:37:20 DEBUG }, 06:37:20 DEBUG ] 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 1712813840, 06:37:20 DEBUG }, 06:37:20 DEBUG } 06:37:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:37:20 DEBUG "value": [ 06:37:20 DEBUG "pod": "ds-cts-2" 06:37:20 DEBUG "value": [ 06:37:20 DEBUG } 06:37:20 DEBUG 06:37:20 DEBUG "pod": "ds-idrepo-1" 06:37:20 DEBUG "82416718.36666667" 06:37:20 DEBUG "value": [ 06:37:20 DEBUG } 06:37:20 DEBUG }, 06:37:20 DEBUG 1712813840, 06:37:20 DEBUG }, 06:37:20 DEBUG 1712813840, 06:37:20 DEBUG ] 06:37:20 DEBUG "pod": "ds-idrepo-1" 06:37:20 DEBUG }, 06:37:20 DEBUG ] 06:37:20 DEBUG 1712813840, 06:37:20 DEBUG "value": [ 06:37:20 DEBUG "1359.8" 06:37:20 DEBUG "value": [ 06:37:20 DEBUG "0.021" 06:37:20 DEBUG } 06:37:20 DEBUG }, 06:37:20 DEBUG "value": [ 06:37:20 DEBUG } 06:37:20 DEBUG "0" 06:37:20 DEBUG 1712813840, 06:37:20 DEBUG ] 06:37:20 DEBUG 1712813840, 06:37:20 DEBUG ] 06:37:20 DEBUG } 06:37:20 DEBUG "value": [ 06:37:20 DEBUG 1712813840, 06:37:20 DEBUG ] 06:37:20 DEBUG ] 06:37:20 DEBUG "1350.1" 06:37:20 DEBUG } 06:37:20 DEBUG "0.00003333333333333336" 06:37:20 DEBUG } 06:37:20 DEBUG 1712813840, 06:37:20 DEBUG "0.3552029871508685" 06:37:20 DEBUG } 06:37:20 DEBUG } 06:37:20 DEBUG ] 06:37:20 DEBUG ] 06:37:20 DEBUG ] 06:37:20 DEBUG ] 06:37:20 DEBUG "2930.4029304029305" 06:37:20 DEBUG ] 06:37:20 DEBUG } 06:37:20 DEBUG ] 06:37:20 DEBUG } 06:37:20 DEBUG } 06:37:20 DEBUG } 06:37:20 DEBUG } 06:37:20 DEBUG ] 06:37:20 DEBUG } 06:37:20 DEBUG } 06:37:20 DEBUG ] 06:37:20 DEBUG } 06:37:20 DEBUG ] 06:37:20 DEBUG } 06:37:20 DEBUG } 06:37:20 DEBUG ] 06:37:20 DEBUG } 06:37:20 DEBUG } 06:37:20 DEBUG } 06:37:20 DEBUG ] 06:37:20 DEBUG } 06:37:20 DEBUG } 06:37:20 DEBUG } 06:37:20 DEBUG } 06:37:20 DEBUG } 06:37:20 DEBUG } 06:37:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813840" 06:37:20 INFO [http_cmd]: http status code OK 06:37:20 DEBUG --- status code --- 06:37:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:20 DEBUG --- http response --- 06:37:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:37:20 DEBUG ----- output ----- 06:37:20 DEBUG { 06:37:20 DEBUG "status": "success", 06:37:20 DEBUG "data": { 06:37:20 DEBUG "resultType": "vector", 06:37:20 DEBUG "result": [ 06:37:20 DEBUG { 06:37:20 DEBUG "metric": { 06:37:20 DEBUG "instance": "10.142.0.107:9100" 06:37:20 DEBUG }, 06:37:20 DEBUG "value": [ 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG [...] 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG "instance": "10.142.0.77:9100" 06:37:20 DEBUG }, 06:37:20 DEBUG "value": [ 06:37:20 DEBUG 1712813840, 06:37:20 DEBUG "0.00220000000000032" 06:37:20 DEBUG ] 06:37:20 DEBUG } 06:37:20 DEBUG ] 06:37:20 DEBUG } 06:37:20 DEBUG } 06:37:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813840" 06:37:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712813840" 06:37:20 INFO [http_cmd]: http status code OK 06:37:20 INFO [http_cmd]: http status code OK 06:37:20 DEBUG --- status code --- 06:37:20 DEBUG --- status code --- 06:37:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:20 DEBUG --- http response --- 06:37:20 DEBUG --- http response --- 06:37:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:37:20 DEBUG ----- output ----- 06:37:20 DEBUG { 06:37:20 DEBUG "status": "success", 06:37:20 DEBUG "data": { 06:37:20 DEBUG "resultType": "vector", 06:37:20 DEBUG "result": [ 06:37:20 DEBUG { 06:37:20 DEBUG "metric": { 06:37:20 DEBUG "pod": "ds-idrepo-2" 06:37:20 DEBUG }, 06:37:20 DEBUG "value": [ 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG [...] 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG "pod": "ds-idrepo-0" 06:37:20 DEBUG }, 06:37:20 DEBUG "value": [ 06:37:20 DEBUG 1712813840, 06:37:20 DEBUG "0" 06:37:20 DEBUG ] 06:37:20 DEBUG } 06:37:20 DEBUG ] 06:37:20 DEBUG } 06:37:20 DEBUG } 06:37:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:37:20 DEBUG ----- output ----- 06:37:20 DEBUG { 06:37:20 DEBUG "status": "success", 06:37:20 DEBUG "data": { 06:37:20 DEBUG "resultType": "vector", 06:37:20 DEBUG "result": [ 06:37:20 DEBUG { 06:37:20 DEBUG "metric": { 06:37:20 DEBUG "pod": "ds-idrepo-2" 06:37:20 DEBUG }, 06:37:20 DEBUG "value": [ 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG [...] 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG "pod": "ds-cts-2" 06:37:20 DEBUG }, 06:37:20 DEBUG "value": [ 06:37:20 DEBUG 1712813840, 06:37:20 DEBUG "0" 06:37:20 DEBUG ] 06:37:20 DEBUG } 06:37:20 DEBUG ] 06:37:20 DEBUG } 06:37:20 DEBUG } 06:37:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813840" 06:37:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712813840" 06:37:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813840" 06:37:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813840" 06:37:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813840" 06:37:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813840" 06:37:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813840" 06:37:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813840" 06:37:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813840" 06:37:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712813840" 06:37:20 INFO [http_cmd]: http status code OK 06:37:20 INFO [http_cmd]: http status code OK 06:37:20 INFO [http_cmd]: http status code OK 06:37:20 DEBUG --- status code --- 06:37:20 INFO [http_cmd]: http status code OK 06:37:20 DEBUG --- status code --- 06:37:20 DEBUG --- status code --- 06:37:20 INFO [http_cmd]: http status code OK 06:37:20 INFO [http_cmd]: http status code OK 06:37:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:20 DEBUG --- status code --- 06:37:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:20 DEBUG --- status code --- 06:37:20 DEBUG --- status code --- 06:37:20 DEBUG --- http response --- 06:37:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:20 DEBUG --- http response --- 06:37:20 DEBUG --- http response --- 06:37:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:20 INFO [http_cmd]: http status code OK 06:37:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:20 DEBUG --- http response --- 06:37:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:37:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:37:20 DEBUG --- http response --- 06:37:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:37:20 DEBUG --- status code --- 06:37:20 DEBUG --- http response --- 06:37:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:37:20 DEBUG ----- output ----- 06:37:20 DEBUG ----- output ----- 06:37:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:37:20 DEBUG ----- output ----- 06:37:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:20 INFO [http_cmd]: http status code OK 06:37:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:37:20 DEBUG ----- output ----- 06:37:20 DEBUG { 06:37:20 DEBUG { 06:37:20 DEBUG "status": "success", 06:37:20 DEBUG "data": { 06:37:20 DEBUG --- http response --- 06:37:20 DEBUG --- status code --- 06:37:20 DEBUG ----- output ----- 06:37:20 INFO [http_cmd]: http status code OK 06:37:20 DEBUG { 06:37:20 DEBUG "status": "success", 06:37:20 DEBUG { 06:37:20 DEBUG ----- output ----- 06:37:20 DEBUG "resultType": "vector", 06:37:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:37:20 DEBUG { 06:37:20 DEBUG --- status code --- 06:37:20 DEBUG "status": "success", 06:37:20 DEBUG "data": { 06:37:20 DEBUG "status": "success", 06:37:20 DEBUG { 06:37:20 DEBUG "result": [ 06:37:20 DEBUG --- http response --- 06:37:20 DEBUG ----- output ----- 06:37:20 DEBUG "status": "success", 06:37:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:20 DEBUG "data": { 06:37:20 DEBUG "resultType": "vector", 06:37:20 DEBUG "data": { 06:37:20 DEBUG "status": "success", 06:37:20 DEBUG { 06:37:20 DEBUG { 06:37:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:37:20 DEBUG "data": { 06:37:20 DEBUG --- http response --- 06:37:20 DEBUG "resultType": "vector", 06:37:20 DEBUG "result": [ 06:37:20 DEBUG "resultType": "vector", 06:37:20 DEBUG "data": { 06:37:20 DEBUG "metric": { 06:37:20 DEBUG "status": "success", 06:37:20 DEBUG ----- output ----- 06:37:20 DEBUG "resultType": "vector", 06:37:20 DEBUG "result": [ 06:37:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:37:20 DEBUG { 06:37:20 INFO [http_cmd]: http status code OK 06:37:20 DEBUG "result": [ 06:37:20 DEBUG "resultType": "vector", 06:37:20 DEBUG "pod": "ds-idrepo-2" 06:37:20 DEBUG "data": { 06:37:20 DEBUG { 06:37:20 DEBUG "result": [ 06:37:20 DEBUG { 06:37:20 DEBUG ----- output ----- 06:37:20 DEBUG "metric": { 06:37:20 DEBUG --- status code --- 06:37:20 DEBUG { 06:37:20 DEBUG "result": [ 06:37:20 DEBUG }, 06:37:20 DEBUG "resultType": "vector", 06:37:20 DEBUG "status": "success", 06:37:20 DEBUG { 06:37:20 DEBUG "metric": { 06:37:20 DEBUG { 06:37:20 DEBUG "pod": "ds-idrepo-2" 06:37:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:37:20 DEBUG "metric": { 06:37:20 DEBUG { 06:37:20 DEBUG "value": [ 06:37:20 DEBUG "result": [ 06:37:20 DEBUG "data": { 06:37:20 DEBUG "metric": { 06:37:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:37:20 DEBUG "status": "success", 06:37:20 DEBUG }, 06:37:20 DEBUG --- http response --- 06:37:20 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:37:20 DEBUG "metric": { 06:37:20 DEBUG 06:37:20 DEBUG { 06:37:20 DEBUG "resultType": "vector", 06:37:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:37:20 DEBUG }, 06:37:20 DEBUG "data": { 06:37:20 DEBUG "value": [ 06:37:20 DEBUG }, 06:37:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:37:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:37:20 DEBUG 06:37:20 DEBUG "metric": { 06:37:20 DEBUG "result": [ 06:37:20 DEBUG }, 06:37:20 DEBUG "value": [ 06:37:20 DEBUG "resultType": "vector", 06:37:20 DEBUG 06:37:20 DEBUG "value": [ 06:37:20 DEBUG ----- output ----- 06:37:20 DEBUG }, 06:37:20 DEBUG 06:37:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:37:20 DEBUG { 06:37:20 DEBUG "value": [ 06:37:20 DEBUG 06:37:20 DEBUG "result": [ 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG { 06:37:20 DEBUG "value": [ 06:37:20 DEBUG [...] 06:37:20 DEBUG }, 06:37:20 DEBUG "metric": { 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG { 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG "status": "success", 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG "value": [ 06:37:20 DEBUG "instance": "10.142.0.107:9100" 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG "metric": { 06:37:20 DEBUG [...] 06:37:20 DEBUG 06:37:20 DEBUG "data": { 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG }, 06:37:20 DEBUG 06:37:20 DEBUG [...] 06:37:20 DEBUG "instance": "10.142.0.107:9100" 06:37:20 DEBUG 06:37:20 DEBUG [...] 06:37:20 DEBUG "resultType": "vector", 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG "value": [ 06:37:20 DEBUG [...] 06:37:20 DEBUG 06:37:20 DEBUG }, 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG "result": [ 06:37:20 DEBUG [...] 06:37:20 DEBUG "pod": "ds-idrepo-0" 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG "value": [ 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG { 06:37:20 DEBUG 06:37:20 DEBUG }, 06:37:20 DEBUG [...] 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG "pod": "ds-cts-2" 06:37:20 DEBUG 06:37:20 DEBUG "metric": { 06:37:20 DEBUG 06:37:20 DEBUG "value": [ 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG "pod": "ds-idrepo-1" 06:37:20 DEBUG 06:37:20 DEBUG }, 06:37:20 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 06:37:20 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:37:20 DEBUG 06:37:20 DEBUG 1712813840, 06:37:20 DEBUG 06:37:20 DEBUG [...] 06:37:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:37:20 DEBUG }, 06:37:20 DEBUG 06:37:20 DEBUG "value": [ 06:37:20 DEBUG }, 06:37:20 DEBUG }, 06:37:20 DEBUG "pod": "ds-idrepo-1" 06:37:20 DEBUG "0" 06:37:20 DEBUG 06:37:20 DEBUG 06:37:20 DEBUG }, 06:37:20 DEBUG "value": [ 06:37:20 DEBUG [...] 06:37:20 DEBUG 1712813840, 06:37:20 DEBUG "value": [ 06:37:20 DEBUG "value": [ 06:37:20 DEBUG }, 06:37:20 DEBUG ] 06:37:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:37:20 DEBUG 06:37:20 DEBUG "value": [ 06:37:20 DEBUG 1712813840, 06:37:20 DEBUG 06:37:20 DEBUG "0" 06:37:20 DEBUG 1712813840, 06:37:20 DEBUG 06:37:20 DEBUG "value": [ 06:37:20 DEBUG } 06:37:20 DEBUG }, 06:37:20 DEBUG 06:37:20 DEBUG 1712813840, 06:37:20 DEBUG "937.7289377289378" 06:37:20 DEBUG 06:37:20 DEBUG ] 06:37:20 DEBUG "14136094720" 06:37:20 DEBUG 06:37:20 DEBUG 1712813840, 06:37:20 DEBUG ] 06:37:20 DEBUG "value": [ 06:37:20 DEBUG "instance": "10.142.0.77:9100" 06:37:20 DEBUG "1359.6333333333332" 06:37:20 DEBUG ] 06:37:20 DEBUG 06:37:20 DEBUG } 06:37:20 DEBUG ] 06:37:20 DEBUG 06:37:20 DEBUG [...] 06:37:20 DEBUG } 06:37:20 DEBUG 1712813840, 06:37:20 DEBUG }, 06:37:20 DEBUG ] 06:37:20 DEBUG } 06:37:20 DEBUG "instance": "10.142.0.77:9100" 06:37:20 DEBUG ] 06:37:20 DEBUG } 06:37:20 DEBUG "0" 06:37:20 DEBUG 06:37:20 DEBUG } 06:37:20 DEBUG "1358.9" 06:37:20 DEBUG "value": [ 06:37:20 DEBUG } 06:37:20 DEBUG ] 06:37:20 DEBUG }, 06:37:20 DEBUG "value": [ 06:37:20 DEBUG ] 06:37:20 DEBUG ] 06:37:20 DEBUG 06:37:20 DEBUG ] 06:37:20 DEBUG 1712813840, 06:37:20 DEBUG ] 06:37:20 DEBUG } 06:37:20 DEBUG } 06:37:20 DEBUG 1712813840, 06:37:20 DEBUG } 06:37:20 DEBUG } 06:37:20 DEBUG 06:37:20 DEBUG } 06:37:20 DEBUG "0" 06:37:20 DEBUG } 06:37:20 DEBUG } 06:37:20 DEBUG } 06:37:20 DEBUG "82414135.33333333" 06:37:20 DEBUG } 06:37:20 DEBUG ] 06:37:20 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:37:20 DEBUG ] 06:37:20 DEBUG ] 06:37:20 DEBUG } 06:37:20 DEBUG ] 06:37:20 DEBUG } 06:37:20 DEBUG } 06:37:20 DEBUG }, 06:37:20 DEBUG } 06:37:20 DEBUG } 06:37:20 DEBUG } 06:37:20 DEBUG "value": [ 06:37:20 DEBUG ] 06:37:20 DEBUG ] 06:37:20 DEBUG } 06:37:20 DEBUG 1712813840, 06:37:20 DEBUG } 06:37:20 DEBUG } 06:37:20 DEBUG "4.138219346712325" 06:37:20 DEBUG } 06:37:20 DEBUG } 06:37:20 DEBUG ] 06:37:20 DEBUG } 06:37:20 DEBUG ] 06:37:20 DEBUG } 06:37:20 DEBUG } 06:37:31 INFO 06:37:31 INFO [loop_until]: kubectl --namespace=xlou top pods 06:37:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:37:31 INFO [loop_until]: OK (rc = 0) 06:37:31 DEBUG --- stdout --- 06:37:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 8937m 8515Mi am-5b5bd8567b-8h2w4 9129m 8563Mi am-5b5bd8567b-qlkw6 7445m 8460Mi ds-cts-0 7952m 13395Mi ds-cts-1 8496m 13399Mi ds-cts-2 8416m 13392Mi ds-idrepo-0 11374m 5498Mi ds-idrepo-1 335m 5479Mi ds-idrepo-2 359m 5783Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 9m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 4m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4083m 1848Mi 06:37:31 DEBUG --- stderr --- 06:37:31 DEBUG 06:37:35 INFO 06:37:35 INFO [loop_until]: kubectl --namespace=xlou top node 06:37:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:37:35 INFO [loop_until]: OK (rc = 0) 06:37:35 DEBUG --- stdout --- 06:37:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 2729Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 60m 0% 2781Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7830m 26% 10122Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10773m 36% 2997Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9390m 31% 10094Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 150m 0% 3847Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9582m 32% 10188Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 423m 2% 6459Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11171m 70% 6187Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8287m 52% 13950Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 8199m 51% 13983Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 8653m 54% 13076Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 414m 2% 6156Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4250m 26% 5618Mi 9% 06:37:35 DEBUG --- stderr --- 06:37:35 DEBUG 06:38:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813900" 06:38:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813900" 06:38:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813900" 06:38:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813900" 06:38:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813900" 06:38:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813900" 06:38:20 INFO [http_cmd]: http status code OK 06:38:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712813900" 06:38:20 INFO [http_cmd]: http status code OK 06:38:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712813900" 06:38:20 DEBUG --- status code --- 06:38:20 DEBUG --- status code --- 06:38:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:20 DEBUG --- http response --- 06:38:20 DEBUG --- http response --- 06:38:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:38:20 INFO [http_cmd]: http status code OK 06:38:20 INFO [http_cmd]: http status code OK 06:38:20 INFO [http_cmd]: http status code OK 06:38:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:38:20 DEBUG ----- output ----- 06:38:20 DEBUG --- status code --- 06:38:20 DEBUG --- status code --- 06:38:20 DEBUG --- status code --- 06:38:20 DEBUG ----- output ----- 06:38:20 DEBUG { 06:38:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:20 DEBUG { 06:38:20 DEBUG "status": "success", 06:38:20 DEBUG --- http response --- 06:38:20 DEBUG --- http response --- 06:38:20 DEBUG --- http response --- 06:38:20 DEBUG "status": "success", 06:38:20 DEBUG "data": { 06:38:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:38:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:38:20 DEBUG "data": { 06:38:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:38:20 DEBUG "resultType": "vector", 06:38:20 DEBUG ----- output ----- 06:38:20 DEBUG ----- output ----- 06:38:20 DEBUG "resultType": "vector", 06:38:20 DEBUG ----- output ----- 06:38:20 DEBUG "result": [ 06:38:20 DEBUG { 06:38:20 DEBUG { 06:38:20 DEBUG "result": [ 06:38:20 DEBUG { 06:38:20 DEBUG { 06:38:20 DEBUG "status": "success", 06:38:20 DEBUG "status": "success", 06:38:20 DEBUG { 06:38:20 DEBUG "status": "success", 06:38:20 DEBUG "metric": { 06:38:20 DEBUG "data": { 06:38:20 DEBUG "data": { 06:38:20 DEBUG "metric": { 06:38:20 DEBUG "data": { 06:38:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:38:20 DEBUG "resultType": "vector", 06:38:20 DEBUG "resultType": "vector", 06:38:20 DEBUG "pod": "ds-idrepo-2" 06:38:20 DEBUG "resultType": "vector", 06:38:20 DEBUG }, 06:38:20 DEBUG "result": [ 06:38:20 DEBUG "result": [ 06:38:20 DEBUG }, 06:38:20 DEBUG "result": [ 06:38:20 DEBUG "value": [ 06:38:20 DEBUG { 06:38:20 DEBUG { 06:38:20 DEBUG "value": [ 06:38:20 DEBUG { 06:38:20 DEBUG 06:38:20 DEBUG "metric": { 06:38:20 DEBUG "metric": { 06:38:20 DEBUG 06:38:20 DEBUG "metric": { 06:38:20 DEBUG 06:38:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:38:20 DEBUG "pod": "ds-idrepo-2" 06:38:20 DEBUG 06:38:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:38:20 DEBUG 06:38:20 DEBUG }, 06:38:20 DEBUG }, 06:38:20 DEBUG 06:38:20 DEBUG }, 06:38:20 DEBUG [...] 06:38:20 DEBUG "value": [ 06:38:20 DEBUG "value": [ 06:38:20 DEBUG [...] 06:38:20 DEBUG "value": [ 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:38:20 DEBUG [...] 06:38:20 DEBUG [...] 06:38:20 DEBUG "pod": "ds-cts-2" 06:38:20 DEBUG [...] 06:38:20 DEBUG }, 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG }, 06:38:20 DEBUG 06:38:20 DEBUG "value": [ 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG "value": [ 06:38:20 DEBUG 06:38:20 DEBUG 1712813900, 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 1712813900, 06:38:20 DEBUG 06:38:20 DEBUG "244955.5" 06:38:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:38:20 DEBUG "pod": "ds-idrepo-0" 06:38:20 DEBUG "29.766666666666666" 06:38:20 INFO [http_cmd]: http status code OK 06:38:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:38:20 ERROR [http_cmd]: ERROR 06:38:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813900" 06:38:20 DEBUG ] 06:38:20 DEBUG }, 06:38:20 DEBUG }, 06:38:20 DEBUG ] 06:38:20 DEBUG --- status code --- 06:38:20 DEBUG }, 06:38:20 ERROR --- status code --- 06:38:20 DEBUG } 06:38:20 DEBUG "value": [ 06:38:20 DEBUG "value": [ 06:38:20 INFO [http_cmd]: http status code OK 06:38:20 DEBUG } 06:38:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:20 DEBUG "value": [ 06:38:20 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:38:20 DEBUG ] 06:38:20 DEBUG 1712813900, 06:38:20 DEBUG 1712813900, 06:38:20 DEBUG --- status code --- 06:38:20 DEBUG ] 06:38:20 DEBUG --- http response --- 06:38:20 DEBUG 1712813900, 06:38:20 ERROR --- http response --- 06:38:20 DEBUG } 06:38:20 DEBUG "1368.1666666666667" 06:38:20 DEBUG "0" 06:38:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:20 DEBUG } 06:38:20 DEBUG "1352.0333333333333" 06:38:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:38:20 DEBUG } 06:38:20 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:38:20 DEBUG ] 06:38:20 DEBUG ] 06:38:20 DEBUG --- http response --- 06:38:20 DEBUG } 06:38:20 DEBUG ] 06:38:20 DEBUG ----- output ----- 06:38:20 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:38:20 DEBUG } 06:38:20 DEBUG } 06:38:20 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:38:20 DEBUG } 06:38:20 DEBUG { 06:38:20 DEBUG ] 06:38:20 DEBUG ] 06:38:20 DEBUG ] 06:38:20 DEBUG "status": "success", 06:38:20 DEBUG } 06:38:20 DEBUG } 06:38:20 DEBUG } 06:38:20 DEBUG "data": { 06:38:20 DEBUG } 06:38:20 DEBUG } 06:38:20 DEBUG } 06:38:20 DEBUG "resultType": "vector", 06:38:20 DEBUG "result": [ 06:38:20 DEBUG { 06:38:20 DEBUG "metric": { 06:38:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:38:20 DEBUG }, 06:38:20 DEBUG "value": [ 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG [...] 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:38:20 DEBUG }, 06:38:20 DEBUG "value": [ 06:38:20 DEBUG 1712813900, 06:38:20 DEBUG "0.021" 06:38:20 DEBUG ] 06:38:20 DEBUG } 06:38:20 DEBUG ] 06:38:20 DEBUG } 06:38:20 DEBUG } 06:38:20 INFO [http_cmd]: http status code OK 06:38:20 DEBUG --- status code --- 06:38:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:20 DEBUG --- http response --- 06:38:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:38:20 DEBUG ----- output ----- 06:38:20 DEBUG { 06:38:20 DEBUG "status": "success", 06:38:20 DEBUG "data": { 06:38:20 DEBUG "resultType": "vector", 06:38:20 DEBUG "result": [ 06:38:20 DEBUG { 06:38:20 DEBUG "metric": { 06:38:20 DEBUG "instance": "10.142.0.107:9100" 06:38:20 DEBUG }, 06:38:20 DEBUG "value": [ 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG [...] 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG "instance": "10.142.0.77:9100" 06:38:20 DEBUG }, 06:38:20 DEBUG "value": [ 06:38:20 DEBUG 1712813900, 06:38:20 DEBUG "52944477184" 06:38:20 DEBUG ] 06:38:20 DEBUG } 06:38:20 DEBUG ] 06:38:20 DEBUG } 06:38:20 DEBUG } 06:38:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813900" 06:38:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813900" 06:38:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813900" 06:38:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813900" 06:38:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813900" 06:38:20 INFO [http_cmd]: http status code OK 06:38:20 DEBUG --- status code --- 06:38:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:20 DEBUG --- http response --- 06:38:20 INFO [http_cmd]: http status code OK 06:38:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813900" 06:38:20 DEBUG --- status code --- 06:38:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:38:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:20 INFO [http_cmd]: http status code OK 06:38:20 DEBUG ----- output ----- 06:38:20 DEBUG --- http response --- 06:38:20 DEBUG --- status code --- 06:38:20 DEBUG { 06:38:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:38:20 DEBUG "status": "success", 06:38:20 DEBUG --- http response --- 06:38:20 DEBUG ----- output ----- 06:38:20 DEBUG "data": { 06:38:20 DEBUG { 06:38:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:38:20 DEBUG "resultType": "vector", 06:38:20 DEBUG "status": "success", 06:38:20 DEBUG ----- output ----- 06:38:20 DEBUG "result": [ 06:38:20 DEBUG "data": { 06:38:20 DEBUG { 06:38:20 INFO [http_cmd]: http status code OK 06:38:20 DEBUG { 06:38:20 DEBUG "resultType": "vector", 06:38:20 DEBUG "status": "success", 06:38:20 DEBUG --- status code --- 06:38:20 DEBUG "metric": { 06:38:20 DEBUG "result": [ 06:38:20 DEBUG "data": { 06:38:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:20 DEBUG "pod": "ds-idrepo-2" 06:38:20 DEBUG { 06:38:20 DEBUG "resultType": "vector", 06:38:20 DEBUG --- http response --- 06:38:20 DEBUG }, 06:38:20 DEBUG "metric": { 06:38:20 DEBUG "result": [ 06:38:20 DEBUG "value": [ 06:38:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:38:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:38:20 INFO [http_cmd]: http status code OK 06:38:20 DEBUG { 06:38:20 DEBUG 06:38:20 DEBUG ----- output ----- 06:38:20 DEBUG }, 06:38:20 DEBUG --- status code --- 06:38:20 DEBUG "metric": { 06:38:20 DEBUG 06:38:20 DEBUG { 06:38:20 DEBUG "value": [ 06:38:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:20 DEBUG "instance": "10.142.0.107:9100" 06:38:20 DEBUG 06:38:20 DEBUG "status": "success", 06:38:20 DEBUG 06:38:20 DEBUG --- http response --- 06:38:20 DEBUG }, 06:38:20 DEBUG [...] 06:38:20 DEBUG "data": { 06:38:20 DEBUG 06:38:20 DEBUG "value": [ 06:38:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:38:20 DEBUG 06:38:20 DEBUG "resultType": "vector", 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG ----- output ----- 06:38:20 DEBUG 06:38:20 DEBUG "result": [ 06:38:20 DEBUG [...] 06:38:20 DEBUG 06:38:20 DEBUG { 06:38:20 DEBUG 06:38:20 DEBUG { 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG "status": "success", 06:38:20 DEBUG "pod": "ds-cts-2" 06:38:20 DEBUG "metric": { 06:38:20 DEBUG 06:38:20 DEBUG [...] 06:38:20 DEBUG "data": { 06:38:20 DEBUG }, 06:38:20 DEBUG "instance": "10.142.0.107:9100" 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG "resultType": "vector", 06:38:20 DEBUG "value": [ 06:38:20 DEBUG }, 06:38:20 DEBUG "pod": "idm-668d898cfd-st2t7" 06:38:20 DEBUG 06:38:20 DEBUG "result": [ 06:38:20 DEBUG 1712813900, 06:38:20 DEBUG "value": [ 06:38:20 DEBUG }, 06:38:20 DEBUG 06:38:20 DEBUG { 06:38:20 DEBUG "0.00009252111111111113" 06:38:20 DEBUG 06:38:20 DEBUG "value": [ 06:38:20 DEBUG "instance": "10.142.0.77:9100" 06:38:20 DEBUG "metric": { 06:38:20 DEBUG ] 06:38:20 DEBUG 06:38:20 DEBUG 1712813900, 06:38:20 DEBUG }, 06:38:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:38:20 DEBUG } 06:38:20 DEBUG 06:38:20 DEBUG "0" 06:38:20 DEBUG "value": [ 06:38:20 DEBUG }, 06:38:20 DEBUG ] 06:38:20 DEBUG [...] 06:38:20 DEBUG ] 06:38:20 DEBUG 1712813900, 06:38:20 DEBUG "value": [ 06:38:20 DEBUG } 06:38:20 DEBUG 06:38:20 INFO [http_cmd]: http status code OK 06:38:20 DEBUG } 06:38:20 DEBUG "0.002166666666666828" 06:38:20 DEBUG 06:38:20 DEBUG } 06:38:20 DEBUG 06:38:20 DEBUG --- status code --- 06:38:20 DEBUG ] 06:38:20 DEBUG ] 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:20 DEBUG } 06:38:20 DEBUG } 06:38:20 DEBUG 06:38:20 DEBUG "instance": "10.142.0.77:9100" 06:38:20 DEBUG --- http response --- 06:38:20 DEBUG } 06:38:20 DEBUG ] 06:38:20 DEBUG [...] 06:38:20 DEBUG }, 06:38:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:38:20 DEBUG } 06:38:20 DEBUG 06:38:20 DEBUG "value": [ 06:38:20 DEBUG ----- output ----- 06:38:20 DEBUG } 06:38:20 DEBUG 06:38:20 DEBUG 1712813900, 06:38:20 DEBUG { 06:38:20 DEBUG 06:38:20 DEBUG "82860049.93333334" 06:38:20 DEBUG "status": "success", 06:38:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:38:20 DEBUG ] 06:38:20 DEBUG "data": { 06:38:20 DEBUG }, 06:38:20 DEBUG } 06:38:20 DEBUG "resultType": "vector", 06:38:20 DEBUG "value": [ 06:38:20 DEBUG ] 06:38:20 DEBUG "result": [ 06:38:20 DEBUG 1712813900, 06:38:20 DEBUG } 06:38:20 DEBUG { 06:38:20 DEBUG "0" 06:38:20 DEBUG } 06:38:20 DEBUG "metric": { 06:38:20 DEBUG ] 06:38:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:38:20 DEBUG } 06:38:20 DEBUG }, 06:38:20 DEBUG ] 06:38:20 DEBUG "value": [ 06:38:20 DEBUG } 06:38:20 DEBUG 06:38:20 DEBUG } 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG [...] 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:38:20 DEBUG }, 06:38:20 DEBUG "value": [ 06:38:20 DEBUG 1712813900, 06:38:20 DEBUG "0" 06:38:20 DEBUG ] 06:38:20 DEBUG } 06:38:20 DEBUG ] 06:38:20 DEBUG } 06:38:20 DEBUG } 06:38:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813900" 06:38:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813900" 06:38:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712813900" 06:38:20 INFO [http_cmd]: http status code OK 06:38:20 DEBUG --- status code --- 06:38:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:20 DEBUG --- http response --- 06:38:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:38:20 DEBUG ----- output ----- 06:38:20 DEBUG { 06:38:20 DEBUG "status": "success", 06:38:20 DEBUG "data": { 06:38:20 DEBUG "resultType": "vector", 06:38:20 DEBUG "result": [ 06:38:20 DEBUG { 06:38:20 DEBUG "metric": { 06:38:20 DEBUG "pod": "ds-idrepo-2" 06:38:20 DEBUG }, 06:38:20 DEBUG "value": [ 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG [...] 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG "pod": "ds-idrepo-0" 06:38:20 DEBUG }, 06:38:20 DEBUG "value": [ 06:38:20 DEBUG 1712813900, 06:38:20 DEBUG "0" 06:38:20 DEBUG ] 06:38:20 DEBUG } 06:38:20 DEBUG ] 06:38:20 DEBUG } 06:38:20 DEBUG } 06:38:20 INFO [http_cmd]: http status code OK 06:38:20 DEBUG --- status code --- 06:38:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:20 DEBUG --- http response --- 06:38:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:38:20 DEBUG ----- output ----- 06:38:20 DEBUG { 06:38:20 DEBUG "status": "success", 06:38:20 DEBUG "data": { 06:38:20 DEBUG "resultType": "vector", 06:38:20 DEBUG "result": [ 06:38:20 DEBUG { 06:38:20 DEBUG "metric": { 06:38:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:38:20 DEBUG }, 06:38:20 DEBUG "value": [ 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG [...] 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG "pod": "idm-668d898cfd-st2t7" 06:38:20 DEBUG }, 06:38:20 DEBUG "value": [ 06:38:20 DEBUG 1712813900, 06:38:20 DEBUG "0" 06:38:20 DEBUG ] 06:38:20 DEBUG } 06:38:20 DEBUG ] 06:38:20 DEBUG } 06:38:20 DEBUG } 06:38:20 INFO [http_cmd]: http status code OK 06:38:20 DEBUG --- status code --- 06:38:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:20 DEBUG --- http response --- 06:38:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:38:20 DEBUG ----- output ----- 06:38:20 DEBUG { 06:38:20 DEBUG "status": "success", 06:38:20 DEBUG "data": { 06:38:20 DEBUG "resultType": "vector", 06:38:20 DEBUG "result": [ 06:38:20 DEBUG { 06:38:20 DEBUG "metric": { 06:38:20 DEBUG "pod": "ds-idrepo-2" 06:38:20 DEBUG }, 06:38:20 DEBUG "value": [ 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG [...] 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG "pod": "ds-cts-2" 06:38:20 DEBUG }, 06:38:20 DEBUG "value": [ 06:38:20 DEBUG 1712813900, 06:38:20 DEBUG "0" 06:38:20 DEBUG ] 06:38:20 DEBUG } 06:38:20 DEBUG ] 06:38:20 DEBUG } 06:38:20 DEBUG } 06:38:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813900" 06:38:20 INFO [http_cmd]: http status code OK 06:38:20 DEBUG --- status code --- 06:38:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:20 DEBUG --- http response --- 06:38:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:38:20 DEBUG ----- output ----- 06:38:20 DEBUG { 06:38:20 DEBUG "status": "success", 06:38:20 DEBUG "data": { 06:38:20 DEBUG "resultType": "vector", 06:38:20 DEBUG "result": [ 06:38:20 DEBUG { 06:38:20 DEBUG "metric": { 06:38:20 DEBUG "instance": "10.142.0.107:9100" 06:38:20 DEBUG }, 06:38:20 DEBUG "value": [ 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG [...] 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG "instance": "10.142.0.77:9100" 06:38:20 DEBUG }, 06:38:20 DEBUG "value": [ 06:38:20 DEBUG 1712813900, 06:38:20 DEBUG "82857555.86666667" 06:38:20 DEBUG ] 06:38:20 DEBUG } 06:38:20 DEBUG ] 06:38:20 DEBUG } 06:38:20 DEBUG } 06:38:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813900" 06:38:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712813900" 06:38:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813900" 06:38:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813900" 06:38:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813900" 06:38:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813900" 06:38:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712813900" 06:38:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813900" 06:38:20 INFO [http_cmd]: http status code OK 06:38:20 DEBUG --- status code --- 06:38:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:20 DEBUG --- http response --- 06:38:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:38:20 DEBUG ----- output ----- 06:38:20 DEBUG { 06:38:20 DEBUG "status": "success", 06:38:20 DEBUG "data": { 06:38:20 DEBUG "resultType": "vector", 06:38:20 DEBUG "result": [ 06:38:20 DEBUG { 06:38:20 DEBUG "metric": { 06:38:20 DEBUG "pod": "ds-idrepo-2" 06:38:20 DEBUG }, 06:38:20 DEBUG "value": [ 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG [...] 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG "pod": "ds-idrepo-0" 06:38:20 DEBUG }, 06:38:20 DEBUG "value": [ 06:38:20 DEBUG 1712813900, 06:38:20 DEBUG "0" 06:38:20 DEBUG ] 06:38:20 DEBUG } 06:38:20 DEBUG ] 06:38:20 DEBUG } 06:38:20 DEBUG } 06:38:20 INFO [http_cmd]: http status code OK 06:38:20 INFO [http_cmd]: http status code OK 06:38:20 DEBUG --- status code --- 06:38:20 DEBUG --- status code --- 06:38:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:20 DEBUG --- http response --- 06:38:20 INFO [http_cmd]: http status code OK 06:38:20 DEBUG --- http response --- 06:38:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:38:20 INFO [http_cmd]: http status code OK 06:38:20 DEBUG --- status code --- 06:38:20 DEBUG ----- output ----- 06:38:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:38:20 INFO [http_cmd]: http status code OK 06:38:20 INFO [http_cmd]: http status code OK 06:38:20 DEBUG --- status code --- 06:38:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:20 DEBUG { 06:38:20 DEBUG ----- output ----- 06:38:20 DEBUG --- status code --- 06:38:20 DEBUG --- status code --- 06:38:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:20 DEBUG --- http response --- 06:38:20 DEBUG "status": "success", 06:38:20 INFO [http_cmd]: http status code OK 06:38:20 DEBUG { 06:38:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:20 DEBUG --- http response --- 06:38:20 DEBUG "data": { 06:38:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:38:20 DEBUG --- status code --- 06:38:20 DEBUG "status": "success", 06:38:20 DEBUG --- http response --- 06:38:20 DEBUG --- http response --- 06:38:20 DEBUG "resultType": "vector", 06:38:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:38:20 DEBUG ----- output ----- 06:38:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:38:20 DEBUG "data": { 06:38:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:38:20 DEBUG "result": [ 06:38:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:38:20 DEBUG ----- output ----- 06:38:20 DEBUG { 06:38:20 DEBUG --- http response --- 06:38:20 DEBUG "resultType": "vector", 06:38:20 DEBUG ----- output ----- 06:38:20 DEBUG { 06:38:20 DEBUG ----- output ----- 06:38:20 DEBUG { 06:38:20 DEBUG "status": "success", 06:38:20 DEBUG "result": [ 06:38:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:38:20 DEBUG { 06:38:20 DEBUG "metric": { 06:38:20 DEBUG { 06:38:20 DEBUG "status": "success", 06:38:20 DEBUG "data": { 06:38:20 DEBUG { 06:38:20 DEBUG ----- output ----- 06:38:20 DEBUG "status": "success", 06:38:20 DEBUG "pod": "ds-idrepo-2" 06:38:20 DEBUG "status": "success", 06:38:20 DEBUG "data": { 06:38:20 DEBUG "resultType": "vector", 06:38:20 DEBUG "metric": { 06:38:20 DEBUG { 06:38:20 DEBUG "data": { 06:38:20 DEBUG }, 06:38:20 DEBUG "data": { 06:38:20 DEBUG "resultType": "vector", 06:38:20 DEBUG "result": [ 06:38:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:38:20 DEBUG "status": "success", 06:38:20 DEBUG "resultType": "vector", 06:38:20 DEBUG "value": [ 06:38:20 DEBUG "resultType": "vector", 06:38:20 DEBUG "result": [ 06:38:20 DEBUG { 06:38:20 DEBUG }, 06:38:20 DEBUG "data": { 06:38:20 DEBUG "result": [ 06:38:20 DEBUG 06:38:20 DEBUG "result": [ 06:38:20 DEBUG { 06:38:20 DEBUG "metric": { 06:38:20 DEBUG "value": [ 06:38:20 DEBUG "resultType": "vector", 06:38:20 DEBUG { 06:38:20 DEBUG 06:38:20 DEBUG { 06:38:20 DEBUG "metric": { 06:38:20 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:38:20 DEBUG 06:38:20 DEBUG "result": [ 06:38:20 DEBUG "metric": { 06:38:20 DEBUG 06:38:20 DEBUG "metric": { 06:38:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:38:20 DEBUG }, 06:38:20 DEBUG 06:38:20 DEBUG { 06:38:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:38:20 DEBUG [...] 06:38:20 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:38:20 DEBUG }, 06:38:20 DEBUG "value": [ 06:38:20 DEBUG 06:38:20 DEBUG "metric": { 06:38:20 DEBUG }, 06:38:20 DEBUG 06:38:20 DEBUG }, 06:38:20 DEBUG "value": [ 06:38:20 DEBUG 06:38:20 DEBUG [...] 06:38:20 DEBUG "instance": "10.142.0.107:9100" 06:38:20 DEBUG "value": [ 06:38:20 DEBUG 06:38:20 DEBUG "value": [ 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG }, 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG "value": [ 06:38:20 DEBUG 06:38:20 DEBUG "pod": "ds-cts-2" 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG [...] 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG }, 06:38:20 DEBUG 06:38:20 DEBUG [...] 06:38:20 DEBUG 06:38:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:38:20 DEBUG 06:38:20 DEBUG [...] 06:38:20 DEBUG "value": [ 06:38:20 DEBUG [...] 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG }, 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 1712813900, 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG "value": [ 06:38:20 DEBUG [...] 06:38:20 DEBUG 06:38:20 DEBUG "0" 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 06:38:20 DEBUG 1712813900, 06:38:20 DEBUG 06:38:20 DEBUG 06:38:20 DEBUG ] 06:38:20 DEBUG 06:38:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:38:20 DEBUG }, 06:38:20 DEBUG "1365.4333333333334" 06:38:20 DEBUG 06:38:20 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:38:20 DEBUG } 06:38:20 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:38:20 DEBUG }, 06:38:20 DEBUG "value": [ 06:38:20 DEBUG ] 06:38:20 DEBUG 06:38:20 DEBUG }, 06:38:20 DEBUG ] 06:38:20 DEBUG }, 06:38:20 DEBUG "value": [ 06:38:20 DEBUG 1712813900, 06:38:20 DEBUG } 06:38:20 DEBUG "instance": "10.142.0.77:9100" 06:38:20 DEBUG "value": [ 06:38:20 DEBUG } 06:38:20 DEBUG "value": [ 06:38:20 DEBUG 1712813900, 06:38:20 DEBUG "14133534720" 06:38:20 DEBUG ] 06:38:20 DEBUG }, 06:38:20 DEBUG 1712813900, 06:38:20 DEBUG } 06:38:20 DEBUG 1712813900, 06:38:20 DEBUG "1368.7333333333333" 06:38:20 DEBUG ] 06:38:20 DEBUG } 06:38:20 DEBUG "value": [ 06:38:20 DEBUG "0" 06:38:20 DEBUG "4.125111569372903" 06:38:20 DEBUG ] 06:38:20 DEBUG } 06:38:20 DEBUG } 06:38:20 DEBUG 1712813900, 06:38:20 DEBUG ] 06:38:20 DEBUG ] 06:38:20 DEBUG } 06:38:20 DEBUG ] 06:38:20 DEBUG "0.0013333333333333049" 06:38:20 DEBUG } 06:38:20 DEBUG } 06:38:20 DEBUG ] 06:38:20 DEBUG } 06:38:20 DEBUG ] 06:38:20 DEBUG ] 06:38:20 DEBUG ] 06:38:20 DEBUG } 06:38:20 DEBUG } 06:38:20 DEBUG } 06:38:20 DEBUG } 06:38:20 DEBUG } 06:38:20 DEBUG } 06:38:20 DEBUG ] 06:38:20 DEBUG } 06:38:20 DEBUG } 06:38:20 DEBUG } 06:38:20 DEBUG } 06:38:31 INFO 06:38:31 INFO [loop_until]: kubectl --namespace=xlou top pods 06:38:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:38:32 INFO [loop_until]: OK (rc = 0) 06:38:32 DEBUG --- stdout --- 06:38:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 9091m 8501Mi am-5b5bd8567b-8h2w4 8778m 8564Mi am-5b5bd8567b-qlkw6 7475m 8459Mi ds-cts-0 8103m 12563Mi ds-cts-1 8814m 13260Mi ds-cts-2 8817m 12581Mi ds-idrepo-0 11493m 5498Mi ds-idrepo-1 397m 5479Mi ds-idrepo-2 368m 5783Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 3m 1010Mi idm-668d898cfd-st2t7 5m 1135Mi lodemon-ccc964465-p7gdm 2m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4077m 1848Mi 06:38:32 DEBUG --- stderr --- 06:38:32 DEBUG 06:38:35 INFO 06:38:35 INFO [loop_until]: kubectl --namespace=xlou top node 06:38:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:38:35 INFO [loop_until]: OK (rc = 0) 06:38:35 DEBUG --- stdout --- 06:38:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 2730Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 63m 0% 2783Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7551m 25% 10113Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10751m 36% 2983Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9596m 32% 10083Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 142m 0% 3853Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9630m 32% 10189Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 425m 2% 6460Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11608m 73% 6187Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8786m 55% 13274Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 8186m 51% 13092Mi 22% gke-xlou-cdm-ds-32e4dcb1-j2hk 8662m 54% 13753Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 453m 2% 6157Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4239m 26% 5617Mi 9% 06:38:35 DEBUG --- stderr --- 06:38:35 DEBUG 06:39:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712813960" 06:39:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813960" 06:39:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813960" 06:39:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813960" 06:39:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813960" 06:39:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813960" 06:39:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813960" 06:39:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813960" 06:39:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813960" 06:39:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712813960" 06:39:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813960" 06:39:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813960" 06:39:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813960" 06:39:20 INFO [http_cmd]: http status code OK 06:39:20 ERROR [http_cmd]: ERROR 06:39:20 DEBUG --- status code --- 06:39:20 ERROR --- status code --- 06:39:20 INFO [http_cmd]: http status code OK 06:39:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:20 INFO [http_cmd]: http status code OK 06:39:20 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:39:20 INFO [http_cmd]: http status code OK 06:39:20 INFO [http_cmd]: http status code OK 06:39:20 INFO [http_cmd]: http status code OK 06:39:20 DEBUG --- status code --- 06:39:20 INFO [http_cmd]: http status code OK 06:39:20 DEBUG --- http response --- 06:39:20 DEBUG --- status code --- 06:39:20 ERROR --- http response --- 06:39:20 DEBUG --- status code --- 06:39:20 DEBUG --- status code --- 06:39:20 DEBUG --- status code --- 06:39:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:20 DEBUG --- status code --- 06:39:20 INFO [http_cmd]: http status code OK 06:39:20 INFO [http_cmd]: http status code OK 06:39:20 INFO [http_cmd]: http status code OK 06:39:20 INFO [http_cmd]: http status code OK 06:39:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:39:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:20 DEBUG --- http response --- 06:39:20 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:39:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:20 DEBUG --- http response --- 06:39:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:20 DEBUG --- status code --- 06:39:20 DEBUG --- status code --- 06:39:20 DEBUG --- status code --- 06:39:20 DEBUG --- status code --- 06:39:20 DEBUG --- http response --- 06:39:20 DEBUG ----- output ----- 06:39:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:20 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:39:20 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:39:20 DEBUG --- http response --- 06:39:20 DEBUG --- http response --- 06:39:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:39:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:20 DEBUG { 06:39:20 DEBUG "status": "success", 06:39:20 DEBUG --- http response --- 06:39:20 INFO [http_cmd]: http status code OK 06:39:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:39:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:39:20 DEBUG ----- output ----- 06:39:20 DEBUG --- http response --- 06:39:20 DEBUG --- http response --- 06:39:20 DEBUG --- http response --- 06:39:20 DEBUG --- http response --- 06:39:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:39:20 DEBUG "data": { 06:39:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:39:20 DEBUG --- status code --- 06:39:20 DEBUG ----- output ----- 06:39:20 DEBUG ----- output ----- 06:39:20 DEBUG { 06:39:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:39:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:39:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:39:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:39:20 DEBUG ----- output ----- 06:39:20 DEBUG "resultType": "vector", 06:39:20 DEBUG ----- output ----- 06:39:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:20 DEBUG { 06:39:20 DEBUG { 06:39:20 DEBUG "status": "success", 06:39:20 DEBUG ----- output ----- 06:39:20 DEBUG ----- output ----- 06:39:20 DEBUG ----- output ----- 06:39:20 DEBUG ----- output ----- 06:39:20 DEBUG { 06:39:20 DEBUG "result": [ 06:39:20 DEBUG { 06:39:20 DEBUG --- http response --- 06:39:20 DEBUG "status": "success", 06:39:20 DEBUG "status": "success", 06:39:20 DEBUG "data": { 06:39:20 DEBUG { 06:39:20 DEBUG { 06:39:20 DEBUG { 06:39:20 DEBUG { 06:39:20 DEBUG "status": "success", 06:39:20 DEBUG { 06:39:20 DEBUG "status": "success", 06:39:20 DEBUG "data": { 06:39:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:39:20 DEBUG "data": { 06:39:20 DEBUG "resultType": "vector", 06:39:20 DEBUG "status": "success", 06:39:20 DEBUG "status": "success", 06:39:20 DEBUG "status": "success", 06:39:20 DEBUG "status": "success", 06:39:20 DEBUG "data": { 06:39:20 DEBUG "metric": { 06:39:20 DEBUG "data": { 06:39:20 DEBUG "resultType": "vector", 06:39:20 DEBUG ----- output ----- 06:39:20 DEBUG "resultType": "vector", 06:39:20 DEBUG "result": [ 06:39:20 DEBUG "data": { 06:39:20 DEBUG "data": { 06:39:20 DEBUG "data": { 06:39:20 DEBUG "data": { 06:39:20 DEBUG "resultType": "vector", 06:39:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:39:20 DEBUG "resultType": "vector", 06:39:20 DEBUG "result": [ 06:39:20 DEBUG { 06:39:20 DEBUG "result": [ 06:39:20 DEBUG { 06:39:20 DEBUG "resultType": "vector", 06:39:20 DEBUG "resultType": "vector", 06:39:20 DEBUG "resultType": "vector", 06:39:20 DEBUG "resultType": "vector", 06:39:20 DEBUG "result": [ 06:39:20 DEBUG }, 06:39:20 DEBUG "result": [ 06:39:20 DEBUG { 06:39:20 DEBUG "status": "success", 06:39:20 DEBUG { 06:39:20 DEBUG "metric": { 06:39:20 DEBUG "result": [ 06:39:20 DEBUG "result": [ 06:39:20 DEBUG "result": [ 06:39:20 DEBUG "result": [ 06:39:20 DEBUG { 06:39:20 DEBUG "value": [ 06:39:20 DEBUG { 06:39:20 DEBUG "metric": { 06:39:20 DEBUG "data": { 06:39:20 DEBUG "metric": { 06:39:20 DEBUG "pod": "ds-idrepo-2" 06:39:20 DEBUG { 06:39:20 DEBUG { 06:39:20 DEBUG { 06:39:20 DEBUG { 06:39:20 DEBUG "metric": { 06:39:20 DEBUG 06:39:20 DEBUG "metric": { 06:39:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:39:20 DEBUG "resultType": "vector", 06:39:20 DEBUG "pod": "ds-idrepo-2" 06:39:20 DEBUG }, 06:39:20 DEBUG "metric": { 06:39:20 DEBUG "metric": { 06:39:20 DEBUG "metric": { 06:39:20 DEBUG "metric": { 06:39:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:39:20 DEBUG 06:39:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:39:20 DEBUG }, 06:39:20 DEBUG "result": [ 06:39:20 DEBUG }, 06:39:20 DEBUG "value": [ 06:39:20 DEBUG "pod": "ds-idrepo-2" 06:39:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:39:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:39:20 DEBUG "instance": "10.142.0.107:9100" 06:39:20 DEBUG }, 06:39:20 DEBUG 06:39:20 DEBUG }, 06:39:20 DEBUG "value": [ 06:39:20 DEBUG { 06:39:20 DEBUG "value": [ 06:39:20 DEBUG 06:39:20 DEBUG }, 06:39:20 DEBUG }, 06:39:20 DEBUG }, 06:39:20 DEBUG }, 06:39:20 DEBUG "value": [ 06:39:20 DEBUG [...] 06:39:20 DEBUG "value": [ 06:39:20 DEBUG 06:39:20 DEBUG "metric": { 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG "value": [ 06:39:20 DEBUG "value": [ 06:39:20 DEBUG "value": [ 06:39:20 DEBUG "value": [ 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG }, 06:39:20 DEBUG 06:39:20 DEBUG [...] 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG [...] 06:39:20 DEBUG "value": [ 06:39:20 DEBUG [...] 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG [...] 06:39:20 DEBUG "pod": "idm-668d898cfd-st2t7" 06:39:20 DEBUG [...] 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG [...] 06:39:20 DEBUG [...] 06:39:20 DEBUG [...] 06:39:20 DEBUG [...] 06:39:20 DEBUG 06:39:20 DEBUG }, 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG "value": [ 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG "pod": "ds-cts-2" 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 1712813960, 06:39:20 DEBUG 06:39:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:39:20 DEBUG [...] 06:39:20 DEBUG "pod": "ds-cts-2" 06:39:20 DEBUG }, 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813960" 06:39:20 DEBUG 06:39:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:39:20 DEBUG "0" 06:39:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:39:20 DEBUG }, 06:39:20 DEBUG 06:39:20 DEBUG }, 06:39:20 DEBUG "value": [ 06:39:20 DEBUG "pod": "ds-idrepo-0" 06:39:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:39:20 DEBUG "pod": "am-5b5bd8567b-qlkw6" 06:39:20 DEBUG "instance": "10.142.0.77:9100" 06:39:20 DEBUG }, 06:39:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813960" 06:39:20 DEBUG ] 06:39:20 DEBUG }, 06:39:20 DEBUG "value": [ 06:39:20 DEBUG 06:39:20 DEBUG "value": [ 06:39:20 DEBUG 1712813960, 06:39:20 DEBUG }, 06:39:20 DEBUG }, 06:39:20 DEBUG }, 06:39:20 DEBUG }, 06:39:20 DEBUG "value": [ 06:39:20 DEBUG } 06:39:20 DEBUG "value": [ 06:39:20 DEBUG 1712813960, 06:39:20 DEBUG 06:39:20 DEBUG 1712813960, 06:39:20 DEBUG "29.166666666666668" 06:39:20 DEBUG "value": [ 06:39:20 DEBUG "value": [ 06:39:20 DEBUG "value": [ 06:39:20 DEBUG "value": [ 06:39:20 DEBUG 1712813960, 06:39:20 DEBUG ] 06:39:20 DEBUG 1712813960, 06:39:20 DEBUG "1373.6333333333332" 06:39:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:39:20 DEBUG "0.00003333333333333336" 06:39:20 DEBUG ] 06:39:20 DEBUG 1712813960, 06:39:20 DEBUG 1712813960, 06:39:20 DEBUG 1712813960, 06:39:20 DEBUG 1712813960, 06:39:20 DEBUG "52945714176" 06:39:20 DEBUG } 06:39:20 DEBUG "245241" 06:39:20 DEBUG ] 06:39:20 DEBUG }, 06:39:20 DEBUG ] 06:39:20 DEBUG } 06:39:20 DEBUG "0" 06:39:20 DEBUG "0.021" 06:39:20 DEBUG "4351616.37084152" 06:39:20 DEBUG "1359.7333333333333" 06:39:20 DEBUG ] 06:39:20 DEBUG } 06:39:20 DEBUG ] 06:39:20 DEBUG } 06:39:20 DEBUG "value": [ 06:39:20 DEBUG } 06:39:20 DEBUG ] 06:39:20 DEBUG ] 06:39:20 DEBUG ] 06:39:20 DEBUG ] 06:39:20 DEBUG ] 06:39:20 DEBUG } 06:39:20 DEBUG } 06:39:20 DEBUG ] 06:39:20 DEBUG 1712813960, 06:39:20 DEBUG ] 06:39:20 DEBUG } 06:39:20 DEBUG } 06:39:20 DEBUG } 06:39:20 DEBUG } 06:39:20 DEBUG } 06:39:20 DEBUG ] 06:39:20 DEBUG ] 06:39:20 DEBUG } 06:39:20 DEBUG "0" 06:39:20 DEBUG } 06:39:20 DEBUG } 06:39:20 DEBUG ] 06:39:20 DEBUG ] 06:39:20 DEBUG ] 06:39:20 DEBUG ] 06:39:20 DEBUG } 06:39:20 DEBUG } 06:39:20 DEBUG } 06:39:20 DEBUG ] 06:39:20 DEBUG } 06:39:20 DEBUG } 06:39:20 DEBUG } 06:39:20 DEBUG } 06:39:20 DEBUG } 06:39:20 DEBUG } 06:39:20 DEBUG } 06:39:20 DEBUG } 06:39:20 DEBUG } 06:39:20 DEBUG } 06:39:20 DEBUG } 06:39:20 DEBUG } 06:39:20 INFO [http_cmd]: http status code OK 06:39:20 INFO [http_cmd]: http status code OK 06:39:20 DEBUG ] 06:39:20 DEBUG --- status code --- 06:39:20 DEBUG --- status code --- 06:39:20 DEBUG } 06:39:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:20 DEBUG } 06:39:20 DEBUG --- http response --- 06:39:20 DEBUG --- http response --- 06:39:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:39:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:39:20 DEBUG ----- output ----- 06:39:20 DEBUG ----- output ----- 06:39:20 DEBUG { 06:39:20 DEBUG { 06:39:20 DEBUG "status": "success", 06:39:20 DEBUG "status": "success", 06:39:20 DEBUG "data": { 06:39:20 DEBUG "data": { 06:39:20 DEBUG "resultType": "vector", 06:39:20 DEBUG "resultType": "vector", 06:39:20 DEBUG "result": [ 06:39:20 DEBUG "result": [ 06:39:20 DEBUG { 06:39:20 DEBUG { 06:39:20 DEBUG "metric": { 06:39:20 DEBUG "metric": { 06:39:20 DEBUG "instance": "10.142.0.107:9100" 06:39:20 DEBUG "instance": "10.142.0.107:9100" 06:39:20 DEBUG }, 06:39:20 DEBUG }, 06:39:20 DEBUG "value": [ 06:39:20 DEBUG "value": [ 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG [...] 06:39:20 DEBUG [...] 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG "instance": "10.142.0.77:9100" 06:39:20 DEBUG "instance": "10.142.0.77:9100" 06:39:20 DEBUG }, 06:39:20 DEBUG }, 06:39:20 DEBUG "value": [ 06:39:20 DEBUG "value": [ 06:39:20 DEBUG 1712813960, 06:39:20 DEBUG 1712813960, 06:39:20 DEBUG "0.0023333333333333426" 06:39:20 DEBUG "82764933.9" 06:39:20 DEBUG ] 06:39:20 DEBUG ] 06:39:20 DEBUG } 06:39:20 DEBUG } 06:39:20 DEBUG ] 06:39:20 DEBUG ] 06:39:20 DEBUG } 06:39:20 DEBUG } 06:39:20 DEBUG } 06:39:20 DEBUG } 06:39:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712813960" 06:39:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813960" 06:39:20 INFO [http_cmd]: http status code OK 06:39:20 DEBUG --- status code --- 06:39:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:20 DEBUG --- http response --- 06:39:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:39:20 DEBUG ----- output ----- 06:39:20 DEBUG { 06:39:20 DEBUG "status": "success", 06:39:20 DEBUG "data": { 06:39:20 DEBUG "resultType": "vector", 06:39:20 DEBUG "result": [ 06:39:20 DEBUG { 06:39:20 DEBUG "metric": { 06:39:20 DEBUG "pod": "ds-idrepo-2" 06:39:20 DEBUG }, 06:39:20 DEBUG "value": [ 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG [...] 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG "pod": "ds-cts-2" 06:39:20 DEBUG }, 06:39:20 DEBUG "value": [ 06:39:20 DEBUG 1712813960, 06:39:20 DEBUG "0" 06:39:20 DEBUG ] 06:39:20 DEBUG } 06:39:20 DEBUG ] 06:39:20 DEBUG } 06:39:20 DEBUG } 06:39:20 INFO [http_cmd]: http status code OK 06:39:20 DEBUG --- status code --- 06:39:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:20 DEBUG --- http response --- 06:39:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:39:20 DEBUG ----- output ----- 06:39:20 DEBUG { 06:39:20 DEBUG "status": "success", 06:39:20 DEBUG "data": { 06:39:20 DEBUG "resultType": "vector", 06:39:20 DEBUG "result": [ 06:39:20 DEBUG { 06:39:20 DEBUG "metric": { 06:39:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:39:20 DEBUG }, 06:39:20 DEBUG "value": [ 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG [...] 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG "pod": "idm-668d898cfd-st2t7" 06:39:20 DEBUG }, 06:39:20 DEBUG "value": [ 06:39:20 DEBUG 1712813960, 06:39:20 DEBUG "0" 06:39:20 DEBUG ] 06:39:20 DEBUG } 06:39:20 DEBUG ] 06:39:20 DEBUG } 06:39:20 DEBUG } 06:39:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813960" 06:39:20 INFO [http_cmd]: http status code OK 06:39:20 DEBUG --- status code --- 06:39:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:20 DEBUG --- http response --- 06:39:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:39:20 DEBUG ----- output ----- 06:39:20 DEBUG { 06:39:20 DEBUG "status": "success", 06:39:20 DEBUG "data": { 06:39:20 DEBUG "resultType": "vector", 06:39:20 DEBUG "result": [ 06:39:20 DEBUG { 06:39:20 DEBUG "metric": { 06:39:20 DEBUG "pod": "ds-idrepo-2" 06:39:20 DEBUG }, 06:39:20 DEBUG "value": [ 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG [...] 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG "pod": "ds-idrepo-0" 06:39:20 DEBUG }, 06:39:20 DEBUG "value": [ 06:39:20 DEBUG 1712813960, 06:39:20 DEBUG "0" 06:39:20 DEBUG ] 06:39:20 DEBUG } 06:39:20 DEBUG ] 06:39:20 DEBUG } 06:39:20 DEBUG } 06:39:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813960" 06:39:20 INFO [http_cmd]: http status code OK 06:39:20 DEBUG --- status code --- 06:39:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:20 DEBUG --- http response --- 06:39:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712813960" 06:39:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:39:20 DEBUG ----- output ----- 06:39:20 DEBUG { 06:39:20 DEBUG "status": "success", 06:39:20 DEBUG "data": { 06:39:20 DEBUG "resultType": "vector", 06:39:20 DEBUG "result": [ 06:39:20 DEBUG { 06:39:20 DEBUG "metric": { 06:39:20 DEBUG "instance": "10.142.0.107:9100" 06:39:20 DEBUG }, 06:39:20 DEBUG "value": [ 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG [...] 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG "instance": "10.142.0.77:9100" 06:39:20 DEBUG }, 06:39:20 DEBUG "value": [ 06:39:20 DEBUG 1712813960, 06:39:20 DEBUG "82762698.36666666" 06:39:20 DEBUG ] 06:39:20 DEBUG } 06:39:20 DEBUG ] 06:39:20 DEBUG } 06:39:20 DEBUG } 06:39:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813960" 06:39:20 INFO [http_cmd]: http status code OK 06:39:20 DEBUG --- status code --- 06:39:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:20 DEBUG --- http response --- 06:39:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:39:20 DEBUG ----- output ----- 06:39:20 DEBUG { 06:39:20 DEBUG "status": "success", 06:39:20 DEBUG "data": { 06:39:20 DEBUG "resultType": "vector", 06:39:20 DEBUG "result": [ 06:39:20 DEBUG { 06:39:20 DEBUG "metric": { 06:39:20 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:39:20 DEBUG }, 06:39:20 DEBUG "value": [ 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG [...] 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 06:39:20 DEBUG }, 06:39:20 DEBUG "value": [ 06:39:20 DEBUG 1712813960, 06:39:20 DEBUG "13585698816" 06:39:20 DEBUG ] 06:39:20 DEBUG } 06:39:20 DEBUG ] 06:39:20 DEBUG } 06:39:20 DEBUG } 06:39:20 INFO [http_cmd]: http status code OK 06:39:20 DEBUG --- status code --- 06:39:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:20 DEBUG --- http response --- 06:39:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:39:20 DEBUG ----- output ----- 06:39:20 DEBUG { 06:39:20 DEBUG "status": "success", 06:39:20 DEBUG "data": { 06:39:20 DEBUG "resultType": "vector", 06:39:20 DEBUG "result": [ 06:39:20 DEBUG { 06:39:20 DEBUG "metric": { 06:39:20 DEBUG "pod": "ds-idrepo-2" 06:39:20 DEBUG }, 06:39:20 DEBUG "value": [ 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG [...] 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG 06:39:20 DEBUG "pod": "ds-idrepo-0" 06:39:20 DEBUG }, 06:39:20 DEBUG "value": [ 06:39:20 DEBUG 1712813960, 06:39:20 DEBUG "0" 06:39:20 DEBUG ] 06:39:20 DEBUG } 06:39:20 DEBUG ] 06:39:20 DEBUG } 06:39:20 DEBUG } 06:39:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813960" 06:39:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813960" 06:39:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712813961" 06:39:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712813961" 06:39:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813961" 06:39:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712813961" 06:39:21 INFO [http_cmd]: http status code OK 06:39:21 INFO [http_cmd]: http status code OK 06:39:21 DEBUG --- status code --- 06:39:21 DEBUG --- status code --- 06:39:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:21 DEBUG --- http response --- 06:39:21 DEBUG --- http response --- 06:39:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:39:21 DEBUG ----- output ----- 06:39:21 DEBUG { 06:39:21 DEBUG "status": "success", 06:39:21 DEBUG "data": { 06:39:21 DEBUG "resultType": "vector", 06:39:21 DEBUG "result": [ 06:39:21 DEBUG { 06:39:21 DEBUG "metric": { 06:39:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:39:21 DEBUG }, 06:39:21 DEBUG "value": [ 06:39:21 DEBUG 06:39:21 DEBUG 06:39:21 DEBUG 06:39:21 DEBUG [...] 06:39:21 DEBUG 06:39:21 DEBUG 06:39:21 DEBUG 06:39:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:39:21 DEBUG }, 06:39:21 DEBUG "value": [ 06:39:21 DEBUG 1712813960, 06:39:21 DEBUG "1366.9666666666667" 06:39:21 DEBUG ] 06:39:21 DEBUG } 06:39:21 DEBUG ] 06:39:21 DEBUG } 06:39:21 DEBUG } 06:39:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:39:21 DEBUG ----- output ----- 06:39:21 DEBUG { 06:39:21 DEBUG "status": "success", 06:39:21 DEBUG "data": { 06:39:21 DEBUG "resultType": "vector", 06:39:21 DEBUG "result": [ 06:39:21 DEBUG { 06:39:21 DEBUG "metric": { 06:39:21 DEBUG "pod": "ds-idrepo-2" 06:39:21 DEBUG }, 06:39:21 DEBUG "value": [ 06:39:21 DEBUG 06:39:21 DEBUG 06:39:21 DEBUG 06:39:21 DEBUG [...] 06:39:21 DEBUG 06:39:21 DEBUG 06:39:21 DEBUG 06:39:21 DEBUG "pod": "ds-cts-2" 06:39:21 DEBUG }, 06:39:21 DEBUG "value": [ 06:39:21 DEBUG 1712813960, 06:39:21 DEBUG "0" 06:39:21 DEBUG ] 06:39:21 DEBUG } 06:39:21 DEBUG ] 06:39:21 DEBUG } 06:39:21 DEBUG } 06:39:21 INFO [http_cmd]: http status code OK 06:39:21 DEBUG --- status code --- 06:39:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:21 DEBUG --- http response --- 06:39:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:39:21 DEBUG ----- output ----- 06:39:21 DEBUG { 06:39:21 DEBUG "status": "success", 06:39:21 DEBUG "data": { 06:39:21 DEBUG "resultType": "vector", 06:39:21 DEBUG "result": [ 06:39:21 DEBUG { 06:39:21 DEBUG "metric": { 06:39:21 DEBUG "instance": "10.142.0.107:9100" 06:39:21 DEBUG }, 06:39:21 DEBUG "value": [ 06:39:21 DEBUG 06:39:21 DEBUG 06:39:21 DEBUG 06:39:21 DEBUG [...] 06:39:21 DEBUG 06:39:21 DEBUG 06:39:21 DEBUG 06:39:21 DEBUG "instance": "10.142.0.77:9100" 06:39:21 DEBUG }, 06:39:21 DEBUG "value": [ 06:39:21 DEBUG 1712813961, 06:39:21 DEBUG "0.0010000000000000378" 06:39:21 DEBUG ] 06:39:21 DEBUG } 06:39:21 DEBUG ] 06:39:21 DEBUG } 06:39:21 DEBUG } 06:39:21 INFO [http_cmd]: http status code OK 06:39:21 DEBUG --- status code --- 06:39:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:21 DEBUG --- http response --- 06:39:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:39:21 DEBUG ----- output ----- 06:39:21 DEBUG { 06:39:21 DEBUG "status": "success", 06:39:21 DEBUG "data": { 06:39:21 DEBUG "resultType": "vector", 06:39:21 DEBUG "result": [ 06:39:21 DEBUG { 06:39:21 DEBUG "metric": { 06:39:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:39:21 DEBUG }, 06:39:21 DEBUG "value": [ 06:39:21 DEBUG 06:39:21 DEBUG 06:39:21 DEBUG 06:39:21 DEBUG [...] 06:39:21 DEBUG 06:39:21 DEBUG 06:39:21 DEBUG 06:39:21 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:39:21 DEBUG }, 06:39:21 DEBUG "value": [ 06:39:21 DEBUG 1712813961, 06:39:21 DEBUG "0" 06:39:21 DEBUG ] 06:39:21 DEBUG } 06:39:21 DEBUG ] 06:39:21 DEBUG } 06:39:21 DEBUG } 06:39:21 INFO [http_cmd]: http status code OK 06:39:21 INFO [http_cmd]: http status code OK 06:39:21 DEBUG --- status code --- 06:39:21 DEBUG --- status code --- 06:39:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:39:21 DEBUG --- http response --- 06:39:21 DEBUG --- http response --- 06:39:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:39:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:39:21 DEBUG ----- output ----- 06:39:21 DEBUG ----- output ----- 06:39:21 DEBUG { 06:39:21 DEBUG { 06:39:21 DEBUG "status": "success", 06:39:21 DEBUG "status": "success", 06:39:21 DEBUG "data": { 06:39:21 DEBUG "data": { 06:39:21 DEBUG "resultType": "vector", 06:39:21 DEBUG "resultType": "vector", 06:39:21 DEBUG "result": [ 06:39:21 DEBUG "result": [ 06:39:21 DEBUG { 06:39:21 DEBUG { 06:39:21 DEBUG "metric": { 06:39:21 DEBUG "metric": { 06:39:21 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:39:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:39:21 DEBUG }, 06:39:21 DEBUG }, 06:39:21 DEBUG "value": [ 06:39:21 DEBUG "value": [ 06:39:21 DEBUG 06:39:21 DEBUG 06:39:21 DEBUG 06:39:21 DEBUG 06:39:21 DEBUG 06:39:21 DEBUG 06:39:21 DEBUG [...] 06:39:21 DEBUG [...] 06:39:21 DEBUG 06:39:21 DEBUG 06:39:21 DEBUG 06:39:21 DEBUG 06:39:21 DEBUG 06:39:21 DEBUG 06:39:21 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:39:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:39:21 DEBUG }, 06:39:21 DEBUG }, 06:39:21 DEBUG "value": [ 06:39:21 DEBUG "value": [ 06:39:21 DEBUG 1712813961, 06:39:21 DEBUG 1712813961, 06:39:21 DEBUG "4.190917898004531" 06:39:21 DEBUG "1373.8333333333333" 06:39:21 DEBUG ] 06:39:21 DEBUG ] 06:39:21 DEBUG } 06:39:21 DEBUG } 06:39:21 DEBUG ] 06:39:21 DEBUG ] 06:39:21 DEBUG } 06:39:21 DEBUG } 06:39:21 DEBUG } 06:39:21 DEBUG } 06:39:32 INFO 06:39:32 INFO [loop_until]: kubectl --namespace=xlou top pods 06:39:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:39:32 INFO [loop_until]: OK (rc = 0) 06:39:32 DEBUG --- stdout --- 06:39:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 8867m 8501Mi am-5b5bd8567b-8h2w4 8714m 8565Mi am-5b5bd8567b-qlkw6 7518m 8459Mi ds-cts-0 8348m 13193Mi ds-cts-1 8620m 12685Mi ds-cts-2 8513m 13202Mi ds-idrepo-0 11311m 5498Mi ds-idrepo-1 349m 5479Mi ds-idrepo-2 400m 5783Mi end-user-ui-58878d7668-gjq6d 1m 4Mi idm-668d898cfd-2vqdm 4m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 1m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4096m 1850Mi 06:39:32 DEBUG --- stderr --- 06:39:32 DEBUG 06:39:35 INFO 06:39:35 INFO [loop_until]: kubectl --namespace=xlou top node 06:39:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:39:35 INFO [loop_until]: OK (rc = 0) 06:39:35 DEBUG --- stdout --- 06:39:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 2744Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 62m 0% 2771Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7836m 26% 10113Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10733m 35% 2985Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9427m 31% 10079Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 142m 0% 3852Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9602m 32% 10191Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 466m 2% 6461Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11596m 72% 6191Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8702m 54% 13883Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 8258m 51% 13706Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 8815m 55% 13264Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 417m 2% 6153Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4223m 26% 5617Mi 9% 06:39:35 DEBUG --- stderr --- 06:39:35 DEBUG 06:40:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814020" 06:40:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814020" 06:40:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712814020" 06:40:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814020" 06:40:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814020" 06:40:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814020" 06:40:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814020" 06:40:20 INFO [http_cmd]: http status code OK 06:40:20 DEBUG --- status code --- 06:40:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:20 DEBUG --- http response --- 06:40:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:40:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814020" 06:40:20 DEBUG ----- output ----- 06:40:20 INFO [http_cmd]: http status code OK 06:40:20 DEBUG { 06:40:20 DEBUG --- status code --- 06:40:20 DEBUG "status": "success", 06:40:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:20 DEBUG "data": { 06:40:20 DEBUG --- http response --- 06:40:20 DEBUG "resultType": "vector", 06:40:20 DEBUG "result": [ 06:40:20 DEBUG { 06:40:20 DEBUG "metric": { 06:40:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:40:20 DEBUG "pod": "ds-idrepo-2" 06:40:20 DEBUG ----- output ----- 06:40:20 DEBUG }, 06:40:20 DEBUG { 06:40:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814020" 06:40:20 DEBUG "value": [ 06:40:20 DEBUG "status": "success", 06:40:20 DEBUG 06:40:20 DEBUG "data": { 06:40:20 INFO [http_cmd]: http status code OK 06:40:20 DEBUG 06:40:20 INFO [http_cmd]: http status code OK 06:40:20 DEBUG "resultType": "vector", 06:40:20 INFO [http_cmd]: http status code OK 06:40:20 DEBUG --- status code --- 06:40:20 DEBUG 06:40:20 DEBUG --- status code --- 06:40:20 DEBUG "result": [ 06:40:20 DEBUG --- status code --- 06:40:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:20 DEBUG [...] 06:40:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:20 DEBUG { 06:40:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:20 DEBUG --- http response --- 06:40:20 DEBUG 06:40:20 DEBUG --- http response --- 06:40:20 DEBUG "metric": { 06:40:20 DEBUG --- http response --- 06:40:20 DEBUG 06:40:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:40:20 DEBUG "instance": "10.142.0.107:9100" 06:40:20 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:40:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:40:20 DEBUG 06:40:20 DEBUG ----- output ----- 06:40:20 DEBUG }, 06:40:20 DEBUG ----- output ----- 06:40:20 DEBUG "pod": "ds-cts-2" 06:40:20 DEBUG { 06:40:20 DEBUG "value": [ 06:40:20 DEBUG { 06:40:20 DEBUG }, 06:40:20 DEBUG "status": "success", 06:40:20 DEBUG 06:40:20 DEBUG "status": "success", 06:40:20 DEBUG "value": [ 06:40:20 DEBUG "data": { 06:40:20 DEBUG 06:40:20 DEBUG "data": { 06:40:20 DEBUG 1712814020, 06:40:20 DEBUG "29.633333333333333" 06:40:20 DEBUG "resultType": "vector", 06:40:20 DEBUG "resultType": "vector", 06:40:20 DEBUG 06:40:20 DEBUG ] 06:40:20 DEBUG "result": [ 06:40:20 DEBUG "result": [ 06:40:20 DEBUG [...] 06:40:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814020" 06:40:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814020" 06:40:20 DEBUG } 06:40:20 DEBUG { 06:40:20 DEBUG { 06:40:20 DEBUG 06:40:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712814020" 06:40:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814020" 06:40:20 INFO [http_cmd]: http status code OK 06:40:20 DEBUG ] 06:40:20 DEBUG "metric": { 06:40:20 INFO [http_cmd]: http status code OK 06:40:20 DEBUG "metric": { 06:40:20 INFO [http_cmd]: http status code OK 06:40:20 INFO [http_cmd]: http status code OK 06:40:20 DEBUG 06:40:20 DEBUG --- status code --- 06:40:20 DEBUG } 06:40:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:40:20 DEBUG --- status code --- 06:40:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:40:20 DEBUG --- status code --- 06:40:20 DEBUG --- status code --- 06:40:20 DEBUG 06:40:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:20 DEBUG } 06:40:20 DEBUG }, 06:40:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:20 DEBUG }, 06:40:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:20 INFO [http_cmd]: http status code OK 06:40:20 DEBUG "instance": "10.142.0.77:9100" 06:40:20 INFO [http_cmd]: http status code OK 06:40:20 DEBUG --- http response --- 06:40:20 DEBUG "value": [ 06:40:20 DEBUG --- http response --- 06:40:20 DEBUG "value": [ 06:40:20 DEBUG --- http response --- 06:40:20 DEBUG --- http response --- 06:40:20 DEBUG --- status code --- 06:40:20 DEBUG }, 06:40:20 DEBUG --- status code --- 06:40:20 DEBUG 06:40:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:40:20 DEBUG 06:40:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:40:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:40:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:40:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:20 DEBUG "value": [ 06:40:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:20 DEBUG 06:40:20 DEBUG ----- output ----- 06:40:20 DEBUG 06:40:20 DEBUG ----- output ----- 06:40:20 DEBUG ----- output ----- 06:40:20 DEBUG ----- output ----- 06:40:20 DEBUG --- http response --- 06:40:20 DEBUG 1712814020, 06:40:20 DEBUG --- http response --- 06:40:20 DEBUG 06:40:20 DEBUG { 06:40:20 DEBUG 06:40:20 DEBUG { 06:40:20 DEBUG { 06:40:20 DEBUG { 06:40:20 DEBUG "82603837.6" 06:40:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:40:20 DEBUG [...] 06:40:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:40:20 DEBUG "status": "success", 06:40:20 DEBUG [...] 06:40:20 DEBUG "status": "success", 06:40:20 DEBUG "status": "success", 06:40:20 DEBUG "status": "success", 06:40:20 DEBUG ] 06:40:20 DEBUG ----- output ----- 06:40:20 DEBUG 06:40:20 DEBUG ----- output ----- 06:40:20 DEBUG "data": { 06:40:20 DEBUG 06:40:20 DEBUG "data": { 06:40:20 DEBUG "data": { 06:40:20 DEBUG "data": { 06:40:20 DEBUG } 06:40:20 DEBUG { 06:40:20 DEBUG "status": "success", 06:40:20 DEBUG { 06:40:20 ERROR [http_cmd]: ERROR 06:40:20 DEBUG "resultType": "vector", 06:40:20 DEBUG "result": [ 06:40:20 DEBUG "resultType": "vector", 06:40:20 DEBUG "resultType": "vector", 06:40:20 DEBUG "resultType": "vector", 06:40:20 DEBUG ] 06:40:20 DEBUG "data": { 06:40:20 DEBUG 06:40:20 INFO [http_cmd]: http status code OK 06:40:20 DEBUG "status": "success", 06:40:20 ERROR --- status code --- 06:40:20 DEBUG { 06:40:20 DEBUG 06:40:20 DEBUG "result": [ 06:40:20 DEBUG "result": [ 06:40:20 DEBUG "result": [ 06:40:20 DEBUG } 06:40:20 DEBUG "resultType": "vector", 06:40:20 DEBUG 06:40:20 DEBUG --- status code --- 06:40:20 DEBUG "data": { 06:40:20 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:40:20 DEBUG "metric": { 06:40:20 DEBUG 06:40:20 DEBUG { 06:40:20 DEBUG { 06:40:20 DEBUG { 06:40:20 DEBUG } 06:40:20 DEBUG "result": [ 06:40:20 DEBUG { 06:40:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:20 DEBUG "resultType": "vector", 06:40:20 ERROR --- http response --- 06:40:20 DEBUG "instance": "10.142.0.107:9100" 06:40:20 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:40:20 DEBUG "metric": { 06:40:20 DEBUG "metric": { 06:40:20 DEBUG "metric": { 06:40:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:40:20 DEBUG "metric": { 06:40:20 DEBUG --- http response --- 06:40:20 DEBUG "result": [ 06:40:20 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:40:20 DEBUG }, 06:40:20 DEBUG "value": [ 06:40:20 DEBUG "instance": "10.142.0.107:9100" 06:40:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:40:20 DEBUG "pod": "ds-idrepo-2" 06:40:20 DEBUG }, 06:40:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:40:20 DEBUG { 06:40:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:40:20 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:40:20 DEBUG }, 06:40:20 DEBUG 06:40:20 DEBUG }, 06:40:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814020" 06:40:20 DEBUG }, 06:40:20 DEBUG }, 06:40:20 DEBUG "value": [ 06:40:20 DEBUG }, 06:40:20 DEBUG "metric": { 06:40:20 DEBUG ----- output ----- 06:40:20 DEBUG "value": [ 06:40:20 DEBUG 06:40:20 DEBUG "value": [ 06:40:20 DEBUG "value": [ 06:40:20 DEBUG "value": [ 06:40:20 DEBUG 06:40:20 DEBUG "value": [ 06:40:20 DEBUG "pod": "ds-idrepo-2" 06:40:20 DEBUG { 06:40:20 DEBUG 1712814020, 06:40:20 DEBUG 06:40:20 DEBUG 06:40:20 DEBUG 06:40:20 DEBUG 1712814020, 06:40:20 DEBUG 06:40:20 DEBUG 06:40:20 DEBUG }, 06:40:20 DEBUG "status": "success", 06:40:20 DEBUG "0" 06:40:20 DEBUG [...] 06:40:20 DEBUG 06:40:20 DEBUG 06:40:20 DEBUG "0.021" 06:40:20 DEBUG 06:40:20 DEBUG 06:40:20 DEBUG "value": [ 06:40:20 DEBUG "data": { 06:40:20 DEBUG ] 06:40:20 DEBUG 06:40:20 DEBUG 06:40:20 DEBUG 06:40:20 DEBUG ] 06:40:20 DEBUG [...] 06:40:20 DEBUG 06:40:20 DEBUG 06:40:20 DEBUG "resultType": "vector", 06:40:20 DEBUG } 06:40:20 DEBUG 06:40:20 DEBUG [...] 06:40:20 DEBUG [...] 06:40:20 DEBUG } 06:40:20 DEBUG 06:40:20 DEBUG [...] 06:40:20 DEBUG 06:40:20 DEBUG "result": [ 06:40:20 DEBUG ] 06:40:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814020" 06:40:20 DEBUG 06:40:20 DEBUG 06:40:20 DEBUG 06:40:20 DEBUG ] 06:40:20 DEBUG 06:40:20 DEBUG 06:40:20 DEBUG 06:40:20 DEBUG { 06:40:20 DEBUG } 06:40:20 DEBUG "instance": "10.142.0.77:9100" 06:40:20 DEBUG 06:40:20 DEBUG 06:40:20 DEBUG } 06:40:20 DEBUG 06:40:20 DEBUG 06:40:20 DEBUG [...] 06:40:20 DEBUG "metric": { 06:40:20 DEBUG } 06:40:20 DEBUG }, 06:40:20 DEBUG 06:40:20 DEBUG "instance": "10.142.0.77:9100" 06:40:20 DEBUG 06:40:20 DEBUG } 06:40:20 DEBUG "pod": "ds-idrepo-0" 06:40:20 DEBUG 06:40:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:40:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:40:20 DEBUG "value": [ 06:40:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814020" 06:40:20 DEBUG }, 06:40:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:40:20 DEBUG }, 06:40:20 DEBUG "value": [ 06:40:20 DEBUG }, 06:40:20 DEBUG }, 06:40:20 DEBUG "value": [ 06:40:20 DEBUG 1712814020, 06:40:20 DEBUG }, 06:40:20 DEBUG 06:40:20 DEBUG 1712814020, 06:40:20 DEBUG "value": [ 06:40:20 DEBUG "value": [ 06:40:20 DEBUG 06:40:20 DEBUG "52947162112" 06:40:20 DEBUG "value": [ 06:40:20 DEBUG 06:40:20 DEBUG "0" 06:40:20 DEBUG 1712814020, 06:40:20 DEBUG 1712814020, 06:40:20 DEBUG 06:40:20 DEBUG ] 06:40:20 DEBUG 1712814020, 06:40:20 DEBUG 06:40:20 DEBUG ] 06:40:20 DEBUG "245598" 06:40:20 DEBUG "0.002233333333333339" 06:40:20 DEBUG 06:40:20 DEBUG } 06:40:20 DEBUG ] 06:40:20 DEBUG "pod": "ds-cts-2" 06:40:20 DEBUG } 06:40:20 DEBUG ] 06:40:20 DEBUG ] 06:40:20 DEBUG [...] 06:40:20 DEBUG "0" 06:40:20 DEBUG } 06:40:20 DEBUG }, 06:40:20 DEBUG ] 06:40:20 DEBUG } 06:40:20 DEBUG } 06:40:20 DEBUG ] 06:40:20 DEBUG 06:40:20 DEBUG 06:40:20 DEBUG "value": [ 06:40:20 INFO [http_cmd]: http status code OK 06:40:20 DEBUG } 06:40:20 DEBUG ] 06:40:20 DEBUG } 06:40:20 DEBUG ] 06:40:20 DEBUG } 06:40:20 DEBUG 06:40:20 DEBUG 1712814020, 06:40:20 DEBUG --- status code --- 06:40:20 DEBUG } 06:40:20 DEBUG } 06:40:20 DEBUG } 06:40:20 DEBUG } 06:40:20 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:40:20 DEBUG "0.000697326111111111" 06:40:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:20 DEBUG } 06:40:20 DEBUG ] 06:40:20 DEBUG }, 06:40:20 DEBUG ] 06:40:20 DEBUG --- http response --- 06:40:20 DEBUG } 06:40:20 DEBUG "value": [ 06:40:20 INFO [http_cmd]: http status code OK 06:40:20 DEBUG } 06:40:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:40:20 DEBUG } 06:40:20 DEBUG 1712814020, 06:40:20 DEBUG --- status code --- 06:40:20 DEBUG ] 06:40:20 DEBUG ----- output ----- 06:40:20 DEBUG "65880995.111352526" 06:40:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:20 DEBUG } 06:40:20 DEBUG { 06:40:20 DEBUG ] 06:40:20 DEBUG --- http response --- 06:40:20 DEBUG } 06:40:20 DEBUG "status": "success", 06:40:20 DEBUG } 06:40:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:40:20 DEBUG "data": { 06:40:20 DEBUG ] 06:40:20 DEBUG ----- output ----- 06:40:20 DEBUG "resultType": "vector", 06:40:20 DEBUG } 06:40:20 DEBUG { 06:40:20 DEBUG "result": [ 06:40:20 DEBUG } 06:40:20 DEBUG "status": "success", 06:40:20 DEBUG { 06:40:20 INFO [http_cmd]: http status code OK 06:40:20 DEBUG "data": { 06:40:20 DEBUG "metric": { 06:40:20 DEBUG --- status code --- 06:40:20 DEBUG "resultType": "vector", 06:40:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:40:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:20 DEBUG "result": [ 06:40:20 DEBUG }, 06:40:20 DEBUG --- http response --- 06:40:20 DEBUG { 06:40:20 DEBUG "value": [ 06:40:20 DEBUG "metric": { 06:40:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:40:20 DEBUG 06:40:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:40:20 DEBUG ----- output ----- 06:40:20 DEBUG 06:40:20 DEBUG }, 06:40:20 DEBUG { 06:40:20 DEBUG 06:40:20 DEBUG "value": [ 06:40:20 DEBUG "status": "success", 06:40:20 DEBUG [...] 06:40:20 DEBUG 06:40:20 DEBUG "data": { 06:40:20 DEBUG 06:40:20 DEBUG 06:40:20 DEBUG "resultType": "vector", 06:40:20 DEBUG 06:40:20 DEBUG 06:40:20 DEBUG "result": [ 06:40:20 DEBUG 06:40:20 DEBUG [...] 06:40:20 DEBUG { 06:40:20 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:40:20 DEBUG 06:40:20 DEBUG "metric": { 06:40:20 DEBUG }, 06:40:20 DEBUG 06:40:20 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:40:20 DEBUG "value": [ 06:40:20 DEBUG 06:40:20 DEBUG }, 06:40:20 DEBUG 1712814020, 06:40:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:40:20 DEBUG "value": [ 06:40:20 DEBUG "4.0872164067082375" 06:40:20 DEBUG }, 06:40:20 DEBUG 06:40:20 DEBUG ] 06:40:20 DEBUG "value": [ 06:40:20 DEBUG 06:40:20 DEBUG } 06:40:20 DEBUG 1712814020, 06:40:20 DEBUG 06:40:20 DEBUG ] 06:40:20 DEBUG "1361.6" 06:40:20 DEBUG [...] 06:40:20 DEBUG } 06:40:20 DEBUG ] 06:40:20 DEBUG 06:40:20 DEBUG } 06:40:20 DEBUG } 06:40:20 DEBUG 06:40:20 DEBUG ] 06:40:20 DEBUG 06:40:20 DEBUG } 06:40:20 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:40:20 DEBUG } 06:40:20 DEBUG }, 06:40:20 DEBUG "value": [ 06:40:20 DEBUG 1712814020, 06:40:20 DEBUG "1359.3333333333333" 06:40:20 DEBUG ] 06:40:20 DEBUG } 06:40:20 DEBUG ] 06:40:20 DEBUG } 06:40:20 DEBUG } 06:40:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712814020" 06:40:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814020" 06:40:20 INFO [http_cmd]: http status code OK 06:40:20 DEBUG --- status code --- 06:40:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:20 DEBUG --- http response --- 06:40:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:40:20 DEBUG ----- output ----- 06:40:20 DEBUG { 06:40:20 DEBUG "status": "success", 06:40:20 DEBUG "data": { 06:40:20 DEBUG "resultType": "vector", 06:40:20 DEBUG "result": [ 06:40:20 DEBUG { 06:40:20 DEBUG "metric": { 06:40:20 DEBUG "pod": "ds-idrepo-2" 06:40:20 DEBUG }, 06:40:20 DEBUG "value": [ 06:40:20 DEBUG 06:40:20 DEBUG 06:40:20 DEBUG 06:40:20 DEBUG [...] 06:40:20 DEBUG 06:40:20 DEBUG 06:40:20 DEBUG 06:40:20 DEBUG "pod": "ds-cts-2" 06:40:20 DEBUG }, 06:40:20 DEBUG "value": [ 06:40:20 DEBUG 1712814020, 06:40:20 DEBUG "0" 06:40:20 DEBUG ] 06:40:20 DEBUG } 06:40:20 DEBUG ] 06:40:20 DEBUG } 06:40:20 DEBUG } 06:40:20 INFO [http_cmd]: http status code OK 06:40:20 DEBUG --- status code --- 06:40:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:20 DEBUG --- http response --- 06:40:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:40:20 DEBUG ----- output ----- 06:40:20 DEBUG { 06:40:20 DEBUG "status": "success", 06:40:20 DEBUG "data": { 06:40:20 DEBUG "resultType": "vector", 06:40:20 DEBUG "result": [ 06:40:20 DEBUG { 06:40:20 DEBUG "metric": { 06:40:20 DEBUG "pod": "ds-idrepo-2" 06:40:20 DEBUG }, 06:40:20 DEBUG "value": [ 06:40:20 DEBUG 06:40:20 DEBUG 06:40:20 DEBUG 06:40:20 DEBUG [...] 06:40:20 DEBUG 06:40:20 DEBUG 06:40:20 DEBUG 06:40:20 DEBUG "pod": "ds-idrepo-0" 06:40:20 DEBUG }, 06:40:20 DEBUG "value": [ 06:40:20 DEBUG 1712814020, 06:40:20 DEBUG "0" 06:40:20 DEBUG ] 06:40:20 DEBUG } 06:40:20 DEBUG ] 06:40:20 DEBUG } 06:40:20 DEBUG } 06:40:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814020" 06:40:20 INFO [http_cmd]: http status code OK 06:40:20 DEBUG --- status code --- 06:40:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:20 DEBUG --- http response --- 06:40:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:40:20 DEBUG ----- output ----- 06:40:20 DEBUG { 06:40:20 DEBUG "status": "success", 06:40:20 DEBUG "data": { 06:40:20 DEBUG "resultType": "vector", 06:40:20 DEBUG "result": [ 06:40:20 DEBUG { 06:40:20 DEBUG "metric": { 06:40:20 DEBUG "instance": "10.142.0.107:9100" 06:40:20 DEBUG }, 06:40:20 DEBUG "value": [ 06:40:20 DEBUG 06:40:20 DEBUG 06:40:20 DEBUG 06:40:20 DEBUG [...] 06:40:20 DEBUG 06:40:20 DEBUG 06:40:20 DEBUG 06:40:20 DEBUG "instance": "10.142.0.77:9100" 06:40:20 DEBUG }, 06:40:20 DEBUG "value": [ 06:40:20 DEBUG 1712814020, 06:40:20 DEBUG "82601373.5" 06:40:20 DEBUG ] 06:40:20 DEBUG } 06:40:20 DEBUG ] 06:40:20 DEBUG } 06:40:20 DEBUG } 06:40:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712814021" 06:40:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814021" 06:40:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814021" 06:40:21 INFO [http_cmd]: http status code OK 06:40:21 DEBUG --- status code --- 06:40:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:21 DEBUG --- http response --- 06:40:21 INFO [http_cmd]: http status code OK 06:40:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:40:21 DEBUG --- status code --- 06:40:21 DEBUG ----- output ----- 06:40:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:21 DEBUG { 06:40:21 DEBUG --- http response --- 06:40:21 DEBUG "status": "success", 06:40:21 DEBUG "data": { 06:40:21 DEBUG "resultType": "vector", 06:40:21 DEBUG "result": [ 06:40:21 DEBUG { 06:40:21 DEBUG "metric": { 06:40:21 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:40:21 DEBUG }, 06:40:21 DEBUG "value": [ 06:40:21 DEBUG 06:40:21 DEBUG 06:40:21 DEBUG 06:40:21 DEBUG [...] 06:40:21 DEBUG 06:40:21 DEBUG 06:40:21 DEBUG 06:40:21 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 06:40:21 DEBUG }, 06:40:21 DEBUG "value": [ 06:40:21 DEBUG 1712814021, 06:40:21 DEBUG "14137638912" 06:40:21 DEBUG ] 06:40:21 DEBUG } 06:40:21 DEBUG ] 06:40:21 DEBUG } 06:40:21 DEBUG } 06:40:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:40:21 DEBUG ----- output ----- 06:40:21 DEBUG { 06:40:21 DEBUG "status": "success", 06:40:21 DEBUG "data": { 06:40:21 DEBUG "resultType": "vector", 06:40:21 DEBUG "result": [ 06:40:21 DEBUG { 06:40:21 DEBUG "metric": { 06:40:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:40:21 DEBUG }, 06:40:21 DEBUG "value": [ 06:40:21 DEBUG 06:40:21 DEBUG 06:40:21 DEBUG 06:40:21 DEBUG [...] 06:40:21 DEBUG 06:40:21 DEBUG 06:40:21 DEBUG 06:40:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:40:21 DEBUG }, 06:40:21 DEBUG "value": [ 06:40:21 DEBUG 1712814021, 06:40:21 DEBUG "1362.0666666666666" 06:40:21 DEBUG ] 06:40:21 DEBUG } 06:40:21 DEBUG ] 06:40:21 DEBUG } 06:40:21 DEBUG } 06:40:21 INFO [http_cmd]: http status code OK 06:40:21 DEBUG --- status code --- 06:40:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:21 DEBUG --- http response --- 06:40:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:40:21 DEBUG ----- output ----- 06:40:21 DEBUG { 06:40:21 DEBUG "status": "success", 06:40:21 DEBUG "data": { 06:40:21 DEBUG "resultType": "vector", 06:40:21 DEBUG "result": [ 06:40:21 DEBUG { 06:40:21 DEBUG "metric": { 06:40:21 DEBUG "pod": "ds-idrepo-2" 06:40:21 DEBUG }, 06:40:21 DEBUG "value": [ 06:40:21 DEBUG 06:40:21 DEBUG 06:40:21 DEBUG 06:40:21 DEBUG [...] 06:40:21 DEBUG 06:40:21 DEBUG 06:40:21 DEBUG 06:40:21 DEBUG "pod": "ds-idrepo-0" 06:40:21 DEBUG }, 06:40:21 DEBUG "value": [ 06:40:21 DEBUG 1712814021, 06:40:21 DEBUG "0" 06:40:21 DEBUG ] 06:40:21 DEBUG } 06:40:21 DEBUG ] 06:40:21 DEBUG } 06:40:21 DEBUG } 06:40:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814021" 06:40:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814021" 06:40:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814021" 06:40:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712814021" 06:40:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814021" 06:40:21 INFO [http_cmd]: http status code OK 06:40:21 DEBUG --- status code --- 06:40:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:21 DEBUG --- http response --- 06:40:21 INFO [http_cmd]: http status code OK 06:40:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:40:21 DEBUG --- status code --- 06:40:21 DEBUG ----- output ----- 06:40:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:21 DEBUG { 06:40:21 DEBUG --- http response --- 06:40:21 DEBUG "status": "success", 06:40:21 DEBUG "data": { 06:40:21 DEBUG "resultType": "vector", 06:40:21 DEBUG "result": [ 06:40:21 DEBUG { 06:40:21 DEBUG "metric": { 06:40:21 DEBUG "pod": "ds-idrepo-2" 06:40:21 DEBUG }, 06:40:21 DEBUG "value": [ 06:40:21 DEBUG 06:40:21 DEBUG 06:40:21 DEBUG 06:40:21 DEBUG [...] 06:40:21 DEBUG 06:40:21 DEBUG 06:40:21 DEBUG 06:40:21 DEBUG "pod": "ds-cts-2" 06:40:21 DEBUG }, 06:40:21 DEBUG "value": [ 06:40:21 DEBUG 1712814021, 06:40:21 DEBUG "0" 06:40:21 DEBUG ] 06:40:21 DEBUG } 06:40:21 DEBUG ] 06:40:21 DEBUG } 06:40:21 DEBUG } 06:40:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:40:21 INFO [http_cmd]: http status code OK 06:40:21 DEBUG ----- output ----- 06:40:21 DEBUG --- status code --- 06:40:21 DEBUG { 06:40:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:21 DEBUG "status": "success", 06:40:21 DEBUG --- http response --- 06:40:21 DEBUG "data": { 06:40:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:40:21 DEBUG "resultType": "vector", 06:40:21 DEBUG ----- output ----- 06:40:21 DEBUG "result": [ 06:40:21 DEBUG { 06:40:21 DEBUG { 06:40:21 DEBUG "status": "success", 06:40:21 DEBUG "metric": { 06:40:21 DEBUG "data": { 06:40:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:40:21 DEBUG "resultType": "vector", 06:40:21 DEBUG }, 06:40:21 DEBUG "result": [ 06:40:21 DEBUG "value": [ 06:40:21 DEBUG { 06:40:21 DEBUG 06:40:21 DEBUG "metric": { 06:40:21 DEBUG 06:40:21 DEBUG "instance": "10.142.0.107:9100" 06:40:21 DEBUG 06:40:21 DEBUG }, 06:40:21 DEBUG [...] 06:40:21 DEBUG "value": [ 06:40:21 DEBUG 06:40:21 DEBUG 06:40:21 DEBUG 06:40:21 DEBUG 06:40:21 DEBUG 06:40:21 DEBUG 06:40:21 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:40:21 DEBUG [...] 06:40:21 DEBUG }, 06:40:21 DEBUG 06:40:21 DEBUG "value": [ 06:40:21 DEBUG 06:40:21 DEBUG 1712814021, 06:40:21 DEBUG 06:40:21 DEBUG "0" 06:40:21 DEBUG "instance": "10.142.0.77:9100" 06:40:21 DEBUG ] 06:40:21 DEBUG }, 06:40:21 DEBUG } 06:40:21 DEBUG "value": [ 06:40:21 DEBUG ] 06:40:21 DEBUG 1712814021, 06:40:21 DEBUG } 06:40:21 DEBUG "0.000666666666666534" 06:40:21 DEBUG } 06:40:21 DEBUG ] 06:40:21 DEBUG } 06:40:21 DEBUG ] 06:40:21 DEBUG } 06:40:21 DEBUG } 06:40:21 INFO [http_cmd]: http status code OK 06:40:21 INFO [http_cmd]: http status code OK 06:40:21 DEBUG --- status code --- 06:40:21 DEBUG --- status code --- 06:40:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:40:21 DEBUG --- http response --- 06:40:21 DEBUG --- http response --- 06:40:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:40:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:40:21 DEBUG ----- output ----- 06:40:21 DEBUG ----- output ----- 06:40:21 DEBUG { 06:40:21 DEBUG { 06:40:21 DEBUG "status": "success", 06:40:21 DEBUG "status": "success", 06:40:21 DEBUG "data": { 06:40:21 DEBUG "data": { 06:40:21 DEBUG "resultType": "vector", 06:40:21 DEBUG "resultType": "vector", 06:40:21 DEBUG "result": [ 06:40:21 DEBUG "result": [ 06:40:21 DEBUG { 06:40:21 DEBUG { 06:40:21 DEBUG "metric": { 06:40:21 DEBUG "metric": { 06:40:21 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:40:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:40:21 DEBUG }, 06:40:21 DEBUG }, 06:40:21 DEBUG "value": [ 06:40:21 DEBUG "value": [ 06:40:21 DEBUG 06:40:21 DEBUG 06:40:21 DEBUG 06:40:21 DEBUG 06:40:21 DEBUG 06:40:21 DEBUG 06:40:21 DEBUG [...] 06:40:21 DEBUG [...] 06:40:21 DEBUG 06:40:21 DEBUG 06:40:21 DEBUG 06:40:21 DEBUG 06:40:21 DEBUG 06:40:21 DEBUG 06:40:21 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:40:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:40:21 DEBUG }, 06:40:21 DEBUG }, 06:40:21 DEBUG "value": [ 06:40:21 DEBUG "value": [ 06:40:21 DEBUG 1712814021, 06:40:21 DEBUG 1712814021, 06:40:21 DEBUG "4.133452596919841" 06:40:21 DEBUG "1361.9" 06:40:21 DEBUG ] 06:40:21 DEBUG ] 06:40:21 DEBUG } 06:40:21 DEBUG } 06:40:21 DEBUG ] 06:40:21 DEBUG ] 06:40:21 DEBUG } 06:40:21 DEBUG } 06:40:21 DEBUG } 06:40:21 DEBUG } 06:40:32 INFO 06:40:32 INFO [loop_until]: kubectl --namespace=xlou top pods 06:40:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:40:32 INFO [loop_until]: OK (rc = 0) 06:40:32 DEBUG --- stdout --- 06:40:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 8946m 8504Mi am-5b5bd8567b-8h2w4 8847m 8562Mi am-5b5bd8567b-qlkw6 7597m 8460Mi ds-cts-0 7958m 13396Mi ds-cts-1 8643m 13298Mi ds-cts-2 8284m 13396Mi ds-idrepo-0 11242m 5501Mi ds-idrepo-1 382m 5479Mi ds-idrepo-2 353m 5783Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 4m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 2m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 3983m 1852Mi 06:40:32 DEBUG --- stderr --- 06:40:32 DEBUG 06:40:35 INFO 06:40:35 INFO [loop_until]: kubectl --namespace=xlou top node 06:40:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:40:35 INFO [loop_until]: OK (rc = 0) 06:40:35 DEBUG --- stdout --- 06:40:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 2730Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 64m 0% 2781Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7879m 26% 10116Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10729m 35% 2984Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9557m 32% 10089Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 147m 0% 3850Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9623m 32% 10200Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 406m 2% 6462Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11546m 72% 6196Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8536m 53% 13987Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 8148m 51% 13988Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 8841m 55% 14012Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 402m 2% 6155Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4236m 26% 5619Mi 9% 06:40:35 DEBUG --- stderr --- 06:40:35 DEBUG 06:41:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814080" 06:41:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814080" 06:41:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814080" 06:41:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814080" 06:41:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814080" 06:41:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712814080" 06:41:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814080" 06:41:20 INFO [http_cmd]: http status code OK 06:41:20 DEBUG --- status code --- 06:41:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:20 DEBUG --- http response --- 06:41:20 INFO [http_cmd]: http status code OK 06:41:20 INFO [http_cmd]: http status code OK 06:41:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:41:20 DEBUG --- status code --- 06:41:20 DEBUG --- status code --- 06:41:20 DEBUG ----- output ----- 06:41:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:20 DEBUG { 06:41:20 DEBUG --- http response --- 06:41:20 DEBUG --- http response --- 06:41:20 DEBUG "status": "success", 06:41:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:41:20 DEBUG "data": { 06:41:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:41:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814080" 06:41:20 DEBUG ----- output ----- 06:41:20 DEBUG "resultType": "vector", 06:41:20 DEBUG ----- output ----- 06:41:20 DEBUG { 06:41:20 DEBUG "result": [ 06:41:20 DEBUG { 06:41:20 DEBUG "status": "success", 06:41:20 DEBUG { 06:41:20 DEBUG "status": "success", 06:41:20 DEBUG "data": { 06:41:20 DEBUG "metric": { 06:41:20 DEBUG "data": { 06:41:20 DEBUG "resultType": "vector", 06:41:20 DEBUG "pod": "ds-idrepo-2" 06:41:20 DEBUG "resultType": "vector", 06:41:20 DEBUG "result": [ 06:41:20 DEBUG }, 06:41:20 DEBUG "result": [ 06:41:20 DEBUG { 06:41:20 DEBUG "value": [ 06:41:20 DEBUG { 06:41:20 DEBUG "metric": { 06:41:20 DEBUG 06:41:20 DEBUG "metric": { 06:41:20 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:41:20 DEBUG 06:41:20 DEBUG "instance": "10.142.0.107:9100" 06:41:20 DEBUG }, 06:41:20 DEBUG 06:41:20 DEBUG }, 06:41:20 DEBUG "value": [ 06:41:20 DEBUG [...] 06:41:20 DEBUG "value": [ 06:41:20 DEBUG 06:41:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814080" 06:41:20 DEBUG 06:41:20 DEBUG 06:41:20 DEBUG 06:41:20 DEBUG 06:41:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814080" 06:41:20 INFO [http_cmd]: http status code OK 06:41:20 INFO [http_cmd]: http status code OK 06:41:20 DEBUG 06:41:20 INFO [http_cmd]: http status code OK 06:41:20 DEBUG 06:41:20 DEBUG 06:41:20 DEBUG --- status code --- 06:41:20 DEBUG --- status code --- 06:41:20 DEBUG 06:41:20 DEBUG --- status code --- 06:41:20 DEBUG [...] 06:41:20 DEBUG "pod": "ds-cts-2" 06:41:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:20 DEBUG [...] 06:41:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:20 DEBUG 06:41:20 DEBUG }, 06:41:20 DEBUG --- http response --- 06:41:20 DEBUG --- http response --- 06:41:20 DEBUG 06:41:20 DEBUG --- http response --- 06:41:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814080" 06:41:20 DEBUG 06:41:20 DEBUG 06:41:20 INFO [http_cmd]: http status code OK 06:41:20 INFO [http_cmd]: http status code OK 06:41:20 DEBUG "value": [ 06:41:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814080" 06:41:20 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:41:20 DEBUG 06:41:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:41:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:41:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712814080" 06:41:20 DEBUG "pod": "ds-cts-1" 06:41:20 DEBUG --- status code --- 06:41:20 INFO [http_cmd]: http status code OK 06:41:20 DEBUG --- status code --- 06:41:20 DEBUG 1712814080, 06:41:20 DEBUG 06:41:20 DEBUG ----- output ----- 06:41:20 DEBUG ----- output ----- 06:41:20 DEBUG { 06:41:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:20 DEBUG --- status code --- 06:41:20 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814080" 06:41:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:20 INFO [http_cmd]: http status code OK 06:41:20 DEBUG "29.566666666666666" 06:41:20 DEBUG "instance": "10.142.0.77:9100" 06:41:20 DEBUG { 06:41:20 DEBUG }, 06:41:20 DEBUG "status": "success", 06:41:20 DEBUG --- http response --- 06:41:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:20 DEBUG --- http response --- 06:41:20 DEBUG --- status code --- 06:41:20 DEBUG ] 06:41:20 DEBUG } 06:41:20 DEBUG }, 06:41:20 DEBUG "value": [ 06:41:20 DEBUG "data": { 06:41:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:41:20 DEBUG --- http response --- 06:41:20 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:20 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:41:20 DEBUG "status": "success", 06:41:20 DEBUG ] 06:41:20 INFO [http_cmd]: http status code OK 06:41:21 DEBUG "value": [ 06:41:21 DEBUG 1712814080, 06:41:21 DEBUG "resultType": "vector", 06:41:21 DEBUG ----- output ----- 06:41:21 DEBUG --- http response --- 06:41:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:41:21 DEBUG ----- output ----- 06:41:21 DEBUG "data": { 06:41:21 INFO [http_cmd]: http status code OK 06:41:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814081" 06:41:21 DEBUG } 06:41:21 ERROR [http_cmd]: ERROR 06:41:21 DEBUG --- status code --- 06:41:21 DEBUG 1712814080, 06:41:21 DEBUG "4.945257986207047" 06:41:21 DEBUG "result": [ 06:41:21 DEBUG { 06:41:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:41:21 DEBUG ----- output ----- 06:41:21 DEBUG { 06:41:21 DEBUG "resultType": "vector", 06:41:21 DEBUG --- status code --- 06:41:21 DEBUG } 06:41:21 ERROR --- status code --- 06:41:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:21 DEBUG "52948397056" 06:41:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814081" 06:41:21 DEBUG ] 06:41:21 DEBUG } 06:41:21 DEBUG "status": "success", 06:41:21 DEBUG ----- output ----- 06:41:21 DEBUG { 06:41:21 DEBUG "status": "success", 06:41:21 DEBUG "result": [ 06:41:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:21 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:41:21 DEBUG --- http response --- 06:41:21 DEBUG ] 06:41:21 DEBUG { 06:41:21 DEBUG ] 06:41:21 DEBUG "data": { 06:41:21 INFO [http_cmd]: http status code OK 06:41:21 DEBUG { 06:41:21 DEBUG "status": "success", 06:41:21 DEBUG "data": { 06:41:21 DEBUG { 06:41:21 DEBUG --- http response --- 06:41:21 ERROR --- http response --- 06:41:21 DEBUG } 06:41:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:41:21 DEBUG ----- output ----- 06:41:21 DEBUG } 06:41:21 DEBUG "resultType": "vector", 06:41:21 DEBUG --- status code --- 06:41:21 DEBUG "status": "success", 06:41:21 DEBUG "data": { 06:41:21 DEBUG "resultType": "vector", 06:41:21 DEBUG "metric": { 06:41:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:41:21 DEBUG ] 06:41:21 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:41:21 DEBUG "metric": { 06:41:21 DEBUG { 06:41:21 DEBUG } 06:41:21 DEBUG "result": [ 06:41:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:21 DEBUG "data": { 06:41:21 DEBUG "resultType": "vector", 06:41:21 DEBUG "result": [ 06:41:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:41:21 DEBUG ----- output ----- 06:41:21 DEBUG } 06:41:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814081" 06:41:21 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:41:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:41:21 DEBUG "status": "success", 06:41:21 DEBUG { 06:41:21 DEBUG --- http response --- 06:41:21 DEBUG "resultType": "vector", 06:41:21 INFO [http_cmd]: http status code OK 06:41:21 DEBUG "result": [ 06:41:21 DEBUG { 06:41:21 DEBUG "metric": { 06:41:21 DEBUG { 06:41:21 DEBUG "status": "success", 06:41:21 DEBUG "data": { 06:41:21 DEBUG "data": { 06:41:21 DEBUG "metric": { 06:41:21 DEBUG "result": [ 06:41:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:41:21 DEBUG --- status code --- 06:41:21 DEBUG { 06:41:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:41:21 DEBUG }, 06:41:21 DEBUG } 06:41:21 DEBUG "resultType": "vector", 06:41:21 DEBUG }, 06:41:21 DEBUG "resultType": "vector", 06:41:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:41:21 DEBUG { 06:41:21 DEBUG ----- output ----- 06:41:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:21 DEBUG "metric": { 06:41:21 DEBUG }, 06:41:21 DEBUG "value": [ 06:41:21 DEBUG "result": [ 06:41:21 DEBUG "value": [ 06:41:21 DEBUG "result": [ 06:41:21 DEBUG { 06:41:21 DEBUG "metric": { 06:41:21 DEBUG { 06:41:21 DEBUG --- http response --- 06:41:21 DEBUG "instance": "10.142.0.107:9100" 06:41:21 DEBUG "value": [ 06:41:21 DEBUG 06:41:21 DEBUG { 06:41:21 DEBUG 06:41:21 DEBUG }, 06:41:21 DEBUG "metric": { 06:41:21 DEBUG "instance": "10.142.0.107:9100" 06:41:21 DEBUG "status": "success", 06:41:21 DEBUG }, 06:41:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG "value": [ 06:41:21 DEBUG "pod": "ds-idrepo-2" 06:41:21 INFO [http_cmd]: http status code OK 06:41:21 DEBUG }, 06:41:21 DEBUG "data": { 06:41:21 DEBUG "value": [ 06:41:21 DEBUG ----- output ----- 06:41:21 DEBUG 06:41:21 DEBUG "metric": { 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG }, 06:41:21 DEBUG --- status code --- 06:41:21 DEBUG "value": [ 06:41:21 DEBUG "resultType": "vector", 06:41:21 DEBUG 06:41:21 DEBUG { 06:41:21 DEBUG [...] 06:41:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:41:21 DEBUG [...] 06:41:21 DEBUG [...] 06:41:21 DEBUG 06:41:21 DEBUG "value": [ 06:41:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:21 DEBUG 06:41:21 DEBUG "result": [ 06:41:21 DEBUG 06:41:21 DEBUG "status": "success", 06:41:21 DEBUG 06:41:21 DEBUG }, 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG --- http response --- 06:41:21 DEBUG 06:41:21 DEBUG { 06:41:21 DEBUG 06:41:21 DEBUG "data": { 06:41:21 DEBUG 06:41:21 DEBUG "value": [ 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG [...] 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:41:21 DEBUG "metric": { 06:41:21 DEBUG [...] 06:41:21 DEBUG "resultType": "vector", 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 INFO [http_cmd]: http status code OK 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG [...] 06:41:21 DEBUG ----- output ----- 06:41:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:41:21 DEBUG 06:41:21 DEBUG "result": [ 06:41:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:41:21 DEBUG 06:41:21 DEBUG "pod": "ds-cts-1" 06:41:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:41:21 DEBUG --- status code --- 06:41:21 DEBUG 06:41:21 DEBUG [...] 06:41:21 DEBUG 06:41:21 DEBUG { 06:41:21 DEBUG }, 06:41:21 DEBUG 06:41:21 DEBUG { 06:41:21 DEBUG }, 06:41:21 DEBUG 06:41:21 DEBUG }, 06:41:21 DEBUG }, 06:41:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG "status": "success", 06:41:21 DEBUG "value": [ 06:41:21 DEBUG 06:41:21 DEBUG "metric": { 06:41:21 DEBUG "pod": "ds-idrepo-2" 06:41:21 DEBUG [...] 06:41:21 DEBUG 06:41:21 DEBUG "value": [ 06:41:21 DEBUG --- http response --- 06:41:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG "data": { 06:41:21 DEBUG 06:41:21 DEBUG "instance": "10.142.0.77:9100" 06:41:21 DEBUG }, 06:41:21 DEBUG }, 06:41:21 DEBUG "value": [ 06:41:21 DEBUG 06:41:21 DEBUG 1712814080, 06:41:21 DEBUG }, 06:41:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:41:21 DEBUG ----- output ----- 06:41:21 DEBUG "instance": "10.142.0.77:9100" 06:41:21 DEBUG "resultType": "vector", 06:41:21 DEBUG 06:41:21 DEBUG "value": [ 06:41:21 DEBUG 1712814080, 06:41:21 DEBUG "value": [ 06:41:21 DEBUG 1712814080, 06:41:21 DEBUG 06:41:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:41:21 DEBUG "value": [ 06:41:21 DEBUG 06:41:21 DEBUG { 06:41:21 DEBUG }, 06:41:21 DEBUG "result": [ 06:41:21 DEBUG 06:41:21 DEBUG "1366" 06:41:21 DEBUG "value": [ 06:41:21 DEBUG 06:41:21 DEBUG "405410910.3070611" 06:41:21 DEBUG "0.021" 06:41:21 DEBUG }, 06:41:21 DEBUG 1712814080, 06:41:21 DEBUG "pod": "ds-idrepo-0" 06:41:21 DEBUG "status": "success", 06:41:21 DEBUG "value": [ 06:41:21 DEBUG { 06:41:21 DEBUG [...] 06:41:21 DEBUG ] 06:41:21 DEBUG 1712814080, 06:41:21 DEBUG 06:41:21 DEBUG ] 06:41:21 DEBUG ] 06:41:21 DEBUG "value": [ 06:41:21 DEBUG "0" 06:41:21 DEBUG }, 06:41:21 DEBUG "data": { 06:41:21 DEBUG 1712814080, 06:41:21 DEBUG "metric": { 06:41:21 DEBUG 06:41:21 DEBUG } 06:41:21 DEBUG ] 06:41:21 DEBUG 06:41:21 DEBUG [...] 06:41:21 DEBUG } 06:41:21 DEBUG 1712814080, 06:41:21 DEBUG ] 06:41:21 DEBUG "value": [ 06:41:21 DEBUG "resultType": "vector", 06:41:21 DEBUG "0.00253333333333335" 06:41:21 DEBUG "pod": "ds-idrepo-2" 06:41:21 DEBUG 06:41:21 DEBUG "83023659.3" 06:41:21 DEBUG } 06:41:21 DEBUG } 06:41:21 DEBUG 06:41:21 DEBUG ] 06:41:21 DEBUG "245096" 06:41:21 DEBUG } 06:41:21 DEBUG 1712814080, 06:41:21 DEBUG "result": [ 06:41:21 DEBUG ] 06:41:21 DEBUG }, 06:41:21 DEBUG 06:41:21 DEBUG ] 06:41:21 DEBUG } 06:41:21 DEBUG ] 06:41:21 DEBUG 06:41:21 DEBUG } 06:41:21 DEBUG ] 06:41:21 DEBUG ] 06:41:21 DEBUG "0" 06:41:21 DEBUG { 06:41:21 DEBUG } 06:41:21 DEBUG "value": [ 06:41:21 DEBUG "pod": "ds-cts-1" 06:41:21 DEBUG } 06:41:21 DEBUG ] 06:41:21 DEBUG } 06:41:21 DEBUG 06:41:21 DEBUG } 06:41:21 DEBUG } 06:41:21 DEBUG } 06:41:21 DEBUG } 06:41:21 DEBUG ] 06:41:21 DEBUG ] 06:41:21 DEBUG 06:41:21 DEBUG }, 06:41:21 DEBUG } 06:41:21 DEBUG } 06:41:21 DEBUG "pod": "ds-idrepo-0" 06:41:21 DEBUG ] 06:41:21 DEBUG "metric": { 06:41:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:41:21 DEBUG } 06:41:21 DEBUG 06:41:21 DEBUG "value": [ 06:41:21 DEBUG } 06:41:21 DEBUG }, 06:41:21 DEBUG } 06:41:21 DEBUG } 06:41:21 DEBUG } 06:41:21 DEBUG } 06:41:21 DEBUG 06:41:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712814081" 06:41:21 DEBUG 1712814080, 06:41:21 DEBUG "value": [ 06:41:21 DEBUG }, 06:41:21 DEBUG "value": [ 06:41:21 DEBUG [...] 06:41:21 DEBUG "11779940.96641474" 06:41:21 DEBUG 1712814081, 06:41:21 DEBUG ] 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG ] 06:41:21 DEBUG "0" 06:41:21 DEBUG } 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG } 06:41:21 DEBUG ] 06:41:21 DEBUG } 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG ] 06:41:21 DEBUG } 06:41:21 DEBUG [...] 06:41:21 DEBUG "pod": "ds-cts-2" 06:41:21 DEBUG } 06:41:21 DEBUG ] 06:41:21 DEBUG 06:41:21 DEBUG }, 06:41:21 DEBUG } 06:41:21 DEBUG } 06:41:21 DEBUG 06:41:21 DEBUG "value": [ 06:41:21 DEBUG } 06:41:21 DEBUG 06:41:21 DEBUG 1712814081, 06:41:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:41:21 DEBUG "0" 06:41:21 DEBUG }, 06:41:21 DEBUG ] 06:41:21 DEBUG "value": [ 06:41:21 DEBUG } 06:41:21 DEBUG 1712814081, 06:41:21 DEBUG ] 06:41:21 DEBUG "1367.3333333333333" 06:41:21 DEBUG } 06:41:21 DEBUG ] 06:41:21 DEBUG } 06:41:21 DEBUG } 06:41:21 DEBUG ] 06:41:21 DEBUG } 06:41:21 DEBUG } 06:41:21 INFO [http_cmd]: http status code OK 06:41:21 DEBUG --- status code --- 06:41:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:21 DEBUG --- http response --- 06:41:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:41:21 DEBUG ----- output ----- 06:41:21 DEBUG { 06:41:21 DEBUG "status": "success", 06:41:21 DEBUG "data": { 06:41:21 DEBUG "resultType": "vector", 06:41:21 DEBUG "result": [ 06:41:21 DEBUG { 06:41:21 DEBUG "metric": { 06:41:21 DEBUG "pod": "ds-idrepo-2" 06:41:21 DEBUG }, 06:41:21 DEBUG "value": [ 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG [...] 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG "pod": "ds-cts-2" 06:41:21 DEBUG }, 06:41:21 DEBUG "value": [ 06:41:21 DEBUG 1712814081, 06:41:21 DEBUG "0" 06:41:21 DEBUG ] 06:41:21 DEBUG } 06:41:21 DEBUG ] 06:41:21 DEBUG } 06:41:21 DEBUG } 06:41:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814081" 06:41:21 INFO [http_cmd]: http status code OK 06:41:21 DEBUG --- status code --- 06:41:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:21 DEBUG --- http response --- 06:41:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:41:21 DEBUG ----- output ----- 06:41:21 DEBUG { 06:41:21 DEBUG "status": "success", 06:41:21 DEBUG "data": { 06:41:21 DEBUG "resultType": "vector", 06:41:21 DEBUG "result": [ 06:41:21 DEBUG { 06:41:21 DEBUG "metric": { 06:41:21 DEBUG "instance": "10.142.0.107:9100" 06:41:21 DEBUG }, 06:41:21 DEBUG "value": [ 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG [...] 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG "instance": "10.142.0.77:9100" 06:41:21 DEBUG }, 06:41:21 DEBUG "value": [ 06:41:21 DEBUG 1712814081, 06:41:21 DEBUG "83021073.43333334" 06:41:21 DEBUG ] 06:41:21 DEBUG } 06:41:21 DEBUG ] 06:41:21 DEBUG } 06:41:21 DEBUG } 06:41:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814081" 06:41:21 INFO [http_cmd]: http status code OK 06:41:21 DEBUG --- status code --- 06:41:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:21 DEBUG --- http response --- 06:41:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:41:21 DEBUG ----- output ----- 06:41:21 DEBUG { 06:41:21 DEBUG "status": "success", 06:41:21 DEBUG "data": { 06:41:21 DEBUG "resultType": "vector", 06:41:21 DEBUG "result": [ 06:41:21 DEBUG { 06:41:21 DEBUG "metric": { 06:41:21 DEBUG "pod": "ds-idrepo-2" 06:41:21 DEBUG }, 06:41:21 DEBUG "value": [ 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG [...] 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG "pod": "ds-cts-2" 06:41:21 DEBUG }, 06:41:21 DEBUG "value": [ 06:41:21 DEBUG 1712814081, 06:41:21 DEBUG "0" 06:41:21 DEBUG ] 06:41:21 DEBUG } 06:41:21 DEBUG ] 06:41:21 DEBUG } 06:41:21 DEBUG } 06:41:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712814081" 06:41:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814081" 06:41:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814081" 06:41:21 INFO [http_cmd]: http status code OK 06:41:21 DEBUG --- status code --- 06:41:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:21 DEBUG --- http response --- 06:41:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:41:21 DEBUG ----- output ----- 06:41:21 DEBUG { 06:41:21 DEBUG "status": "success", 06:41:21 DEBUG "data": { 06:41:21 DEBUG "resultType": "vector", 06:41:21 DEBUG "result": [ 06:41:21 DEBUG { 06:41:21 DEBUG "metric": { 06:41:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:41:21 DEBUG }, 06:41:21 DEBUG "value": [ 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG [...] 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:41:21 DEBUG }, 06:41:21 DEBUG "value": [ 06:41:21 DEBUG 1712814081, 06:41:21 DEBUG "1365.7666666666667" 06:41:21 DEBUG ] 06:41:21 DEBUG } 06:41:21 DEBUG ] 06:41:21 DEBUG } 06:41:21 DEBUG } 06:41:21 INFO [http_cmd]: http status code OK 06:41:21 DEBUG --- status code --- 06:41:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:21 DEBUG --- http response --- 06:41:21 INFO [http_cmd]: http status code OK 06:41:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:41:21 DEBUG --- status code --- 06:41:21 DEBUG ----- output ----- 06:41:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:21 DEBUG { 06:41:21 DEBUG --- http response --- 06:41:21 DEBUG "status": "success", 06:41:21 DEBUG "data": { 06:41:21 DEBUG "resultType": "vector", 06:41:21 DEBUG "result": [ 06:41:21 DEBUG { 06:41:21 DEBUG "metric": { 06:41:21 DEBUG "pod": "ds-idrepo-2" 06:41:21 DEBUG }, 06:41:21 DEBUG "value": [ 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG [...] 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG "pod": "ds-idrepo-0" 06:41:21 DEBUG }, 06:41:21 DEBUG "value": [ 06:41:21 DEBUG 1712814081, 06:41:21 DEBUG "0" 06:41:21 DEBUG ] 06:41:21 DEBUG } 06:41:21 DEBUG ] 06:41:21 DEBUG } 06:41:21 DEBUG } 06:41:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:41:21 DEBUG ----- output ----- 06:41:21 DEBUG { 06:41:21 DEBUG "status": "success", 06:41:21 DEBUG "data": { 06:41:21 DEBUG "resultType": "vector", 06:41:21 DEBUG "result": [ 06:41:21 DEBUG { 06:41:21 DEBUG "metric": { 06:41:21 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:41:21 DEBUG }, 06:41:21 DEBUG "value": [ 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG [...] 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 06:41:21 DEBUG }, 06:41:21 DEBUG "value": [ 06:41:21 DEBUG 1712814081, 06:41:21 DEBUG "13169459200" 06:41:21 DEBUG ] 06:41:21 DEBUG } 06:41:21 DEBUG ] 06:41:21 DEBUG } 06:41:21 DEBUG } 06:41:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814081" 06:41:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814081" 06:41:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712814081" 06:41:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814081" 06:41:21 INFO [http_cmd]: http status code OK 06:41:21 DEBUG --- status code --- 06:41:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:21 DEBUG --- http response --- 06:41:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:41:21 DEBUG ----- output ----- 06:41:21 DEBUG { 06:41:21 DEBUG "status": "success", 06:41:21 DEBUG "data": { 06:41:21 DEBUG "resultType": "vector", 06:41:21 DEBUG "result": [ 06:41:21 DEBUG { 06:41:21 DEBUG "metric": { 06:41:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:41:21 DEBUG }, 06:41:21 DEBUG "value": [ 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG [...] 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG "pod": "ds-cts-1" 06:41:21 DEBUG }, 06:41:21 DEBUG "value": [ 06:41:21 DEBUG 1712814081, 06:41:21 DEBUG "29864233.559231263" 06:41:21 DEBUG ] 06:41:21 DEBUG } 06:41:21 DEBUG ] 06:41:21 DEBUG } 06:41:21 DEBUG } 06:41:21 INFO [http_cmd]: http status code OK 06:41:21 DEBUG --- status code --- 06:41:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:21 DEBUG --- http response --- 06:41:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:41:21 DEBUG ----- output ----- 06:41:21 DEBUG { 06:41:21 DEBUG "status": "success", 06:41:21 DEBUG "data": { 06:41:21 DEBUG "resultType": "vector", 06:41:21 DEBUG "result": [ 06:41:21 DEBUG { 06:41:21 DEBUG "metric": { 06:41:21 DEBUG "instance": "10.142.0.107:9100" 06:41:21 DEBUG }, 06:41:21 DEBUG "value": [ 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG [...] 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG "instance": "10.142.0.77:9100" 06:41:21 DEBUG }, 06:41:21 DEBUG "value": [ 06:41:21 DEBUG 1712814081, 06:41:21 DEBUG "0.000666666666666534" 06:41:21 DEBUG ] 06:41:21 DEBUG } 06:41:21 DEBUG ] 06:41:21 DEBUG } 06:41:21 DEBUG } 06:41:21 INFO [http_cmd]: http status code OK 06:41:21 DEBUG --- status code --- 06:41:21 INFO [http_cmd]: http status code OK 06:41:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:21 DEBUG --- status code --- 06:41:21 DEBUG --- http response --- 06:41:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:41:21 DEBUG --- http response --- 06:41:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:41:21 DEBUG ----- output ----- 06:41:21 DEBUG { 06:41:21 DEBUG "status": "success", 06:41:21 DEBUG "data": { 06:41:21 DEBUG "resultType": "vector", 06:41:21 DEBUG "result": [ 06:41:21 DEBUG { 06:41:21 DEBUG "metric": { 06:41:21 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:41:21 DEBUG }, 06:41:21 DEBUG "value": [ 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG [...] 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:41:21 DEBUG }, 06:41:21 DEBUG "value": [ 06:41:21 DEBUG 1712814081, 06:41:21 DEBUG "4.1357544505881965" 06:41:21 DEBUG ] 06:41:21 DEBUG } 06:41:21 DEBUG ] 06:41:21 DEBUG } 06:41:21 DEBUG } 06:41:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:41:21 DEBUG ----- output ----- 06:41:21 DEBUG { 06:41:21 DEBUG "status": "success", 06:41:21 DEBUG "data": { 06:41:21 DEBUG "resultType": "vector", 06:41:21 DEBUG "result": [ 06:41:21 DEBUG { 06:41:21 DEBUG "metric": { 06:41:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:41:21 DEBUG }, 06:41:21 DEBUG "value": [ 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG [...] 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG 06:41:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:41:21 DEBUG }, 06:41:21 DEBUG "value": [ 06:41:21 DEBUG 1712814081, 06:41:21 DEBUG "1367.2" 06:41:21 DEBUG ] 06:41:21 DEBUG } 06:41:21 DEBUG ] 06:41:21 DEBUG } 06:41:21 DEBUG } 06:41:32 INFO 06:41:32 INFO [loop_until]: kubectl --namespace=xlou top pods 06:41:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:41:32 INFO [loop_until]: OK (rc = 0) 06:41:32 DEBUG --- stdout --- 06:41:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 8876m 8504Mi am-5b5bd8567b-8h2w4 9044m 8568Mi am-5b5bd8567b-qlkw6 7555m 8459Mi ds-cts-0 8316m 12629Mi ds-cts-1 8548m 12845Mi ds-cts-2 8531m 12726Mi ds-idrepo-0 11587m 5501Mi ds-idrepo-1 350m 5479Mi ds-idrepo-2 343m 5784Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 4m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 2m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4175m 1851Mi 06:41:32 DEBUG --- stderr --- 06:41:32 DEBUG 06:41:35 INFO 06:41:35 INFO [loop_until]: kubectl --namespace=xlou top node 06:41:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:41:35 INFO [loop_until]: OK (rc = 0) 06:41:35 DEBUG --- stdout --- 06:41:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 2731Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 60m 0% 2774Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7843m 26% 10119Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10772m 36% 2988Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9614m 32% 10090Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 143m 0% 3847Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9816m 32% 10188Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 408m 2% 6458Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11605m 73% 6194Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8643m 54% 13316Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 8208m 51% 13205Mi 22% gke-xlou-cdm-ds-32e4dcb1-j2hk 8595m 54% 13442Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 411m 2% 6158Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4257m 26% 5613Mi 9% 06:41:35 DEBUG --- stderr --- 06:41:35 DEBUG 06:42:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814141" 06:42:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814141" 06:42:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814141" 06:42:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712814141" 06:42:21 INFO [http_cmd]: http status code OK 06:42:21 INFO [http_cmd]: http status code OK 06:42:21 DEBUG --- status code --- 06:42:21 DEBUG --- status code --- 06:42:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712814141" 06:42:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814141" 06:42:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:21 DEBUG --- http response --- 06:42:21 DEBUG --- http response --- 06:42:21 INFO [http_cmd]: http status code OK 06:42:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:42:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814141" 06:42:21 DEBUG --- status code --- 06:42:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:42:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814141" 06:42:21 DEBUG ----- output ----- 06:42:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:21 DEBUG ----- output ----- 06:42:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814141" 06:42:21 DEBUG { 06:42:21 DEBUG --- http response --- 06:42:21 DEBUG { 06:42:21 DEBUG "status": "success", 06:42:21 DEBUG "status": "success", 06:42:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:42:21 DEBUG "data": { 06:42:21 DEBUG "data": { 06:42:21 DEBUG ----- output ----- 06:42:21 DEBUG "resultType": "vector", 06:42:21 DEBUG "resultType": "vector", 06:42:21 DEBUG { 06:42:21 DEBUG "result": [ 06:42:21 DEBUG "result": [ 06:42:21 DEBUG "status": "success", 06:42:21 DEBUG { 06:42:21 DEBUG { 06:42:21 DEBUG "data": { 06:42:21 DEBUG "metric": { 06:42:21 DEBUG "metric": { 06:42:21 DEBUG "resultType": "vector", 06:42:21 DEBUG "instance": "10.142.0.107:9100" 06:42:21 DEBUG "pod": "ds-idrepo-2" 06:42:21 DEBUG "result": [ 06:42:21 DEBUG }, 06:42:21 DEBUG }, 06:42:21 DEBUG { 06:42:21 DEBUG "value": [ 06:42:21 DEBUG "value": [ 06:42:21 DEBUG "metric": { 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG "pod": "ds-idrepo-2" 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG }, 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG "value": [ 06:42:21 DEBUG [...] 06:42:21 DEBUG [...] 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG [...] 06:42:21 DEBUG "instance": "10.142.0.77:9100" 06:42:21 DEBUG "pod": "ds-cts-2" 06:42:21 DEBUG 06:42:21 DEBUG }, 06:42:21 DEBUG }, 06:42:21 DEBUG 06:42:21 DEBUG "value": [ 06:42:21 DEBUG "value": [ 06:42:21 DEBUG 06:42:21 DEBUG 1712814141, 06:42:21 DEBUG 1712814141, 06:42:21 DEBUG "pod": "ds-idrepo-0" 06:42:21 DEBUG "82460658.66666667" 06:42:21 DEBUG "29.566666666666666" 06:42:21 DEBUG }, 06:42:21 DEBUG ] 06:42:21 DEBUG ] 06:42:21 DEBUG "value": [ 06:42:21 DEBUG } 06:42:21 DEBUG } 06:42:21 DEBUG 1712814141, 06:42:21 DEBUG ] 06:42:21 DEBUG ] 06:42:21 DEBUG "0" 06:42:21 DEBUG } 06:42:21 DEBUG } 06:42:21 DEBUG ] 06:42:21 DEBUG } 06:42:21 DEBUG } 06:42:21 DEBUG } 06:42:21 DEBUG ] 06:42:21 DEBUG } 06:42:21 DEBUG } 06:42:21 INFO [http_cmd]: http status code OK 06:42:21 DEBUG --- status code --- 06:42:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:21 DEBUG --- http response --- 06:42:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:42:21 DEBUG ----- output ----- 06:42:21 DEBUG { 06:42:21 DEBUG "status": "success", 06:42:21 DEBUG "data": { 06:42:21 DEBUG "resultType": "vector", 06:42:21 DEBUG "result": [ 06:42:21 DEBUG { 06:42:21 DEBUG "metric": { 06:42:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:42:21 DEBUG }, 06:42:21 DEBUG "value": [ 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG [...] 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:42:21 DEBUG }, 06:42:21 DEBUG "value": [ 06:42:21 DEBUG 1712814141, 06:42:21 DEBUG "0.021" 06:42:21 DEBUG ] 06:42:21 DEBUG } 06:42:21 DEBUG ] 06:42:21 DEBUG } 06:42:21 DEBUG } 06:42:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814141" 06:42:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814141" 06:42:21 INFO [http_cmd]: http status code OK 06:42:21 DEBUG --- status code --- 06:42:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:21 DEBUG --- http response --- 06:42:21 INFO [http_cmd]: http status code OK 06:42:21 INFO [http_cmd]: http status code OK 06:42:21 INFO [http_cmd]: http status code OK 06:42:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:42:21 DEBUG --- status code --- 06:42:21 DEBUG --- status code --- 06:42:21 DEBUG --- status code --- 06:42:21 DEBUG ----- output ----- 06:42:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:21 DEBUG { 06:42:21 DEBUG --- http response --- 06:42:21 DEBUG --- http response --- 06:42:21 INFO [http_cmd]: http status code OK 06:42:21 DEBUG --- http response --- 06:42:21 DEBUG "status": "success", 06:42:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:42:21 DEBUG --- status code --- 06:42:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:42:21 DEBUG "data": { 06:42:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:42:21 DEBUG ----- output ----- 06:42:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712814141" 06:42:21 DEBUG ----- output ----- 06:42:21 DEBUG "resultType": "vector", 06:42:21 DEBUG ----- output ----- 06:42:21 DEBUG { 06:42:21 DEBUG --- http response --- 06:42:21 DEBUG { 06:42:21 DEBUG "result": [ 06:42:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814141" 06:42:21 INFO [http_cmd]: http status code OK 06:42:21 DEBUG { 06:42:21 DEBUG "status": "success", 06:42:21 DEBUG "status": "success", 06:42:21 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:42:21 DEBUG { 06:42:21 DEBUG --- status code --- 06:42:21 DEBUG "status": "success", 06:42:21 DEBUG "data": { 06:42:21 DEBUG "data": { 06:42:21 DEBUG "metric": { 06:42:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:21 DEBUG "data": { 06:42:21 DEBUG "resultType": "vector", 06:42:21 DEBUG "resultType": "vector", 06:42:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:42:21 DEBUG --- http response --- 06:42:21 DEBUG "resultType": "vector", 06:42:21 DEBUG "result": [ 06:42:21 DEBUG "result": [ 06:42:21 DEBUG }, 06:42:21 INFO [http_cmd]: http status code OK 06:42:21 DEBUG "result": [ 06:42:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:42:21 DEBUG { 06:42:21 DEBUG { 06:42:21 DEBUG "value": [ 06:42:21 DEBUG --- status code --- 06:42:21 DEBUG { 06:42:21 DEBUG ----- output ----- 06:42:21 DEBUG "metric": { 06:42:21 DEBUG "metric": { 06:42:21 DEBUG 06:42:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:21 DEBUG "metric": { 06:42:21 DEBUG { 06:42:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:42:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:42:21 DEBUG 06:42:21 DEBUG --- http response --- 06:42:21 DEBUG "pod": "ds-idrepo-2" 06:42:21 DEBUG "status": "success", 06:42:21 DEBUG }, 06:42:21 DEBUG }, 06:42:21 DEBUG 06:42:21 DEBUG }, 06:42:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:42:21 DEBUG "data": { 06:42:21 DEBUG "value": [ 06:42:21 DEBUG "value": [ 06:42:21 DEBUG [...] 06:42:21 DEBUG "value": [ 06:42:21 DEBUG ----- output ----- 06:42:21 DEBUG "resultType": "vector", 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG { 06:42:21 DEBUG "result": [ 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG "status": "success", 06:42:21 DEBUG { 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 ERROR [http_cmd]: ERROR 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG "data": { 06:42:21 DEBUG "metric": { 06:42:21 DEBUG [...] 06:42:21 DEBUG [...] 06:42:21 ERROR --- status code --- 06:42:21 DEBUG "pod": "ds-cts-1" 06:42:21 DEBUG [...] 06:42:21 DEBUG "resultType": "vector", 06:42:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:42:21 DEBUG }, 06:42:21 DEBUG 06:42:21 DEBUG "result": [ 06:42:21 DEBUG }, 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 ERROR --- http response --- 06:42:21 DEBUG "value": [ 06:42:21 DEBUG 06:42:21 DEBUG { 06:42:21 DEBUG "value": [ 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 1712814141, 06:42:21 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:42:21 DEBUG 06:42:21 DEBUG "metric": { 06:42:21 DEBUG 06:42:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:42:21 DEBUG "pod": "ds-cts-1" 06:42:21 DEBUG "15697577.087053165" 06:42:21 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:42:21 DEBUG "pod": "ds-cts-2" 06:42:21 DEBUG "pod": "ds-idrepo-2" 06:42:21 DEBUG 06:42:21 DEBUG }, 06:42:21 DEBUG }, 06:42:21 DEBUG ] 06:42:21 DEBUG }, 06:42:21 DEBUG }, 06:42:21 DEBUG 06:42:21 DEBUG "value": [ 06:42:21 DEBUG "value": [ 06:42:21 DEBUG } 06:42:21 DEBUG "value": [ 06:42:21 DEBUG "value": [ 06:42:21 INFO [http_cmd]: http status code OK 06:42:21 DEBUG [...] 06:42:21 DEBUG 1712814141, 06:42:21 DEBUG 1712814141, 06:42:21 DEBUG ] 06:42:21 DEBUG 1712814141, 06:42:21 DEBUG 06:42:21 DEBUG --- status code --- 06:42:21 DEBUG 06:42:21 DEBUG "1372.4" 06:42:21 DEBUG "8.825038020223746" 06:42:21 DEBUG } 06:42:21 DEBUG "0" 06:42:21 DEBUG 06:42:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:21 DEBUG 06:42:21 DEBUG ] 06:42:21 DEBUG ] 06:42:21 DEBUG } 06:42:21 DEBUG ] 06:42:21 DEBUG 06:42:21 DEBUG --- http response --- 06:42:21 DEBUG 06:42:21 DEBUG } 06:42:21 DEBUG } 06:42:21 DEBUG } 06:42:21 DEBUG [...] 06:42:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:42:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:42:21 DEBUG ] 06:42:21 DEBUG ] 06:42:21 DEBUG ] 06:42:21 DEBUG 06:42:21 DEBUG ----- output ----- 06:42:21 DEBUG }, 06:42:21 DEBUG } 06:42:21 DEBUG } 06:42:21 DEBUG } 06:42:21 DEBUG 06:42:21 DEBUG { 06:42:21 DEBUG "value": [ 06:42:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814141" 06:42:21 DEBUG } 06:42:21 DEBUG } 06:42:21 DEBUG } 06:42:21 DEBUG 06:42:21 DEBUG "status": "success", 06:42:21 DEBUG 1712814141, 06:42:21 DEBUG "pod": "ds-cts-2" 06:42:21 DEBUG "data": { 06:42:21 DEBUG "0" 06:42:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814141" 06:42:21 DEBUG }, 06:42:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814141" 06:42:21 DEBUG "resultType": "vector", 06:42:21 DEBUG ] 06:42:21 DEBUG "value": [ 06:42:21 DEBUG "result": [ 06:42:21 DEBUG } 06:42:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814141" 06:42:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814141" 06:42:21 DEBUG 1712814141, 06:42:21 DEBUG { 06:42:21 DEBUG ] 06:42:21 DEBUG "0.000030149444444444445" 06:42:21 DEBUG "metric": { 06:42:21 DEBUG } 06:42:21 DEBUG ] 06:42:21 DEBUG "instance": "10.142.0.107:9100" 06:42:21 DEBUG } 06:42:21 DEBUG } 06:42:21 DEBUG }, 06:42:21 DEBUG ] 06:42:21 DEBUG "value": [ 06:42:21 DEBUG } 06:42:21 DEBUG 06:42:21 DEBUG } 06:42:21 INFO [http_cmd]: http status code OK 06:42:21 DEBUG 06:42:21 DEBUG --- status code --- 06:42:21 DEBUG 06:42:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:21 DEBUG [...] 06:42:21 DEBUG --- http response --- 06:42:21 DEBUG 06:42:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:42:21 DEBUG 06:42:21 DEBUG ----- output ----- 06:42:21 DEBUG 06:42:21 DEBUG { 06:42:21 INFO [http_cmd]: http status code OK 06:42:21 DEBUG "instance": "10.142.0.77:9100" 06:42:21 DEBUG "status": "success", 06:42:21 DEBUG --- status code --- 06:42:21 DEBUG }, 06:42:21 DEBUG "data": { 06:42:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:21 DEBUG "value": [ 06:42:21 DEBUG "resultType": "vector", 06:42:21 DEBUG --- http response --- 06:42:21 DEBUG 1712814141, 06:42:21 INFO [http_cmd]: http status code OK 06:42:21 INFO [http_cmd]: http status code OK 06:42:21 DEBUG "result": [ 06:42:21 DEBUG "52949793792" 06:42:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:42:21 DEBUG --- status code --- 06:42:21 DEBUG --- status code --- 06:42:21 DEBUG { 06:42:21 DEBUG ] 06:42:21 DEBUG ----- output ----- 06:42:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:21 DEBUG "metric": { 06:42:21 DEBUG } 06:42:21 DEBUG { 06:42:21 DEBUG --- http response --- 06:42:21 DEBUG --- http response --- 06:42:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:42:21 DEBUG ] 06:42:21 DEBUG "status": "success", 06:42:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:42:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:42:21 DEBUG }, 06:42:21 INFO [http_cmd]: http status code OK 06:42:21 DEBUG } 06:42:21 DEBUG "data": { 06:42:21 DEBUG ----- output ----- 06:42:21 DEBUG ----- output ----- 06:42:21 DEBUG "value": [ 06:42:21 DEBUG --- status code --- 06:42:21 DEBUG } 06:42:21 DEBUG "resultType": "vector", 06:42:21 DEBUG { 06:42:21 DEBUG { 06:42:21 DEBUG 06:42:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:21 DEBUG "result": [ 06:42:21 DEBUG "status": "success", 06:42:21 DEBUG "status": "success", 06:42:21 DEBUG 06:42:21 DEBUG --- http response --- 06:42:21 DEBUG { 06:42:21 DEBUG "data": { 06:42:21 DEBUG "data": { 06:42:21 DEBUG 06:42:21 DEBUG "metric": { 06:42:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:42:21 DEBUG "resultType": "vector", 06:42:21 DEBUG "resultType": "vector", 06:42:21 DEBUG [...] 06:42:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:42:21 DEBUG ----- output ----- 06:42:21 DEBUG "result": [ 06:42:21 DEBUG "result": [ 06:42:21 DEBUG 06:42:21 DEBUG }, 06:42:21 DEBUG { 06:42:21 DEBUG { 06:42:21 DEBUG { 06:42:21 DEBUG 06:42:21 DEBUG "value": [ 06:42:21 DEBUG "status": "success", 06:42:21 DEBUG "metric": { 06:42:21 DEBUG "metric": { 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG "data": { 06:42:21 DEBUG "instance": "10.142.0.107:9100" 06:42:21 DEBUG "pod": "ds-idrepo-2" 06:42:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:42:21 DEBUG 06:42:21 DEBUG "resultType": "vector", 06:42:21 DEBUG }, 06:42:21 DEBUG }, 06:42:21 DEBUG }, 06:42:21 DEBUG 06:42:21 DEBUG "result": [ 06:42:21 DEBUG "value": [ 06:42:21 DEBUG "value": [ 06:42:21 DEBUG "value": [ 06:42:21 DEBUG [...] 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 1712814141, 06:42:21 DEBUG 06:42:21 DEBUG { 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG "1357.2333333333333" 06:42:21 DEBUG 06:42:21 DEBUG "metric": { 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG ] 06:42:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:42:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:42:21 DEBUG [...] 06:42:21 DEBUG [...] 06:42:21 DEBUG } 06:42:21 DEBUG }, 06:42:21 DEBUG }, 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG ] 06:42:21 DEBUG "value": [ 06:42:21 DEBUG "value": [ 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG } 06:42:21 DEBUG 1712814141, 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG } 06:42:21 DEBUG "244988.5" 06:42:21 DEBUG 06:42:21 DEBUG "instance": "10.142.0.77:9100" 06:42:21 DEBUG "pod": "ds-idrepo-0" 06:42:21 DEBUG ] 06:42:21 DEBUG 06:42:21 DEBUG }, 06:42:21 DEBUG }, 06:42:21 DEBUG } 06:42:21 DEBUG [...] 06:42:21 DEBUG "value": [ 06:42:21 DEBUG "value": [ 06:42:21 DEBUG ] 06:42:21 DEBUG 06:42:21 DEBUG 1712814141, 06:42:21 DEBUG 1712814141, 06:42:21 DEBUG } 06:42:21 DEBUG 06:42:21 DEBUG "0.0024333333333331096" 06:42:21 DEBUG "0" 06:42:21 DEBUG } 06:42:21 DEBUG 06:42:21 DEBUG ] 06:42:21 DEBUG ] 06:42:21 DEBUG "pod": "ds-cts-1" 06:42:21 DEBUG } 06:42:21 DEBUG } 06:42:21 DEBUG }, 06:42:21 DEBUG ] 06:42:21 DEBUG ] 06:42:21 DEBUG "value": [ 06:42:21 DEBUG } 06:42:21 DEBUG } 06:42:21 DEBUG 1712814141, 06:42:21 DEBUG } 06:42:21 DEBUG } 06:42:21 DEBUG "33145913.201909415" 06:42:21 DEBUG ] 06:42:21 DEBUG } 06:42:21 DEBUG ] 06:42:21 DEBUG } 06:42:21 DEBUG } 06:42:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814141" 06:42:21 INFO [http_cmd]: http status code OK 06:42:21 DEBUG --- status code --- 06:42:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:21 DEBUG --- http response --- 06:42:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:42:21 DEBUG ----- output ----- 06:42:21 DEBUG { 06:42:21 DEBUG "status": "success", 06:42:21 DEBUG "data": { 06:42:21 DEBUG "resultType": "vector", 06:42:21 DEBUG "result": [ 06:42:21 DEBUG { 06:42:21 DEBUG "metric": { 06:42:21 DEBUG "instance": "10.142.0.107:9100" 06:42:21 DEBUG }, 06:42:21 DEBUG "value": [ 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG [...] 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG "instance": "10.142.0.77:9100" 06:42:21 DEBUG }, 06:42:21 DEBUG "value": [ 06:42:21 DEBUG 1712814141, 06:42:21 DEBUG "82458325.13333333" 06:42:21 DEBUG ] 06:42:21 DEBUG } 06:42:21 DEBUG ] 06:42:21 DEBUG } 06:42:21 DEBUG } 06:42:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814141" 06:42:21 INFO [http_cmd]: http status code OK 06:42:21 DEBUG --- status code --- 06:42:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:21 DEBUG --- http response --- 06:42:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:42:21 DEBUG ----- output ----- 06:42:21 DEBUG { 06:42:21 DEBUG "status": "success", 06:42:21 DEBUG "data": { 06:42:21 DEBUG "resultType": "vector", 06:42:21 DEBUG "result": [ 06:42:21 DEBUG { 06:42:21 DEBUG "metric": { 06:42:21 DEBUG "pod": "ds-idrepo-2" 06:42:21 DEBUG }, 06:42:21 DEBUG "value": [ 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG [...] 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG "pod": "ds-cts-2" 06:42:21 DEBUG }, 06:42:21 DEBUG "value": [ 06:42:21 DEBUG 1712814141, 06:42:21 DEBUG "0" 06:42:21 DEBUG ] 06:42:21 DEBUG } 06:42:21 DEBUG ] 06:42:21 DEBUG } 06:42:21 DEBUG } 06:42:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814141" 06:42:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712814141" 06:42:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814141" 06:42:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814141" 06:42:21 INFO [http_cmd]: http status code OK 06:42:21 DEBUG --- status code --- 06:42:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:21 DEBUG --- http response --- 06:42:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:42:21 DEBUG ----- output ----- 06:42:21 DEBUG { 06:42:21 DEBUG "status": "success", 06:42:21 DEBUG "data": { 06:42:21 DEBUG "resultType": "vector", 06:42:21 DEBUG "result": [ 06:42:21 DEBUG { 06:42:21 DEBUG "metric": { 06:42:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:42:21 DEBUG }, 06:42:21 DEBUG "value": [ 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG [...] 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:42:21 DEBUG }, 06:42:21 DEBUG "value": [ 06:42:21 DEBUG 1712814141, 06:42:21 DEBUG "1360.8333333333333" 06:42:21 DEBUG ] 06:42:21 DEBUG } 06:42:21 DEBUG ] 06:42:21 DEBUG } 06:42:21 DEBUG } 06:42:21 INFO [http_cmd]: http status code OK 06:42:21 DEBUG --- status code --- 06:42:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:21 DEBUG --- http response --- 06:42:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:42:21 DEBUG ----- output ----- 06:42:21 DEBUG { 06:42:21 DEBUG "status": "success", 06:42:21 DEBUG "data": { 06:42:21 DEBUG "resultType": "vector", 06:42:21 DEBUG "result": [ 06:42:21 DEBUG { 06:42:21 DEBUG "metric": { 06:42:21 DEBUG "pod": "ds-idrepo-2" 06:42:21 DEBUG }, 06:42:21 DEBUG "value": [ 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG [...] 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG "pod": "ds-idrepo-0" 06:42:21 DEBUG }, 06:42:21 DEBUG "value": [ 06:42:21 DEBUG 1712814141, 06:42:21 DEBUG "0" 06:42:21 DEBUG ] 06:42:21 DEBUG } 06:42:21 DEBUG ] 06:42:21 DEBUG } 06:42:21 DEBUG } 06:42:21 INFO [http_cmd]: http status code OK 06:42:21 DEBUG --- status code --- 06:42:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:21 DEBUG --- http response --- 06:42:21 INFO [http_cmd]: http status code OK 06:42:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:42:21 DEBUG --- status code --- 06:42:21 DEBUG ----- output ----- 06:42:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:21 DEBUG { 06:42:21 DEBUG --- http response --- 06:42:21 DEBUG "status": "success", 06:42:21 DEBUG "data": { 06:42:21 DEBUG "resultType": "vector", 06:42:21 DEBUG "result": [ 06:42:21 DEBUG { 06:42:21 DEBUG "metric": { 06:42:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:42:21 DEBUG }, 06:42:21 DEBUG "value": [ 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG [...] 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG "pod": "ds-cts-1" 06:42:21 DEBUG }, 06:42:21 DEBUG "value": [ 06:42:21 DEBUG 1712814141, 06:42:21 DEBUG "36433250.59213643" 06:42:21 DEBUG ] 06:42:21 DEBUG } 06:42:21 DEBUG ] 06:42:21 DEBUG } 06:42:21 DEBUG } 06:42:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:42:21 DEBUG ----- output ----- 06:42:21 DEBUG { 06:42:21 DEBUG "status": "success", 06:42:21 DEBUG "data": { 06:42:21 DEBUG "resultType": "vector", 06:42:21 DEBUG "result": [ 06:42:21 DEBUG { 06:42:21 DEBUG "metric": { 06:42:21 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:42:21 DEBUG }, 06:42:21 DEBUG "value": [ 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG [...] 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 06:42:21 DEBUG }, 06:42:21 DEBUG "value": [ 06:42:21 DEBUG 1712814141, 06:42:21 DEBUG "13686480896" 06:42:21 DEBUG ] 06:42:21 DEBUG } 06:42:21 DEBUG ] 06:42:21 DEBUG } 06:42:21 DEBUG } 06:42:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814141" 06:42:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712814141" 06:42:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814141" 06:42:21 INFO [http_cmd]: http status code OK 06:42:21 DEBUG --- status code --- 06:42:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:21 DEBUG --- http response --- 06:42:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:42:21 DEBUG ----- output ----- 06:42:21 DEBUG { 06:42:21 DEBUG "status": "success", 06:42:21 DEBUG "data": { 06:42:21 DEBUG "resultType": "vector", 06:42:21 DEBUG "result": [ 06:42:21 DEBUG { 06:42:21 DEBUG "metric": { 06:42:21 DEBUG "instance": "10.142.0.107:9100" 06:42:21 DEBUG }, 06:42:21 DEBUG "value": [ 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG [...] 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG "instance": "10.142.0.77:9100" 06:42:21 DEBUG }, 06:42:21 DEBUG "value": [ 06:42:21 DEBUG 1712814141, 06:42:21 DEBUG "0" 06:42:21 DEBUG ] 06:42:21 DEBUG } 06:42:21 DEBUG ] 06:42:21 DEBUG } 06:42:21 DEBUG } 06:42:21 INFO [http_cmd]: http status code OK 06:42:21 DEBUG --- status code --- 06:42:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:21 DEBUG --- http response --- 06:42:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:42:21 DEBUG ----- output ----- 06:42:21 DEBUG { 06:42:21 DEBUG "status": "success", 06:42:21 DEBUG "data": { 06:42:21 DEBUG "resultType": "vector", 06:42:21 DEBUG "result": [ 06:42:21 DEBUG { 06:42:21 DEBUG "metric": { 06:42:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:42:21 DEBUG }, 06:42:21 DEBUG "value": [ 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG [...] 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:42:21 DEBUG }, 06:42:21 DEBUG "value": [ 06:42:21 DEBUG 1712814141, 06:42:21 DEBUG "1356.5666666666666" 06:42:21 DEBUG ] 06:42:21 DEBUG } 06:42:21 DEBUG ] 06:42:21 DEBUG } 06:42:21 DEBUG } 06:42:21 INFO [http_cmd]: http status code OK 06:42:21 DEBUG --- status code --- 06:42:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:42:21 DEBUG --- http response --- 06:42:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:42:21 DEBUG ----- output ----- 06:42:21 DEBUG { 06:42:21 DEBUG "status": "success", 06:42:21 DEBUG "data": { 06:42:21 DEBUG "resultType": "vector", 06:42:21 DEBUG "result": [ 06:42:21 DEBUG { 06:42:21 DEBUG "metric": { 06:42:21 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:42:21 DEBUG }, 06:42:21 DEBUG "value": [ 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG [...] 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG 06:42:21 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:42:21 DEBUG }, 06:42:21 DEBUG "value": [ 06:42:21 DEBUG 1712814141, 06:42:21 DEBUG "4.127779487126578" 06:42:21 DEBUG ] 06:42:21 DEBUG } 06:42:21 DEBUG ] 06:42:21 DEBUG } 06:42:21 DEBUG } 06:42:32 INFO 06:42:32 INFO [loop_until]: kubectl --namespace=xlou top pods 06:42:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:42:32 INFO [loop_until]: OK (rc = 0) 06:42:32 DEBUG --- stdout --- 06:42:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 8901m 8502Mi am-5b5bd8567b-8h2w4 8482m 8571Mi am-5b5bd8567b-qlkw6 7515m 8460Mi ds-cts-0 8085m 13279Mi ds-cts-1 8329m 13399Mi ds-cts-2 8488m 13310Mi ds-idrepo-0 11528m 5502Mi ds-idrepo-1 348m 5479Mi ds-idrepo-2 364m 5783Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 3m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 4m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4166m 1853Mi 06:42:32 DEBUG --- stderr --- 06:42:32 DEBUG 06:42:35 INFO 06:42:35 INFO [loop_until]: kubectl --namespace=xlou top node 06:42:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:42:36 INFO [loop_until]: OK (rc = 0) 06:42:36 DEBUG --- stdout --- 06:42:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 2735Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 59m 0% 2763Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7814m 26% 10115Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10755m 36% 2983Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9535m 31% 10083Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 143m 0% 3847Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9647m 32% 10200Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 424m 2% 6461Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11530m 72% 6194Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8449m 53% 13920Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 8232m 51% 13819Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 8682m 54% 14017Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 412m 2% 6157Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4106m 25% 5615Mi 9% 06:42:36 DEBUG --- stderr --- 06:42:36 DEBUG 06:43:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814201" 06:43:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814201" 06:43:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814201" 06:43:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814201" 06:43:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814201" 06:43:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712814201" 06:43:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814201" 06:43:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814201" 06:43:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814201" 06:43:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814201" 06:43:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712814201" 06:43:21 INFO [http_cmd]: http status code OK 06:43:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712814201" 06:43:21 DEBUG --- status code --- 06:43:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:21 DEBUG --- http response --- 06:43:21 INFO [http_cmd]: http status code OK 06:43:21 DEBUG --- status code --- 06:43:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:21 DEBUG --- http response --- 06:43:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814201" 06:43:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:43:21 INFO [http_cmd]: http status code OK 06:43:21 INFO [http_cmd]: http status code OK 06:43:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:43:21 INFO [http_cmd]: http status code OK 06:43:21 INFO [http_cmd]: http status code OK 06:43:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814201" 06:43:21 DEBUG ----- output ----- 06:43:21 DEBUG --- status code --- 06:43:21 INFO [http_cmd]: http status code OK 06:43:21 DEBUG --- status code --- 06:43:21 INFO [http_cmd]: http status code OK 06:43:21 INFO [http_cmd]: http status code OK 06:43:21 DEBUG ----- output ----- 06:43:21 INFO [http_cmd]: http status code OK 06:43:21 DEBUG --- status code --- 06:43:21 ERROR [http_cmd]: ERROR 06:43:21 DEBUG --- status code --- 06:43:21 DEBUG { 06:43:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:21 DEBUG --- status code --- 06:43:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:21 DEBUG --- status code --- 06:43:21 DEBUG --- status code --- 06:43:21 DEBUG { 06:43:21 DEBUG --- status code --- 06:43:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:21 ERROR --- status code --- 06:43:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:21 DEBUG "status": "success", 06:43:21 DEBUG --- http response --- 06:43:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:21 DEBUG --- http response --- 06:43:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:21 DEBUG "status": "success", 06:43:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:21 DEBUG --- http response --- 06:43:21 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:43:21 DEBUG --- http response --- 06:43:21 DEBUG "data": { 06:43:21 DEBUG --- http response --- 06:43:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:43:21 INFO [http_cmd]: http status code OK 06:43:21 INFO [http_cmd]: http status code OK 06:43:21 DEBUG --- http response --- 06:43:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:43:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814201" 06:43:21 DEBUG --- http response --- 06:43:21 DEBUG "data": { 06:43:21 DEBUG --- http response --- 06:43:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:43:21 ERROR --- http response --- 06:43:21 DEBUG "resultType": "vector", 06:43:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:43:21 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:43:21 DEBUG ----- output ----- 06:43:21 DEBUG --- status code --- 06:43:21 DEBUG --- status code --- 06:43:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:43:21 DEBUG ----- output ----- 06:43:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:43:21 DEBUG "resultType": "vector", 06:43:21 DEBUG ----- output ----- 06:43:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:43:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814201" 06:43:21 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:43:21 DEBUG "result": [ 06:43:21 DEBUG ----- output ----- 06:43:21 DEBUG { 06:43:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:21 INFO [http_cmd]: http status code OK 06:43:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:21 DEBUG ----- output ----- 06:43:21 DEBUG { 06:43:21 DEBUG ----- output ----- 06:43:21 DEBUG "result": [ 06:43:21 DEBUG { 06:43:21 DEBUG ----- output ----- 06:43:21 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:43:21 DEBUG { 06:43:21 DEBUG { 06:43:21 DEBUG "status": "success", 06:43:21 DEBUG --- http response --- 06:43:21 DEBUG --- status code --- 06:43:21 DEBUG --- http response --- 06:43:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814201" 06:43:21 DEBUG { 06:43:21 DEBUG "status": "success", 06:43:21 DEBUG { 06:43:21 DEBUG { 06:43:21 DEBUG "status": "success", 06:43:21 DEBUG { 06:43:21 DEBUG "metric": { 06:43:21 DEBUG "status": "success", 06:43:21 DEBUG "data": { 06:43:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:43:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814201" 06:43:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:43:21 DEBUG "status": "success", 06:43:21 DEBUG "data": { 06:43:21 DEBUG "status": "success", 06:43:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814201" 06:43:21 DEBUG "metric": { 06:43:21 DEBUG "data": { 06:43:21 DEBUG "status": "success", 06:43:21 DEBUG "pod": "ds-idrepo-2" 06:43:21 DEBUG "data": { 06:43:21 DEBUG "resultType": "vector", 06:43:21 DEBUG --- http response --- 06:43:21 DEBUG ----- output ----- 06:43:21 DEBUG ----- output ----- 06:43:21 INFO [http_cmd]: http status code OK 06:43:21 DEBUG "data": { 06:43:21 DEBUG "resultType": "vector", 06:43:21 DEBUG "data": { 06:43:21 DEBUG "instance": "10.142.0.107:9100" 06:43:21 DEBUG "resultType": "vector", 06:43:21 DEBUG "data": { 06:43:21 DEBUG }, 06:43:21 DEBUG "resultType": "vector", 06:43:21 DEBUG "result": [ 06:43:21 DEBUG { 06:43:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:43:21 DEBUG { 06:43:21 DEBUG --- status code --- 06:43:21 DEBUG "resultType": "vector", 06:43:21 DEBUG "result": [ 06:43:21 DEBUG "resultType": "vector", 06:43:21 INFO [http_cmd]: http status code OK 06:43:21 DEBUG }, 06:43:21 DEBUG "result": [ 06:43:21 DEBUG "resultType": "vector", 06:43:21 DEBUG "value": [ 06:43:21 DEBUG "result": [ 06:43:21 DEBUG { 06:43:21 DEBUG "status": "success", 06:43:21 DEBUG ----- output ----- 06:43:21 DEBUG "status": "success", 06:43:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:21 DEBUG "result": [ 06:43:21 DEBUG { 06:43:21 DEBUG "result": [ 06:43:21 DEBUG --- status code --- 06:43:21 DEBUG "value": [ 06:43:21 DEBUG { 06:43:21 DEBUG "result": [ 06:43:21 DEBUG 06:43:21 DEBUG { 06:43:21 DEBUG "metric": { 06:43:21 DEBUG "data": { 06:43:21 DEBUG { 06:43:21 DEBUG "data": { 06:43:21 DEBUG --- http response --- 06:43:21 DEBUG { 06:43:21 DEBUG "metric": { 06:43:21 DEBUG { 06:43:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:21 DEBUG 06:43:21 DEBUG "metric": { 06:43:21 DEBUG { 06:43:21 DEBUG 06:43:21 DEBUG "metric": { 06:43:21 INFO [http_cmd]: http status code OK 06:43:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:43:21 DEBUG "resultType": "vector", 06:43:21 DEBUG "status": "success", 06:43:21 DEBUG "resultType": "vector", 06:43:21 DEBUG "metric": { 06:43:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:43:21 DEBUG "instance": "10.142.0.107:9100" 06:43:21 DEBUG "metric": { 06:43:21 DEBUG --- http response --- 06:43:21 DEBUG 06:43:21 DEBUG "pod": "ds-idrepo-2" 06:43:21 DEBUG "metric": { 06:43:21 DEBUG 06:43:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:43:21 DEBUG --- status code --- 06:43:21 DEBUG }, 06:43:21 DEBUG "result": [ 06:43:21 DEBUG "data": { 06:43:21 DEBUG "result": [ 06:43:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:43:21 INFO [http_cmd]: http status code OK 06:43:21 DEBUG ----- output ----- 06:43:21 INFO [http_cmd]: http status code OK 06:43:21 DEBUG }, 06:43:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:43:21 DEBUG 06:43:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:43:21 DEBUG }, 06:43:21 DEBUG "instance": "10.142.0.107:9100" 06:43:21 DEBUG [...] 06:43:21 DEBUG }, 06:43:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:21 DEBUG "value": [ 06:43:21 DEBUG { 06:43:21 DEBUG "resultType": "vector", 06:43:21 DEBUG { 06:43:21 DEBUG }, 06:43:21 DEBUG --- status code --- 06:43:21 DEBUG { 06:43:21 DEBUG --- status code --- 06:43:21 DEBUG "value": [ 06:43:21 DEBUG }, 06:43:21 DEBUG [...] 06:43:21 DEBUG ----- output ----- 06:43:21 DEBUG "value": [ 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG }, 06:43:21 DEBUG --- http response --- 06:43:21 DEBUG 06:43:21 DEBUG "metric": { 06:43:21 DEBUG "result": [ 06:43:21 DEBUG "metric": { 06:43:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:43:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:21 DEBUG "status": "success", 06:43:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:21 DEBUG 06:43:21 DEBUG "value": [ 06:43:21 DEBUG 06:43:21 DEBUG { 06:43:21 DEBUG 06:43:21 DEBUG "value": [ 06:43:21 DEBUG 06:43:21 DEBUG [...] 06:43:21 DEBUG 06:43:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:43:21 DEBUG "pod": "ds-idrepo-2" 06:43:21 DEBUG { 06:43:21 DEBUG "value": [ 06:43:21 DEBUG }, 06:43:21 DEBUG --- http response --- 06:43:21 DEBUG "data": { 06:43:21 DEBUG --- http response --- 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG "status": "success", 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG "value": [ 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG ----- output ----- 06:43:21 DEBUG }, 06:43:21 DEBUG "metric": { 06:43:21 DEBUG 06:43:21 DEBUG "value": [ 06:43:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:43:21 DEBUG "resultType": "vector", 06:43:21 DEBUG 06:43:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG "data": { 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG { 06:43:21 DEBUG "value": [ 06:43:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:43:21 DEBUG }, 06:43:21 DEBUG 06:43:21 DEBUG ----- output ----- 06:43:21 DEBUG "result": [ 06:43:21 DEBUG [...] 06:43:21 DEBUG ----- output ----- 06:43:21 DEBUG { 06:43:21 DEBUG "instance": "10.142.0.77:9100" 06:43:21 DEBUG "resultType": "vector", 06:43:21 DEBUG "pod": "ds-idrepo-0" 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG [...] 06:43:21 DEBUG "pod": "ds-cts-2" 06:43:21 DEBUG "status": "success", 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG "value": [ 06:43:21 DEBUG 06:43:21 DEBUG { 06:43:21 DEBUG { 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG "status": "success", 06:43:21 DEBUG }, 06:43:21 DEBUG "result": [ 06:43:21 DEBUG }, 06:43:21 DEBUG [...] 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG }, 06:43:21 DEBUG "data": { 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG [...] 06:43:21 DEBUG "status": "success", 06:43:21 DEBUG "metric": { 06:43:21 DEBUG 06:43:21 DEBUG [...] 06:43:21 DEBUG "data": { 06:43:21 DEBUG "value": [ 06:43:21 DEBUG { 06:43:21 DEBUG "value": [ 06:43:21 DEBUG 06:43:21 DEBUG [...] 06:43:21 DEBUG 06:43:21 DEBUG "value": [ 06:43:21 DEBUG "resultType": "vector", 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG "data": { 06:43:21 DEBUG "pod": "ds-idrepo-2" 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG "resultType": "vector", 06:43:21 DEBUG 1712814201, 06:43:21 DEBUG "metric": { 06:43:21 DEBUG 1712814201, 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG 1712814201, 06:43:21 DEBUG "result": [ 06:43:21 DEBUG [...] 06:43:21 DEBUG [...] 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG "resultType": "vector", 06:43:21 DEBUG }, 06:43:21 DEBUG "instance": "10.142.0.77:9100" 06:43:21 DEBUG 06:43:21 DEBUG "result": [ 06:43:21 DEBUG "82498339.36666667" 06:43:21 DEBUG "instance": "10.142.0.107:9100" 06:43:21 DEBUG "0" 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:43:21 DEBUG "29.133333333333333" 06:43:21 DEBUG { 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG [...] 06:43:21 DEBUG 06:43:21 DEBUG "result": [ 06:43:21 DEBUG "value": [ 06:43:21 DEBUG }, 06:43:21 DEBUG 06:43:21 DEBUG { 06:43:21 DEBUG ] 06:43:21 DEBUG }, 06:43:21 DEBUG ] 06:43:21 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:43:21 DEBUG 06:43:21 DEBUG }, 06:43:21 DEBUG ] 06:43:21 DEBUG "metric": { 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:43:21 DEBUG { 06:43:21 DEBUG 06:43:21 DEBUG "value": [ 06:43:21 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:43:21 DEBUG "metric": { 06:43:21 DEBUG } 06:43:21 DEBUG "value": [ 06:43:21 DEBUG } 06:43:21 DEBUG }, 06:43:21 DEBUG "instance": "10.142.0.77:9100" 06:43:21 DEBUG "value": [ 06:43:21 DEBUG } 06:43:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG }, 06:43:21 DEBUG "metric": { 06:43:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:43:21 DEBUG 06:43:21 DEBUG }, 06:43:21 DEBUG "pod": "ds-idrepo-2" 06:43:21 DEBUG ] 06:43:21 DEBUG 06:43:21 DEBUG ] 06:43:21 DEBUG "value": [ 06:43:21 DEBUG }, 06:43:21 DEBUG 1712814201, 06:43:21 DEBUG ] 06:43:21 DEBUG }, 06:43:21 DEBUG "pod": "ds-cts-2" 06:43:21 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:43:21 DEBUG 06:43:21 DEBUG "value": [ 06:43:21 DEBUG 1712814201, 06:43:21 DEBUG }, 06:43:21 DEBUG 06:43:21 DEBUG "value": [ 06:43:21 DEBUG }, 06:43:21 DEBUG } 06:43:21 DEBUG 06:43:21 DEBUG } 06:43:21 DEBUG 1712814201, 06:43:21 DEBUG "value": [ 06:43:21 DEBUG "0.021" 06:43:21 DEBUG } 06:43:21 DEBUG "value": [ 06:43:21 DEBUG }, 06:43:21 DEBUG }, 06:43:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:43:21 DEBUG }, 06:43:21 DEBUG "0.002299999999999613" 06:43:21 DEBUG "value": [ 06:43:21 DEBUG [...] 06:43:21 DEBUG 06:43:21 DEBUG 1712814201, 06:43:21 DEBUG } 06:43:21 DEBUG 06:43:21 DEBUG } 06:43:21 DEBUG "0" 06:43:21 DEBUG 1712814201, 06:43:21 DEBUG "82495989.76666667" 06:43:21 DEBUG } 06:43:21 DEBUG 06:43:21 DEBUG "value": [ 06:43:21 DEBUG "value": [ 06:43:21 DEBUG 1712814201, 06:43:21 DEBUG "value": [ 06:43:21 DEBUG ] 06:43:21 DEBUG 06:43:21 DEBUG "value": [ 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG [...] 06:43:21 DEBUG ] 06:43:21 DEBUG ] 06:43:21 DEBUG ] 06:43:21 DEBUG 06:43:21 DEBUG 1712814201, 06:43:21 DEBUG 1712814201, 06:43:21 DEBUG "1360.4333333333334" 06:43:21 DEBUG 1712814201, 06:43:21 DEBUG } 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG "4.120591018397055" 06:43:21 DEBUG "pod": "ds-cts-2" 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG } 06:43:21 DEBUG ] 06:43:21 DEBUG 06:43:21 DEBUG "0" 06:43:21 DEBUG "6806.962302107933" 06:43:21 DEBUG ] 06:43:21 DEBUG "0" 06:43:21 DEBUG ] 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG ] 06:43:21 DEBUG }, 06:43:21 DEBUG 06:43:21 DEBUG } 06:43:21 DEBUG } 06:43:21 DEBUG } 06:43:21 DEBUG [...] 06:43:21 DEBUG ] 06:43:21 DEBUG ] 06:43:21 DEBUG } 06:43:21 DEBUG ] 06:43:21 DEBUG } 06:43:21 DEBUG [...] 06:43:21 DEBUG 06:43:21 DEBUG } 06:43:21 DEBUG "value": [ 06:43:21 DEBUG "instance": "10.142.0.77:9100" 06:43:21 DEBUG ] 06:43:21 DEBUG ] 06:43:21 DEBUG } 06:43:21 DEBUG 06:43:21 DEBUG } 06:43:21 DEBUG } 06:43:21 DEBUG ] 06:43:21 DEBUG } 06:43:21 DEBUG } 06:43:21 DEBUG 06:43:21 DEBUG [...] 06:43:21 DEBUG ] 06:43:21 DEBUG 1712814201, 06:43:21 DEBUG }, 06:43:21 DEBUG "value": [ 06:43:21 DEBUG 1712814201, 06:43:21 DEBUG } 06:43:21 DEBUG ] 06:43:21 DEBUG ] 06:43:21 DEBUG } 06:43:21 DEBUG ] 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG } 06:43:21 DEBUG "0.00009999999999999998" 06:43:21 DEBUG } 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG "52951124992" 06:43:21 DEBUG } 06:43:21 DEBUG } 06:43:21 DEBUG } 06:43:21 DEBUG } 06:43:21 DEBUG 06:43:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:43:21 DEBUG } 06:43:21 DEBUG ] 06:43:21 DEBUG } 06:43:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:43:21 DEBUG } 06:43:21 DEBUG ] 06:43:21 DEBUG } 06:43:21 DEBUG } 06:43:21 DEBUG } 06:43:21 DEBUG 06:43:21 DEBUG }, 06:43:21 DEBUG } 06:43:21 DEBUG }, 06:43:21 DEBUG } 06:43:21 DEBUG 06:43:21 DEBUG "value": [ 06:43:21 DEBUG ] 06:43:21 DEBUG "value": [ 06:43:21 DEBUG ] 06:43:21 DEBUG "pod": "ds-idrepo-0" 06:43:21 DEBUG 1712814201, 06:43:21 DEBUG } 06:43:21 DEBUG 1712814201, 06:43:21 DEBUG } 06:43:21 DEBUG }, 06:43:21 DEBUG "244252.5" 06:43:21 DEBUG } 06:43:21 DEBUG "1347" 06:43:21 DEBUG } 06:43:21 DEBUG "value": [ 06:43:21 DEBUG 1712814201, 06:43:21 DEBUG ] 06:43:21 DEBUG ] 06:43:21 DEBUG "0" 06:43:21 DEBUG } 06:43:21 DEBUG } 06:43:21 DEBUG ] 06:43:21 DEBUG ] 06:43:21 DEBUG ] 06:43:21 DEBUG } 06:43:21 DEBUG } 06:43:21 DEBUG } 06:43:21 DEBUG ] 06:43:21 DEBUG } 06:43:21 DEBUG } 06:43:21 DEBUG } 06:43:21 DEBUG } 06:43:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814201" 06:43:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712814201" 06:43:21 INFO [http_cmd]: http status code OK 06:43:21 DEBUG --- status code --- 06:43:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:21 DEBUG --- http response --- 06:43:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:43:21 DEBUG ----- output ----- 06:43:21 DEBUG { 06:43:21 DEBUG "status": "success", 06:43:21 DEBUG "data": { 06:43:21 DEBUG "resultType": "vector", 06:43:21 DEBUG "result": [ 06:43:21 DEBUG { 06:43:21 DEBUG "metric": { 06:43:21 DEBUG "pod": "ds-idrepo-2" 06:43:21 DEBUG }, 06:43:21 DEBUG "value": [ 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG [...] 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG "pod": "ds-cts-2" 06:43:21 DEBUG }, 06:43:21 DEBUG "value": [ 06:43:21 DEBUG 1712814201, 06:43:21 DEBUG "0" 06:43:21 DEBUG ] 06:43:21 DEBUG } 06:43:21 DEBUG ] 06:43:21 DEBUG } 06:43:21 DEBUG } 06:43:21 INFO [http_cmd]: http status code OK 06:43:21 DEBUG --- status code --- 06:43:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:21 DEBUG --- http response --- 06:43:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:43:21 DEBUG ----- output ----- 06:43:21 DEBUG { 06:43:21 DEBUG "status": "success", 06:43:21 DEBUG "data": { 06:43:21 DEBUG "resultType": "vector", 06:43:21 DEBUG "result": [ 06:43:21 DEBUG { 06:43:21 DEBUG "metric": { 06:43:21 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:43:21 DEBUG }, 06:43:21 DEBUG "value": [ 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG [...] 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 06:43:21 DEBUG }, 06:43:21 DEBUG "value": [ 06:43:21 DEBUG 1712814201, 06:43:21 DEBUG "14137999360" 06:43:21 DEBUG ] 06:43:21 DEBUG } 06:43:21 DEBUG ] 06:43:21 DEBUG } 06:43:21 DEBUG } 06:43:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814201" 06:43:21 INFO [http_cmd]: http status code OK 06:43:21 DEBUG --- status code --- 06:43:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:21 DEBUG --- http response --- 06:43:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:43:21 DEBUG ----- output ----- 06:43:21 DEBUG { 06:43:21 DEBUG "status": "success", 06:43:21 DEBUG "data": { 06:43:21 DEBUG "resultType": "vector", 06:43:21 DEBUG "result": [ 06:43:21 DEBUG { 06:43:21 DEBUG "metric": { 06:43:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:43:21 DEBUG }, 06:43:21 DEBUG "value": [ 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG [...] 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:43:21 DEBUG }, 06:43:21 DEBUG "value": [ 06:43:21 DEBUG 1712814201, 06:43:21 DEBUG "1346.7666666666667" 06:43:21 DEBUG ] 06:43:21 DEBUG } 06:43:21 DEBUG ] 06:43:21 DEBUG } 06:43:21 DEBUG } 06:43:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814201" 06:43:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814201" 06:43:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814201" 06:43:21 INFO [http_cmd]: http status code OK 06:43:21 DEBUG --- status code --- 06:43:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:21 DEBUG --- http response --- 06:43:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:43:21 DEBUG ----- output ----- 06:43:21 DEBUG { 06:43:21 DEBUG "status": "success", 06:43:21 DEBUG "data": { 06:43:21 DEBUG "resultType": "vector", 06:43:21 DEBUG "result": [ 06:43:21 DEBUG { 06:43:21 DEBUG "metric": { 06:43:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:43:21 DEBUG }, 06:43:21 DEBUG "value": [ 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG [...] 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:43:21 DEBUG }, 06:43:21 DEBUG "value": [ 06:43:21 DEBUG 1712814201, 06:43:21 DEBUG "1355.7333333333333" 06:43:21 DEBUG ] 06:43:21 DEBUG } 06:43:21 DEBUG ] 06:43:21 DEBUG } 06:43:21 DEBUG } 06:43:21 INFO [http_cmd]: http status code OK 06:43:21 DEBUG --- status code --- 06:43:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:21 DEBUG --- http response --- 06:43:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:43:21 DEBUG ----- output ----- 06:43:21 DEBUG { 06:43:21 DEBUG "status": "success", 06:43:21 DEBUG "data": { 06:43:21 DEBUG "resultType": "vector", 06:43:21 DEBUG "result": [ 06:43:21 DEBUG { 06:43:21 DEBUG "metric": { 06:43:21 DEBUG "pod": "ds-idrepo-2" 06:43:21 DEBUG }, 06:43:21 DEBUG "value": [ 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG [...] 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG "pod": "ds-idrepo-0" 06:43:21 DEBUG }, 06:43:21 DEBUG "value": [ 06:43:21 DEBUG 1712814201, 06:43:21 DEBUG "0" 06:43:21 DEBUG ] 06:43:21 DEBUG } 06:43:21 DEBUG ] 06:43:21 DEBUG } 06:43:21 DEBUG } 06:43:21 INFO [http_cmd]: http status code OK 06:43:21 DEBUG --- status code --- 06:43:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:21 DEBUG --- http response --- 06:43:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:43:21 DEBUG ----- output ----- 06:43:21 DEBUG { 06:43:21 DEBUG "status": "success", 06:43:21 DEBUG "data": { 06:43:21 DEBUG "resultType": "vector", 06:43:21 DEBUG "result": [ 06:43:21 DEBUG { 06:43:21 DEBUG "metric": { 06:43:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:43:21 DEBUG }, 06:43:21 DEBUG "value": [ 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG [...] 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:43:21 DEBUG }, 06:43:21 DEBUG "value": [ 06:43:21 DEBUG 1712814201, 06:43:21 DEBUG "0" 06:43:21 DEBUG ] 06:43:21 DEBUG } 06:43:21 DEBUG ] 06:43:21 DEBUG } 06:43:21 DEBUG } 06:43:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814201" 06:43:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712814201" 06:43:21 INFO [http_cmd]: http status code OK 06:43:21 DEBUG --- status code --- 06:43:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:21 DEBUG --- http response --- 06:43:21 INFO [http_cmd]: http status code OK 06:43:21 DEBUG --- status code --- 06:43:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:43:21 DEBUG --- http response --- 06:43:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:43:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:43:21 DEBUG ----- output ----- 06:43:21 DEBUG ----- output ----- 06:43:21 DEBUG { 06:43:21 DEBUG { 06:43:21 DEBUG "status": "success", 06:43:21 DEBUG "status": "success", 06:43:21 DEBUG "data": { 06:43:21 DEBUG "data": { 06:43:21 DEBUG "resultType": "vector", 06:43:21 DEBUG "resultType": "vector", 06:43:21 DEBUG "result": [ 06:43:21 DEBUG "result": [ 06:43:21 DEBUG { 06:43:21 DEBUG { 06:43:21 DEBUG "metric": { 06:43:21 DEBUG "metric": { 06:43:21 DEBUG "instance": "10.142.0.107:9100" 06:43:21 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:43:21 DEBUG }, 06:43:21 DEBUG }, 06:43:21 DEBUG "value": [ 06:43:21 DEBUG "value": [ 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG [...] 06:43:21 DEBUG [...] 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG 06:43:21 DEBUG "instance": "10.142.0.77:9100" 06:43:21 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:43:21 DEBUG }, 06:43:21 DEBUG }, 06:43:21 DEBUG "value": [ 06:43:21 DEBUG "value": [ 06:43:21 DEBUG 1712814201, 06:43:21 DEBUG 1712814201, 06:43:21 DEBUG "0.0006666666666668892" 06:43:21 DEBUG "4.15364215109301" 06:43:21 DEBUG ] 06:43:21 DEBUG ] 06:43:21 DEBUG } 06:43:21 DEBUG } 06:43:21 DEBUG ] 06:43:21 DEBUG ] 06:43:21 DEBUG } 06:43:21 DEBUG } 06:43:21 DEBUG } 06:43:21 DEBUG } 06:43:32 INFO 06:43:32 INFO [loop_until]: kubectl --namespace=xlou top pods 06:43:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:43:32 INFO [loop_until]: OK (rc = 0) 06:43:32 DEBUG --- stdout --- 06:43:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 9054m 8500Mi am-5b5bd8567b-8h2w4 8624m 8564Mi am-5b5bd8567b-qlkw6 7637m 8457Mi ds-cts-0 7994m 13393Mi ds-cts-1 8507m 12947Mi ds-cts-2 8851m 13394Mi ds-idrepo-0 11304m 5503Mi ds-idrepo-1 381m 5479Mi ds-idrepo-2 364m 5783Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 4m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 2m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4094m 1856Mi 06:43:32 DEBUG --- stderr --- 06:43:32 DEBUG 06:43:36 INFO 06:43:36 INFO [loop_until]: kubectl --namespace=xlou top node 06:43:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:43:36 INFO [loop_until]: OK (rc = 0) 06:43:36 DEBUG --- stdout --- 06:43:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 2726Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 64m 0% 2788Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7854m 26% 10118Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10773m 36% 2981Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9550m 31% 10080Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 144m 0% 3857Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9772m 32% 10188Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 421m 2% 6465Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11548m 72% 6193Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8757m 55% 13988Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 8126m 51% 13996Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 8574m 53% 13625Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 450m 2% 6159Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4242m 26% 5612Mi 9% 06:43:36 DEBUG --- stderr --- 06:43:36 DEBUG 06:44:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814261" 06:44:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814261" 06:44:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814261" 06:44:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814261" 06:44:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712814261" 06:44:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712814261" 06:44:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814261" 06:44:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814261" 06:44:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814261" 06:44:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814261" 06:44:21 INFO [http_cmd]: http status code OK 06:44:21 DEBUG --- status code --- 06:44:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:21 DEBUG --- http response --- 06:44:21 INFO [http_cmd]: http status code OK 06:44:21 INFO [http_cmd]: http status code OK 06:44:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:44:21 DEBUG --- status code --- 06:44:21 DEBUG --- status code --- 06:44:21 DEBUG ----- output ----- 06:44:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:21 DEBUG { 06:44:21 DEBUG --- http response --- 06:44:21 DEBUG --- http response --- 06:44:21 DEBUG "status": "success", 06:44:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:44:21 DEBUG "data": { 06:44:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 06:44:21 DEBUG ----- output ----- 06:44:21 DEBUG "resultType": "vector", 06:44:21 DEBUG ----- output ----- 06:44:21 DEBUG { 06:44:21 DEBUG "result": [ 06:44:21 DEBUG { 06:44:21 DEBUG "status": "success", 06:44:21 DEBUG { 06:44:21 DEBUG "status": "success", 06:44:21 DEBUG "data": { 06:44:21 DEBUG "metric": { 06:44:21 DEBUG "data": { 06:44:21 DEBUG "resultType": "vector", 06:44:21 DEBUG "instance": "10.142.0.107:9100" 06:44:21 DEBUG "resultType": "vector", 06:44:21 DEBUG "result": [ 06:44:21 DEBUG }, 06:44:21 DEBUG "result": [ 06:44:21 DEBUG { 06:44:21 DEBUG "value": [ 06:44:21 DEBUG { 06:44:21 DEBUG "metric": { 06:44:21 DEBUG 06:44:21 DEBUG "metric": { 06:44:21 DEBUG "instance": "10.142.0.107:9100" 06:44:21 ERROR [http_cmd]: ERROR 06:44:21 DEBUG 06:44:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:44:21 DEBUG }, 06:44:21 ERROR --- status code --- 06:44:21 DEBUG 06:44:21 DEBUG }, 06:44:21 DEBUG "value": [ 06:44:21 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:44:21 DEBUG [...] 06:44:21 DEBUG "value": [ 06:44:21 DEBUG 06:44:21 ERROR --- http response --- 06:44:21 INFO [http_cmd]: http status code OK 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:44:21 DEBUG --- status code --- 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:44:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG [...] 06:44:21 DEBUG --- http response --- 06:44:21 DEBUG "instance": "10.142.0.77:9100" 06:44:21 DEBUG [...] 06:44:21 DEBUG 06:44:21 DEBUG }, 06:44:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG "value": [ 06:44:21 DEBUG ----- output ----- 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 1712814261, 06:44:21 DEBUG { 06:44:21 DEBUG 06:44:21 DEBUG "instance": "10.142.0.77:9100" 06:44:21 DEBUG "0.002066666666666587" 06:44:21 DEBUG "status": "success", 06:44:21 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:44:21 DEBUG }, 06:44:21 INFO [http_cmd]: http status code OK 06:44:21 INFO [http_cmd]: http status code OK 06:44:21 INFO [http_cmd]: http status code OK 06:44:21 INFO [http_cmd]: http status code OK 06:44:21 DEBUG ] 06:44:21 INFO [http_cmd]: http status code OK 06:44:21 DEBUG "data": { 06:44:21 DEBUG }, 06:44:21 DEBUG "value": [ 06:44:21 DEBUG --- status code --- 06:44:21 DEBUG --- status code --- 06:44:21 DEBUG --- status code --- 06:44:21 DEBUG --- status code --- 06:44:21 DEBUG } 06:44:21 DEBUG --- status code --- 06:44:21 DEBUG "resultType": "vector", 06:44:21 DEBUG "value": [ 06:44:21 DEBUG 1712814261, 06:44:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:21 DEBUG ] 06:44:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:21 DEBUG "result": [ 06:44:21 DEBUG 1712814261, 06:44:21 DEBUG "82460745.53333333" 06:44:21 DEBUG --- http response --- 06:44:21 DEBUG --- http response --- 06:44:21 DEBUG --- http response --- 06:44:21 DEBUG --- http response --- 06:44:21 DEBUG } 06:44:21 DEBUG --- http response --- 06:44:21 DEBUG { 06:44:21 DEBUG "2.8395424937795877" 06:44:21 DEBUG ] 06:44:21 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:44:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:44:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:44:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:44:21 DEBUG } 06:44:21 DEBUG "metric": { 06:44:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:44:21 DEBUG ] 06:44:21 DEBUG } 06:44:21 DEBUG ----- output ----- 06:44:21 DEBUG ----- output ----- 06:44:21 DEBUG ----- output ----- 06:44:21 DEBUG "instance": "10.142.0.107:9100" 06:44:21 DEBUG ----- output ----- 06:44:21 DEBUG } 06:44:21 DEBUG ] 06:44:21 DEBUG { 06:44:21 DEBUG { 06:44:21 DEBUG { 06:44:21 DEBUG }, 06:44:21 DEBUG { 06:44:21 DEBUG ] 06:44:21 DEBUG } 06:44:21 DEBUG "status": "success", 06:44:21 DEBUG "status": "success", 06:44:21 DEBUG "status": "success", 06:44:21 DEBUG "value": [ 06:44:21 DEBUG "status": "success", 06:44:21 DEBUG } 06:44:21 DEBUG } 06:44:21 DEBUG "data": { 06:44:21 DEBUG "data": { 06:44:21 DEBUG "data": { 06:44:21 DEBUG 06:44:21 DEBUG "data": { 06:44:21 DEBUG } 06:44:21 DEBUG "resultType": "vector", 06:44:21 DEBUG "resultType": "vector", 06:44:21 DEBUG "resultType": "vector", 06:44:21 DEBUG 06:44:21 DEBUG "resultType": "vector", 06:44:21 DEBUG "result": [ 06:44:21 DEBUG "result": [ 06:44:21 DEBUG "result": [ 06:44:21 DEBUG 06:44:21 DEBUG "result": [ 06:44:21 DEBUG { 06:44:21 DEBUG { 06:44:21 DEBUG { 06:44:21 DEBUG [...] 06:44:21 DEBUG { 06:44:21 DEBUG "metric": { 06:44:21 DEBUG "metric": { 06:44:21 DEBUG "metric": { 06:44:21 DEBUG 06:44:21 DEBUG "metric": { 06:44:21 DEBUG "pod": "ds-idrepo-2" 06:44:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:44:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:44:21 DEBUG 06:44:21 DEBUG "pod": "ds-idrepo-2" 06:44:21 DEBUG }, 06:44:21 DEBUG }, 06:44:21 DEBUG }, 06:44:21 DEBUG 06:44:21 DEBUG }, 06:44:21 DEBUG "value": [ 06:44:21 DEBUG "value": [ 06:44:21 DEBUG "value": [ 06:44:21 DEBUG "instance": "10.142.0.77:9100" 06:44:21 DEBUG "value": [ 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814261" 06:44:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814261" 06:44:21 DEBUG }, 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG "value": [ 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 1712814261, 06:44:21 DEBUG 06:44:21 DEBUG [...] 06:44:21 DEBUG [...] 06:44:21 DEBUG [...] 06:44:21 DEBUG "82463269.16666666" 06:44:21 DEBUG [...] 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG ] 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814261" 06:44:21 DEBUG 06:44:21 DEBUG } 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG ] 06:44:21 DEBUG 06:44:21 DEBUG "pod": "ds-idrepo-0" 06:44:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:44:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:44:21 DEBUG } 06:44:21 DEBUG "pod": "ds-cts-2" 06:44:21 DEBUG }, 06:44:21 DEBUG }, 06:44:21 DEBUG }, 06:44:21 DEBUG } 06:44:21 DEBUG }, 06:44:21 DEBUG "value": [ 06:44:21 DEBUG "value": [ 06:44:21 DEBUG "value": [ 06:44:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814261" 06:44:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814261" 06:44:21 DEBUG "value": [ 06:44:21 DEBUG 1712814261, 06:44:21 DEBUG 1712814261, 06:44:21 DEBUG 1712814261, 06:44:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712814261" 06:44:21 DEBUG 1712814261, 06:44:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814261" 06:44:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712814261" 06:44:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814261" 06:44:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814261" 06:44:21 DEBUG "0" 06:44:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814261" 06:44:21 DEBUG "1363.3666666666666" 06:44:21 DEBUG "244618" 06:44:21 DEBUG "0" 06:44:21 DEBUG ] 06:44:21 INFO [http_cmd]: http status code OK 06:44:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814261" 06:44:21 INFO [http_cmd]: http status code OK 06:44:21 DEBUG ] 06:44:21 INFO [http_cmd]: http status code OK 06:44:21 INFO [http_cmd]: http status code OK 06:44:21 DEBUG ] 06:44:21 INFO [http_cmd]: http status code OK 06:44:21 INFO [http_cmd]: http status code OK 06:44:21 INFO [http_cmd]: http status code OK 06:44:21 DEBUG ] 06:44:21 DEBUG } 06:44:21 DEBUG --- status code --- 06:44:21 DEBUG --- status code --- 06:44:21 DEBUG } 06:44:21 DEBUG --- status code --- 06:44:21 INFO [http_cmd]: http status code OK 06:44:21 DEBUG --- status code --- 06:44:21 DEBUG } 06:44:21 DEBUG ] 06:44:21 DEBUG --- status code --- 06:44:21 DEBUG --- status code --- 06:44:21 DEBUG } 06:44:21 DEBUG ] 06:44:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:21 DEBUG ] 06:44:21 INFO [http_cmd]: http status code OK 06:44:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:21 INFO [http_cmd]: http status code OK 06:44:21 DEBUG --- status code --- 06:44:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:21 DEBUG } 06:44:21 DEBUG --- status code --- 06:44:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:21 DEBUG ] 06:44:21 DEBUG } 06:44:21 DEBUG --- http response --- 06:44:21 DEBUG --- http response --- 06:44:21 DEBUG } 06:44:21 DEBUG --- status code --- 06:44:21 DEBUG --- http response --- 06:44:21 DEBUG --- status code --- 06:44:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:21 DEBUG --- http response --- 06:44:21 DEBUG } 06:44:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:21 DEBUG --- http response --- 06:44:21 DEBUG --- http response --- 06:44:21 DEBUG } 06:44:21 DEBUG } 06:44:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:44:21 DEBUG } 06:44:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:44:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:44:21 DEBUG --- http response --- 06:44:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 06:44:21 DEBUG --- http response --- 06:44:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:44:21 DEBUG } 06:44:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:44:21 DEBUG ----- output ----- 06:44:21 DEBUG ----- output ----- 06:44:21 DEBUG --- http response --- 06:44:21 INFO [http_cmd]: http status code OK 06:44:21 DEBUG --- http response --- 06:44:21 DEBUG ----- output ----- 06:44:21 DEBUG ----- output ----- 06:44:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:44:21 INFO [http_cmd]: http status code OK 06:44:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 06:44:21 DEBUG ----- output ----- 06:44:21 DEBUG ----- output ----- 06:44:21 DEBUG { 06:44:21 DEBUG { 06:44:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:44:21 DEBUG --- status code --- 06:44:21 DEBUG { 06:44:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:44:21 DEBUG { 06:44:21 DEBUG ----- output ----- 06:44:21 DEBUG --- status code --- 06:44:21 DEBUG ----- output ----- 06:44:21 DEBUG { 06:44:21 DEBUG { 06:44:21 DEBUG "status": "success", 06:44:21 DEBUG "status": "success", 06:44:21 DEBUG ----- output ----- 06:44:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:21 DEBUG "status": "success", 06:44:21 DEBUG ----- output ----- 06:44:21 DEBUG "status": "success", 06:44:21 DEBUG { 06:44:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:21 DEBUG { 06:44:21 DEBUG "status": "success", 06:44:21 DEBUG "status": "success", 06:44:21 DEBUG "data": { 06:44:21 DEBUG "data": { 06:44:21 DEBUG { 06:44:21 DEBUG --- http response --- 06:44:21 DEBUG "data": { 06:44:21 DEBUG { 06:44:21 DEBUG "data": { 06:44:21 DEBUG "status": "success", 06:44:21 DEBUG --- http response --- 06:44:21 DEBUG "status": "success", 06:44:21 DEBUG "data": { 06:44:21 DEBUG "data": { 06:44:21 DEBUG "resultType": "vector", 06:44:21 DEBUG "resultType": "vector", 06:44:21 DEBUG "status": "success", 06:44:21 DEBUG "resultType": "vector", 06:44:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:44:21 DEBUG "status": "success", 06:44:21 DEBUG "resultType": "vector", 06:44:21 DEBUG "data": { 06:44:21 DEBUG "data": { 06:44:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:44:21 DEBUG "resultType": "vector", 06:44:21 DEBUG "resultType": "vector", 06:44:21 DEBUG "result": [ 06:44:21 DEBUG "result": [ 06:44:21 DEBUG "data": { 06:44:21 DEBUG "result": [ 06:44:21 DEBUG ----- output ----- 06:44:21 DEBUG "data": { 06:44:21 DEBUG "result": [ 06:44:21 DEBUG "resultType": "vector", 06:44:21 DEBUG "resultType": "vector", 06:44:21 DEBUG ----- output ----- 06:44:21 DEBUG "result": [ 06:44:21 DEBUG "result": [ 06:44:21 DEBUG { 06:44:21 DEBUG { 06:44:21 DEBUG "resultType": "vector", 06:44:21 DEBUG { 06:44:21 DEBUG { 06:44:21 DEBUG "resultType": "vector", 06:44:21 DEBUG { 06:44:21 DEBUG "result": [ 06:44:21 DEBUG "result": [ 06:44:21 DEBUG { 06:44:21 DEBUG { 06:44:21 DEBUG { 06:44:21 DEBUG "metric": { 06:44:21 DEBUG "metric": { 06:44:21 DEBUG "result": [ 06:44:21 DEBUG "metric": { 06:44:21 DEBUG "status": "success", 06:44:21 DEBUG "result": [ 06:44:21 DEBUG "metric": { 06:44:21 DEBUG { 06:44:21 DEBUG { 06:44:21 DEBUG "status": "success", 06:44:21 DEBUG "metric": { 06:44:21 DEBUG "metric": { 06:44:21 DEBUG "pod": "ds-idrepo-2" 06:44:21 DEBUG "pod": "ds-idrepo-2" 06:44:21 DEBUG { 06:44:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:44:21 DEBUG "data": { 06:44:21 DEBUG { 06:44:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:44:21 DEBUG "metric": { 06:44:21 DEBUG "metric": { 06:44:21 DEBUG "data": { 06:44:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:44:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:44:21 DEBUG }, 06:44:21 DEBUG }, 06:44:21 DEBUG "metric": { 06:44:21 DEBUG }, 06:44:21 DEBUG "resultType": "vector", 06:44:21 DEBUG "metric": { 06:44:21 DEBUG }, 06:44:21 DEBUG "instance": "10.142.0.107:9100" 06:44:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:44:21 DEBUG "resultType": "vector", 06:44:21 DEBUG }, 06:44:21 DEBUG }, 06:44:21 DEBUG "value": [ 06:44:21 DEBUG "value": [ 06:44:21 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:44:21 DEBUG "value": [ 06:44:21 DEBUG "result": [ 06:44:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:44:21 DEBUG "value": [ 06:44:21 DEBUG }, 06:44:21 DEBUG }, 06:44:21 DEBUG "result": [ 06:44:21 DEBUG "value": [ 06:44:21 DEBUG "value": [ 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG }, 06:44:21 DEBUG 06:44:21 DEBUG { 06:44:21 DEBUG }, 06:44:21 DEBUG 06:44:21 DEBUG "value": [ 06:44:21 DEBUG "value": [ 06:44:21 DEBUG { 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG "value": [ 06:44:21 DEBUG 06:44:21 DEBUG "metric": { 06:44:21 DEBUG "value": [ 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG "metric": { 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG "pod": "ds-idrepo-2" 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG "pod": "ds-idrepo-2" 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG [...] 06:44:21 DEBUG [...] 06:44:21 DEBUG 06:44:21 DEBUG [...] 06:44:21 DEBUG }, 06:44:21 DEBUG 06:44:21 DEBUG [...] 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG }, 06:44:21 DEBUG [...] 06:44:21 DEBUG [...] 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG "value": [ 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG [...] 06:44:21 DEBUG [...] 06:44:21 DEBUG "value": [ 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG [...] 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG [...] 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG "pod": "ds-idrepo-0" 06:44:21 DEBUG "pod": "ds-cts-2" 06:44:21 DEBUG 06:44:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:44:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:44:21 DEBUG }, 06:44:21 DEBUG }, 06:44:21 DEBUG 06:44:21 DEBUG }, 06:44:21 DEBUG [...] 06:44:21 DEBUG 06:44:21 DEBUG }, 06:44:21 DEBUG "instance": "10.142.0.77:9100" 06:44:21 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:44:21 DEBUG [...] 06:44:21 DEBUG }, 06:44:21 DEBUG }, 06:44:21 DEBUG "value": [ 06:44:21 DEBUG "value": [ 06:44:21 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 06:44:21 DEBUG "value": [ 06:44:21 DEBUG 06:44:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:44:21 DEBUG "value": [ 06:44:21 DEBUG }, 06:44:21 DEBUG }, 06:44:21 DEBUG 06:44:21 DEBUG "value": [ 06:44:21 DEBUG "value": [ 06:44:21 DEBUG 1712814261, 06:44:21 DEBUG 1712814261, 06:44:21 DEBUG }, 06:44:21 DEBUG 1712814261, 06:44:21 DEBUG 06:44:21 DEBUG }, 06:44:21 DEBUG 1712814261, 06:44:21 DEBUG "value": [ 06:44:21 DEBUG "value": [ 06:44:21 DEBUG 06:44:21 DEBUG 1712814261, 06:44:21 DEBUG 1712814261, 06:44:21 DEBUG "0" 06:44:21 DEBUG "29.566666666666666" 06:44:21 DEBUG "value": [ 06:44:21 DEBUG "1363.5333333333333" 06:44:21 DEBUG 06:44:21 DEBUG "value": [ 06:44:21 DEBUG "0" 06:44:21 DEBUG 1712814261, 06:44:21 DEBUG 1712814261, 06:44:21 DEBUG 06:44:21 DEBUG "0.021" 06:44:21 DEBUG "1364.9666666666667" 06:44:21 DEBUG ] 06:44:21 DEBUG ] 06:44:21 DEBUG 1712814261, 06:44:21 DEBUG ] 06:44:21 DEBUG "pod": "ds-cts-2" 06:44:21 DEBUG 1712814261, 06:44:21 DEBUG ] 06:44:21 DEBUG "52952445952" 06:44:21 DEBUG "62320174.22489356" 06:44:21 DEBUG "pod": "ds-cts-2" 06:44:21 DEBUG ] 06:44:21 DEBUG ] 06:44:21 DEBUG } 06:44:21 DEBUG } 06:44:21 DEBUG "13276454912" 06:44:21 DEBUG } 06:44:21 DEBUG }, 06:44:21 DEBUG "0" 06:44:21 DEBUG } 06:44:21 DEBUG ] 06:44:21 DEBUG ] 06:44:21 DEBUG }, 06:44:21 DEBUG } 06:44:21 DEBUG } 06:44:21 DEBUG ] 06:44:21 DEBUG ] 06:44:21 DEBUG ] 06:44:21 DEBUG ] 06:44:21 DEBUG "value": [ 06:44:21 DEBUG ] 06:44:21 DEBUG ] 06:44:21 DEBUG } 06:44:21 DEBUG } 06:44:21 DEBUG "value": [ 06:44:21 DEBUG ] 06:44:21 DEBUG ] 06:44:21 DEBUG } 06:44:21 DEBUG } 06:44:21 DEBUG } 06:44:21 DEBUG } 06:44:21 DEBUG 1712814261, 06:44:21 DEBUG } 06:44:21 DEBUG } 06:44:21 DEBUG ] 06:44:21 DEBUG ] 06:44:21 DEBUG 1712814261, 06:44:21 DEBUG } 06:44:21 DEBUG } 06:44:21 DEBUG } 06:44:21 DEBUG } 06:44:21 DEBUG ] 06:44:21 DEBUG } 06:44:21 DEBUG "0.000030149444444444445" 06:44:21 DEBUG ] 06:44:21 DEBUG } 06:44:21 DEBUG } 06:44:21 DEBUG } 06:44:21 DEBUG "0" 06:44:21 DEBUG ] 06:44:21 DEBUG } 06:44:21 DEBUG } 06:44:21 DEBUG ] 06:44:21 DEBUG } 06:44:21 DEBUG } 06:44:21 DEBUG } 06:44:21 DEBUG } 06:44:21 DEBUG ] 06:44:21 DEBUG } 06:44:21 DEBUG } 06:44:21 DEBUG } 06:44:21 DEBUG } 06:44:21 DEBUG } 06:44:21 DEBUG ] 06:44:21 DEBUG } 06:44:21 DEBUG } 06:44:21 DEBUG } 06:44:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814261" 06:44:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814261" 06:44:21 INFO [http_cmd]: http status code OK 06:44:21 DEBUG --- status code --- 06:44:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:21 DEBUG --- http response --- 06:44:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:44:21 DEBUG ----- output ----- 06:44:21 DEBUG { 06:44:21 DEBUG "status": "success", 06:44:21 DEBUG "data": { 06:44:21 DEBUG "resultType": "vector", 06:44:21 DEBUG "result": [ 06:44:21 DEBUG { 06:44:21 DEBUG "metric": { 06:44:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:44:21 DEBUG }, 06:44:21 DEBUG "value": [ 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG [...] 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:44:21 DEBUG }, 06:44:21 DEBUG "value": [ 06:44:21 DEBUG 1712814261, 06:44:21 DEBUG "1362.4" 06:44:21 DEBUG ] 06:44:21 DEBUG } 06:44:21 DEBUG ] 06:44:21 DEBUG } 06:44:21 DEBUG } 06:44:21 INFO [http_cmd]: http status code OK 06:44:21 DEBUG --- status code --- 06:44:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:21 DEBUG --- http response --- 06:44:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:44:21 DEBUG ----- output ----- 06:44:21 DEBUG { 06:44:21 DEBUG "status": "success", 06:44:21 DEBUG "data": { 06:44:21 DEBUG "resultType": "vector", 06:44:21 DEBUG "result": [ 06:44:21 DEBUG { 06:44:21 DEBUG "metric": { 06:44:21 DEBUG "pod": "ds-idrepo-2" 06:44:21 DEBUG }, 06:44:21 DEBUG "value": [ 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG [...] 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG "pod": "ds-idrepo-0" 06:44:21 DEBUG }, 06:44:21 DEBUG "value": [ 06:44:21 DEBUG 1712814261, 06:44:21 DEBUG "0" 06:44:21 DEBUG ] 06:44:21 DEBUG } 06:44:21 DEBUG ] 06:44:21 DEBUG } 06:44:21 DEBUG } 06:44:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814261" 06:44:21 INFO [http_cmd]: http status code OK 06:44:21 DEBUG --- status code --- 06:44:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:21 DEBUG --- http response --- 06:44:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 06:44:21 DEBUG ----- output ----- 06:44:21 DEBUG { 06:44:21 DEBUG "status": "success", 06:44:21 DEBUG "data": { 06:44:21 DEBUG "resultType": "vector", 06:44:21 DEBUG "result": [ 06:44:21 DEBUG { 06:44:21 DEBUG "metric": { 06:44:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:44:21 DEBUG }, 06:44:21 DEBUG "value": [ 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG [...] 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:44:21 DEBUG }, 06:44:21 DEBUG "value": [ 06:44:21 DEBUG 1712814261, 06:44:21 DEBUG "0" 06:44:21 DEBUG ] 06:44:21 DEBUG } 06:44:21 DEBUG ] 06:44:21 DEBUG } 06:44:21 DEBUG } 06:44:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814261" 06:44:21 INFO [http_cmd]: http status code OK 06:44:21 DEBUG --- status code --- 06:44:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:21 DEBUG --- http response --- 06:44:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712814261" 06:44:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:44:21 DEBUG ----- output ----- 06:44:21 DEBUG { 06:44:21 DEBUG "status": "success", 06:44:21 DEBUG "data": { 06:44:21 DEBUG "resultType": "vector", 06:44:21 DEBUG "result": [ 06:44:21 DEBUG { 06:44:21 DEBUG "metric": { 06:44:21 DEBUG "instance": "10.142.0.107:9100" 06:44:21 DEBUG }, 06:44:21 DEBUG "value": [ 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG [...] 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG "instance": "10.142.0.77:9100" 06:44:21 DEBUG }, 06:44:21 DEBUG "value": [ 06:44:21 DEBUG 1712814261, 06:44:21 DEBUG "0" 06:44:21 DEBUG ] 06:44:21 DEBUG } 06:44:21 DEBUG ] 06:44:21 DEBUG } 06:44:21 DEBUG } 06:44:21 INFO [http_cmd]: http status code OK 06:44:21 DEBUG --- status code --- 06:44:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:44:21 DEBUG --- http response --- 06:44:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:44:21 DEBUG ----- output ----- 06:44:21 DEBUG { 06:44:21 DEBUG "status": "success", 06:44:21 DEBUG "data": { 06:44:21 DEBUG "resultType": "vector", 06:44:21 DEBUG "result": [ 06:44:21 DEBUG { 06:44:21 DEBUG "metric": { 06:44:21 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:44:21 DEBUG }, 06:44:21 DEBUG "value": [ 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG [...] 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG 06:44:21 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:44:21 DEBUG }, 06:44:21 DEBUG "value": [ 06:44:21 DEBUG 1712814261, 06:44:21 DEBUG "4.184901174486092" 06:44:21 DEBUG ] 06:44:21 DEBUG } 06:44:21 DEBUG ] 06:44:21 DEBUG } 06:44:21 DEBUG } 06:44:32 INFO 06:44:32 INFO [loop_until]: kubectl --namespace=xlou top pods 06:44:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:44:32 INFO [loop_until]: OK (rc = 0) 06:44:32 DEBUG --- stdout --- 06:44:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 8903m 8511Mi am-5b5bd8567b-8h2w4 8661m 8566Mi am-5b5bd8567b-qlkw6 7487m 8460Mi ds-cts-0 7908m 12684Mi ds-cts-1 8569m 12614Mi ds-cts-2 8525m 12751Mi ds-idrepo-0 11451m 5503Mi ds-idrepo-1 358m 5479Mi ds-idrepo-2 376m 5783Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 3m 1010Mi idm-668d898cfd-st2t7 5m 1135Mi lodemon-ccc964465-p7gdm 2m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4239m 1856Mi 06:44:32 DEBUG --- stderr --- 06:44:32 DEBUG 06:44:36 INFO 06:44:36 INFO [loop_until]: kubectl --namespace=xlou top node 06:44:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:44:36 INFO [loop_until]: OK (rc = 0) 06:44:36 DEBUG --- stdout --- 06:44:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 2742Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 62m 0% 2782Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7976m 26% 10117Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10731m 35% 2992Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9635m 32% 10093Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 139m 0% 3842Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9612m 32% 10191Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 439m 2% 6464Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11543m 72% 6195Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8592m 54% 13427Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 8223m 51% 13312Mi 22% gke-xlou-cdm-ds-32e4dcb1-j2hk 8661m 54% 13196Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 415m 2% 6157Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4308m 27% 5613Mi 9% 06:44:36 DEBUG --- stderr --- 06:44:36 DEBUG 06:45:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814321" 06:45:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814321" 06:45:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712814321" 06:45:21 INFO [http_cmd]: http status code OK 06:45:21 DEBUG --- status code --- 06:45:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:21 DEBUG --- http response --- 06:45:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:45:21 DEBUG ----- output ----- 06:45:21 DEBUG { 06:45:21 DEBUG "status": "success", 06:45:21 DEBUG "data": { 06:45:21 DEBUG "resultType": "vector", 06:45:21 DEBUG "result": [ 06:45:21 DEBUG { 06:45:21 DEBUG "metric": { 06:45:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:45:21 DEBUG }, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG [...] 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:45:21 DEBUG }, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 1712814321, 06:45:21 DEBUG "4.121170489632456" 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG } 06:45:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814321" 06:45:21 ERROR [http_cmd]: ERROR 06:45:21 ERROR --- status code --- 06:45:21 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:45:21 ERROR --- http response --- 06:45:21 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:45:21 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:45:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814321" 06:45:21 INFO [http_cmd]: http status code OK 06:45:21 DEBUG --- status code --- 06:45:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:21 DEBUG --- http response --- 06:45:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814321" 06:45:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:45:21 DEBUG ----- output ----- 06:45:21 DEBUG { 06:45:21 DEBUG "status": "success", 06:45:21 DEBUG "data": { 06:45:21 DEBUG "resultType": "vector", 06:45:21 DEBUG "result": [ 06:45:21 DEBUG { 06:45:21 DEBUG "metric": { 06:45:21 DEBUG "instance": "10.142.0.107:9100" 06:45:21 DEBUG }, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG [...] 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG "instance": "10.142.0.77:9100" 06:45:21 DEBUG }, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 1712814321, 06:45:21 DEBUG "82832133.56666666" 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG } 06:45:21 INFO [http_cmd]: http status code OK 06:45:21 INFO [http_cmd]: http status code OK 06:45:21 DEBUG --- status code --- 06:45:21 DEBUG --- status code --- 06:45:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:21 DEBUG --- http response --- 06:45:21 DEBUG --- http response --- 06:45:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:45:21 DEBUG ----- output ----- 06:45:21 DEBUG { 06:45:21 DEBUG "status": "success", 06:45:21 DEBUG "data": { 06:45:21 DEBUG "resultType": "vector", 06:45:21 DEBUG "result": [ 06:45:21 DEBUG { 06:45:21 DEBUG "metric": { 06:45:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:45:21 DEBUG }, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG [...] 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:45:21 DEBUG }, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 1712814321, 06:45:21 DEBUG "1370.7666666666667" 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG } 06:45:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:45:21 DEBUG ----- output ----- 06:45:21 DEBUG { 06:45:21 DEBUG "status": "success", 06:45:21 DEBUG "data": { 06:45:21 DEBUG "resultType": "vector", 06:45:21 DEBUG "result": [ 06:45:21 DEBUG { 06:45:21 DEBUG "metric": { 06:45:21 DEBUG "instance": "10.142.0.107:9100" 06:45:21 DEBUG }, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG [...] 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG "instance": "10.142.0.77:9100" 06:45:21 DEBUG }, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 1712814321, 06:45:21 DEBUG "0.0027000000000001025" 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG } 06:45:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814321" 06:45:21 INFO [http_cmd]: http status code OK 06:45:21 DEBUG --- status code --- 06:45:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:21 DEBUG --- http response --- 06:45:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:45:21 DEBUG ----- output ----- 06:45:21 DEBUG { 06:45:21 DEBUG "status": "success", 06:45:21 DEBUG "data": { 06:45:21 DEBUG "resultType": "vector", 06:45:21 DEBUG "result": [ 06:45:21 DEBUG { 06:45:21 DEBUG "metric": { 06:45:21 DEBUG "instance": "10.142.0.107:9100" 06:45:21 DEBUG }, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG [...] 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG "instance": "10.142.0.77:9100" 06:45:21 DEBUG }, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 1712814321, 06:45:21 DEBUG "82834491.4" 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG } 06:45:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712814321" 06:45:21 INFO [http_cmd]: http status code OK 06:45:21 DEBUG --- status code --- 06:45:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:21 DEBUG --- http response --- 06:45:21 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:45:21 INFO [http_cmd]: http status code OK 06:45:21 DEBUG --- status code --- 06:45:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:21 DEBUG --- http response --- 06:45:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:45:21 DEBUG ----- output ----- 06:45:21 DEBUG { 06:45:21 DEBUG "status": "success", 06:45:21 DEBUG "data": { 06:45:21 DEBUG "resultType": "vector", 06:45:21 DEBUG "result": [ 06:45:21 DEBUG { 06:45:21 DEBUG "metric": { 06:45:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:45:21 DEBUG }, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG [...] 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:45:21 DEBUG }, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 1712814321, 06:45:21 DEBUG "0.021" 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG } 06:45:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814321" 06:45:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712814321" 06:45:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814321" 06:45:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814321" 06:45:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814321" 06:45:21 INFO [http_cmd]: http status code OK 06:45:21 DEBUG --- status code --- 06:45:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:21 DEBUG --- http response --- 06:45:21 INFO [http_cmd]: http status code OK 06:45:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:45:21 DEBUG --- status code --- 06:45:21 DEBUG ----- output ----- 06:45:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:21 DEBUG { 06:45:21 DEBUG --- http response --- 06:45:21 DEBUG "status": "success", 06:45:21 DEBUG "data": { 06:45:21 DEBUG "resultType": "vector", 06:45:21 DEBUG "result": [ 06:45:21 DEBUG { 06:45:21 DEBUG "metric": { 06:45:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:45:21 DEBUG }, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG [...] 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:45:21 DEBUG }, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 1712814321, 06:45:21 DEBUG "244803" 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG } 06:45:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:45:21 DEBUG ----- output ----- 06:45:21 DEBUG { 06:45:21 DEBUG "status": "success", 06:45:21 DEBUG "data": { 06:45:21 DEBUG "resultType": "vector", 06:45:21 DEBUG "result": [ 06:45:21 DEBUG { 06:45:21 DEBUG "metric": { 06:45:21 DEBUG "pod": "ds-idrepo-2" 06:45:21 DEBUG }, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG [...] 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG "pod": "ds-cts-2" 06:45:21 DEBUG }, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 1712814321, 06:45:21 DEBUG "0" 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG } 06:45:21 INFO [http_cmd]: http status code OK 06:45:21 DEBUG --- status code --- 06:45:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:21 DEBUG --- http response --- 06:45:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:45:21 DEBUG ----- output ----- 06:45:21 DEBUG { 06:45:21 DEBUG "status": "success", 06:45:21 DEBUG "data": { 06:45:21 DEBUG "resultType": "vector", 06:45:21 DEBUG "result": [ 06:45:21 DEBUG { 06:45:21 DEBUG "metric": { 06:45:21 DEBUG "pod": "ds-idrepo-2" 06:45:21 DEBUG }, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG [...] 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG "pod": "ds-idrepo-0" 06:45:21 DEBUG }, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 1712814321, 06:45:21 DEBUG "0" 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG } 06:45:21 INFO [http_cmd]: http status code OK 06:45:21 DEBUG --- status code --- 06:45:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:21 DEBUG --- http response --- 06:45:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:45:21 DEBUG ----- output ----- 06:45:21 DEBUG { 06:45:21 DEBUG "status": "success", 06:45:21 DEBUG "data": { 06:45:21 DEBUG "resultType": "vector", 06:45:21 DEBUG "result": [ 06:45:21 DEBUG { 06:45:21 DEBUG "metric": { 06:45:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:45:21 DEBUG }, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG [...] 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:45:21 DEBUG }, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 1712814321, 06:45:21 DEBUG "67489553.39805825" 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG } 06:45:21 INFO [http_cmd]: http status code OK 06:45:21 DEBUG --- status code --- 06:45:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:21 DEBUG --- http response --- 06:45:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:45:21 DEBUG ----- output ----- 06:45:21 DEBUG { 06:45:21 DEBUG "status": "success", 06:45:21 DEBUG "data": { 06:45:21 DEBUG "resultType": "vector", 06:45:21 DEBUG "result": [ 06:45:21 DEBUG { 06:45:21 DEBUG "metric": { 06:45:21 DEBUG "instance": "10.142.0.107:9100" 06:45:21 DEBUG }, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG [...] 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG "instance": "10.142.0.77:9100" 06:45:21 DEBUG }, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 1712814321, 06:45:21 DEBUG "52954014720" 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG } 06:45:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814321" 06:45:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814321" 06:45:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814321" 06:45:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814321" 06:45:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712814321" 06:45:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814321" 06:45:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814321" 06:45:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814321" 06:45:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814321" 06:45:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814321" 06:45:21 INFO [http_cmd]: http status code OK 06:45:21 INFO [http_cmd]: http status code OK 06:45:21 INFO [http_cmd]: http status code OK 06:45:21 INFO [http_cmd]: http status code OK 06:45:21 DEBUG --- status code --- 06:45:21 INFO [http_cmd]: http status code OK 06:45:21 DEBUG --- status code --- 06:45:21 DEBUG --- status code --- 06:45:21 DEBUG --- status code --- 06:45:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:21 DEBUG --- status code --- 06:45:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:21 DEBUG --- http response --- 06:45:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:21 DEBUG --- http response --- 06:45:21 DEBUG --- http response --- 06:45:21 DEBUG --- http response --- 06:45:21 INFO [http_cmd]: http status code OK 06:45:21 DEBUG --- http response --- 06:45:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:45:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:45:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:45:21 INFO [http_cmd]: http status code OK 06:45:21 INFO [http_cmd]: http status code OK 06:45:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:45:21 DEBUG --- status code --- 06:45:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:45:21 DEBUG ----- output ----- 06:45:21 DEBUG ----- output ----- 06:45:21 DEBUG ----- output ----- 06:45:21 DEBUG --- status code --- 06:45:21 DEBUG --- status code --- 06:45:21 DEBUG ----- output ----- 06:45:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:21 DEBUG ----- output ----- 06:45:21 DEBUG { 06:45:21 DEBUG { 06:45:21 DEBUG { 06:45:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:21 DEBUG { 06:45:21 DEBUG --- http response --- 06:45:21 INFO [http_cmd]: http status code OK 06:45:21 DEBUG { 06:45:21 INFO [http_cmd]: http status code OK 06:45:21 DEBUG "status": "success", 06:45:21 DEBUG "status": "success", 06:45:21 DEBUG "status": "success", 06:45:21 DEBUG --- http response --- 06:45:21 DEBUG --- http response --- 06:45:21 DEBUG "status": "success", 06:45:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:45:21 DEBUG --- status code --- 06:45:21 DEBUG "status": "success", 06:45:21 DEBUG --- status code --- 06:45:21 DEBUG "data": { 06:45:21 DEBUG "data": { 06:45:21 DEBUG "data": { 06:45:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:45:21 DEBUG "data": { 06:45:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:45:21 DEBUG ----- output ----- 06:45:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:21 DEBUG "data": { 06:45:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:21 DEBUG "resultType": "vector", 06:45:21 DEBUG "resultType": "vector", 06:45:21 DEBUG "resultType": "vector", 06:45:21 DEBUG ----- output ----- 06:45:21 DEBUG "resultType": "vector", 06:45:21 DEBUG ----- output ----- 06:45:21 DEBUG { 06:45:21 DEBUG --- http response --- 06:45:21 DEBUG "resultType": "vector", 06:45:21 DEBUG --- http response --- 06:45:21 DEBUG "result": [ 06:45:21 DEBUG "result": [ 06:45:21 DEBUG { 06:45:21 DEBUG { 06:45:21 DEBUG "result": [ 06:45:21 DEBUG { 06:45:21 DEBUG "status": "success", 06:45:21 DEBUG "result": [ 06:45:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:45:21 DEBUG { 06:45:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:45:21 DEBUG "metric": { 06:45:21 DEBUG "result": [ 06:45:21 DEBUG "status": "success", 06:45:21 DEBUG { 06:45:21 DEBUG "status": "success", 06:45:21 DEBUG "data": { 06:45:21 DEBUG { 06:45:21 DEBUG ----- output ----- 06:45:21 DEBUG "metric": { 06:45:21 DEBUG ----- output ----- 06:45:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:45:21 DEBUG { 06:45:21 DEBUG "data": { 06:45:21 DEBUG "metric": { 06:45:21 DEBUG "data": { 06:45:21 DEBUG "resultType": "vector", 06:45:21 DEBUG "metric": { 06:45:21 DEBUG { 06:45:21 DEBUG "pod": "ds-idrepo-2" 06:45:21 DEBUG { 06:45:21 DEBUG }, 06:45:21 DEBUG "metric": { 06:45:21 DEBUG "resultType": "vector", 06:45:21 DEBUG "pod": "ds-idrepo-2" 06:45:21 DEBUG "resultType": "vector", 06:45:21 DEBUG "result": [ 06:45:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:45:21 DEBUG "status": "success", 06:45:21 DEBUG }, 06:45:21 DEBUG "status": "success", 06:45:21 DEBUG "value": [ 06:45:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:45:21 DEBUG "result": [ 06:45:21 DEBUG }, 06:45:21 DEBUG "result": [ 06:45:21 DEBUG { 06:45:21 DEBUG }, 06:45:21 DEBUG "data": { 06:45:21 DEBUG "value": [ 06:45:21 DEBUG "data": { 06:45:21 DEBUG 06:45:21 DEBUG }, 06:45:21 DEBUG { 06:45:21 DEBUG "value": [ 06:45:21 DEBUG { 06:45:21 DEBUG "metric": { 06:45:21 DEBUG "value": [ 06:45:21 DEBUG "resultType": "vector", 06:45:21 DEBUG 06:45:21 DEBUG "resultType": "vector", 06:45:21 DEBUG 06:45:21 DEBUG "value": [ 06:45:21 DEBUG "metric": { 06:45:21 DEBUG 06:45:21 DEBUG "metric": { 06:45:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:45:21 DEBUG 06:45:21 DEBUG "result": [ 06:45:21 DEBUG 06:45:21 DEBUG "result": [ 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG "pod": "ds-idrepo-2" 06:45:21 DEBUG 06:45:21 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:45:21 DEBUG }, 06:45:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814321" 06:45:21 DEBUG 06:45:21 DEBUG { 06:45:21 DEBUG 06:45:21 DEBUG [...] 06:45:21 DEBUG [...] 06:45:21 DEBUG 06:45:21 DEBUG }, 06:45:21 DEBUG 06:45:21 DEBUG }, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 06:45:21 DEBUG "metric": { 06:45:21 DEBUG { 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG "value": [ 06:45:21 DEBUG "value": [ 06:45:21 DEBUG [...] 06:45:21 DEBUG 06:45:21 DEBUG [...] 06:45:21 DEBUG "pod": "ds-idrepo-2" 06:45:21 DEBUG "metric": { 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG [...] 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG }, 06:45:21 DEBUG "pod": "ds-idrepo-2" 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG "value": [ 06:45:21 DEBUG }, 06:45:21 DEBUG "pod": "ds-idrepo-0" 06:45:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG [...] 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG "value": [ 06:45:21 DEBUG }, 06:45:21 DEBUG }, 06:45:21 DEBUG 06:45:21 DEBUG [...] 06:45:21 DEBUG [...] 06:45:21 DEBUG "pod": "ds-cts-2" 06:45:21 DEBUG 06:45:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG "value": [ 06:45:21 DEBUG "value": [ 06:45:21 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG }, 06:45:21 DEBUG 06:45:21 DEBUG }, 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 1712814321, 06:45:21 DEBUG 1712814321, 06:45:21 DEBUG }, 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 06:45:21 DEBUG "value": [ 06:45:21 DEBUG [...] 06:45:21 DEBUG 06:45:21 DEBUG "0" 06:45:21 DEBUG "1371.1" 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 1712814321, 06:45:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:45:21 DEBUG 1712814321, 06:45:21 DEBUG 06:45:21 DEBUG [...] 06:45:21 DEBUG ] 06:45:21 DEBUG ] 06:45:21 DEBUG 1712814321, 06:45:21 DEBUG "pod": "ds-cts-2" 06:45:21 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 06:45:21 DEBUG "29.933333333333334" 06:45:21 DEBUG }, 06:45:21 DEBUG "0" 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG } 06:45:21 DEBUG } 06:45:21 DEBUG "0" 06:45:21 DEBUG }, 06:45:21 DEBUG }, 06:45:21 DEBUG ] 06:45:21 DEBUG "value": [ 06:45:21 DEBUG ] 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG ] 06:45:21 DEBUG ] 06:45:21 DEBUG ] 06:45:21 DEBUG "value": [ 06:45:21 DEBUG "value": [ 06:45:21 DEBUG } 06:45:21 DEBUG 1712814321, 06:45:21 DEBUG } 06:45:21 DEBUG "pod": "ds-idrepo-0" 06:45:21 DEBUG 06:45:21 DEBUG } 06:45:21 DEBUG } 06:45:21 DEBUG } 06:45:21 DEBUG 1712814321, 06:45:21 DEBUG 1712814321, 06:45:21 DEBUG ] 06:45:21 DEBUG "1359.3" 06:45:21 DEBUG ] 06:45:21 DEBUG }, 06:45:21 DEBUG "pod": "ds-cts-2" 06:45:21 DEBUG } 06:45:21 DEBUG } 06:45:21 DEBUG ] 06:45:21 DEBUG "0" 06:45:21 DEBUG "13806555136" 06:45:21 DEBUG } 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG "value": [ 06:45:21 DEBUG }, 06:45:21 DEBUG } 06:45:21 DEBUG ] 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG } 06:45:21 DEBUG } 06:45:21 DEBUG 1712814321, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 1712814321, 06:45:21 DEBUG } 06:45:21 DEBUG } 06:45:21 DEBUG ] 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG } 06:45:21 DEBUG "0.000030149444444444445" 06:45:21 DEBUG } 06:45:21 DEBUG "0" 06:45:21 DEBUG } 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG ] 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG } 06:45:21 DEBUG } 06:45:21 DEBUG ] 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG } 06:45:21 DEBUG } 06:45:21 DEBUG } 06:45:21 DEBUG } 06:45:21 INFO [http_cmd]: http status code OK 06:45:21 DEBUG --- status code --- 06:45:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:21 DEBUG --- http response --- 06:45:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:45:21 DEBUG ----- output ----- 06:45:21 DEBUG { 06:45:21 DEBUG "status": "success", 06:45:21 DEBUG "data": { 06:45:21 DEBUG "resultType": "vector", 06:45:21 DEBUG "result": [ 06:45:21 DEBUG { 06:45:21 DEBUG "metric": { 06:45:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:45:21 DEBUG }, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG [...] 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:45:21 DEBUG }, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 1712814321, 06:45:21 DEBUG "1365.2" 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG } 06:45:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814321" 06:45:21 INFO [http_cmd]: http status code OK 06:45:21 DEBUG --- status code --- 06:45:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:21 DEBUG --- http response --- 06:45:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:45:21 DEBUG ----- output ----- 06:45:21 DEBUG { 06:45:21 DEBUG "status": "success", 06:45:21 DEBUG "data": { 06:45:21 DEBUG "resultType": "vector", 06:45:21 DEBUG "result": [ 06:45:21 DEBUG { 06:45:21 DEBUG "metric": { 06:45:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:45:21 DEBUG }, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG [...] 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:45:21 DEBUG }, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 1712814321, 06:45:21 DEBUG "0" 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG } 06:45:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814321" 06:45:21 INFO [http_cmd]: http status code OK 06:45:21 DEBUG --- status code --- 06:45:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:21 DEBUG --- http response --- 06:45:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:45:21 DEBUG ----- output ----- 06:45:21 DEBUG { 06:45:21 DEBUG "status": "success", 06:45:21 DEBUG "data": { 06:45:21 DEBUG "resultType": "vector", 06:45:21 DEBUG "result": [ 06:45:21 DEBUG { 06:45:21 DEBUG "metric": { 06:45:21 DEBUG "instance": "10.142.0.107:9100" 06:45:21 DEBUG }, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG [...] 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG "instance": "10.142.0.77:9100" 06:45:21 DEBUG }, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 1712814321, 06:45:21 DEBUG "0.00033333333333350387" 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG } 06:45:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712814321" 06:45:21 INFO [http_cmd]: http status code OK 06:45:21 DEBUG --- status code --- 06:45:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:45:21 DEBUG --- http response --- 06:45:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:45:21 DEBUG ----- output ----- 06:45:21 DEBUG { 06:45:21 DEBUG "status": "success", 06:45:21 DEBUG "data": { 06:45:21 DEBUG "resultType": "vector", 06:45:21 DEBUG "result": [ 06:45:21 DEBUG { 06:45:21 DEBUG "metric": { 06:45:21 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:45:21 DEBUG }, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG [...] 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG 06:45:21 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:45:21 DEBUG }, 06:45:21 DEBUG "value": [ 06:45:21 DEBUG 1712814321, 06:45:21 DEBUG "4.153840046582586" 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG ] 06:45:21 DEBUG } 06:45:21 DEBUG } 06:45:32 INFO 06:45:32 INFO [loop_until]: kubectl --namespace=xlou top pods 06:45:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:45:32 INFO [loop_until]: OK (rc = 0) 06:45:32 DEBUG --- stdout --- 06:45:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 8891m 8509Mi am-5b5bd8567b-8h2w4 8737m 8565Mi am-5b5bd8567b-qlkw6 7527m 8461Mi ds-cts-0 8236m 13256Mi ds-cts-1 8926m 13384Mi ds-cts-2 8492m 13317Mi ds-idrepo-0 11691m 5503Mi ds-idrepo-1 351m 5480Mi ds-idrepo-2 367m 5783Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 3m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 2m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4176m 1857Mi 06:45:32 DEBUG --- stderr --- 06:45:32 DEBUG 06:45:36 INFO 06:45:36 INFO [loop_until]: kubectl --namespace=xlou top node 06:45:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:45:36 INFO [loop_until]: OK (rc = 0) 06:45:36 DEBUG --- stdout --- 06:45:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 2735Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 63m 0% 2792Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7966m 26% 10125Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10801m 36% 2982Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9600m 32% 10095Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 144m 0% 3861Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9727m 32% 10200Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 423m 2% 6464Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11624m 73% 6197Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8896m 55% 13985Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 7807m 49% 13921Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 8903m 56% 13987Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 404m 2% 6166Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4275m 26% 5613Mi 9% 06:45:36 DEBUG --- stderr --- 06:45:36 DEBUG 06:46:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814381" 06:46:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712814381" 06:46:21 INFO [http_cmd]: http status code OK 06:46:21 DEBUG --- status code --- 06:46:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:21 DEBUG --- http response --- 06:46:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:46:21 DEBUG ----- output ----- 06:46:21 DEBUG { 06:46:21 DEBUG "status": "success", 06:46:21 DEBUG "data": { 06:46:21 DEBUG "resultType": "vector", 06:46:21 DEBUG "result": [ 06:46:21 DEBUG { 06:46:21 DEBUG "metric": { 06:46:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:46:21 DEBUG }, 06:46:21 DEBUG "value": [ 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG [...] 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG "pod": "ds-idrepo-1" 06:46:21 DEBUG }, 06:46:21 DEBUG "value": [ 06:46:21 DEBUG 1712814381, 06:46:21 DEBUG "0.3413330296607552" 06:46:21 DEBUG ] 06:46:21 DEBUG } 06:46:21 DEBUG ] 06:46:21 DEBUG } 06:46:21 DEBUG } 06:46:21 ERROR [http_cmd]: ERROR 06:46:21 ERROR --- status code --- 06:46:21 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:46:21 ERROR --- http response --- 06:46:21 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:46:21 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:46:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814381" 06:46:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814381" 06:46:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814381" 06:46:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814381" 06:46:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814381" 06:46:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814381" 06:46:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712814381" 06:46:21 INFO [http_cmd]: http status code OK 06:46:21 DEBUG --- status code --- 06:46:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:21 DEBUG --- http response --- 06:46:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:46:21 DEBUG ----- output ----- 06:46:21 DEBUG { 06:46:21 DEBUG "status": "success", 06:46:21 DEBUG "data": { 06:46:21 DEBUG "resultType": "vector", 06:46:21 INFO [http_cmd]: http status code OK 06:46:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814381" 06:46:21 DEBUG "result": [ 06:46:21 INFO [http_cmd]: http status code OK 06:46:21 DEBUG --- status code --- 06:46:21 INFO [http_cmd]: http status code OK 06:46:21 DEBUG { 06:46:21 DEBUG --- status code --- 06:46:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:21 INFO [http_cmd]: http status code OK 06:46:21 DEBUG --- status code --- 06:46:21 DEBUG "metric": { 06:46:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:21 DEBUG --- http response --- 06:46:21 DEBUG --- status code --- 06:46:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:21 DEBUG "instance": "10.142.0.107:9100" 06:46:21 DEBUG --- http response --- 06:46:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:46:21 DEBUG --- http response --- 06:46:21 DEBUG }, 06:46:21 DEBUG --- http response --- 06:46:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:46:21 DEBUG ----- output ----- 06:46:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:46:21 DEBUG "value": [ 06:46:21 INFO [http_cmd]: http status code OK 06:46:21 DEBUG ----- output ----- 06:46:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:46:21 INFO [http_cmd]: http status code OK 06:46:21 DEBUG { 06:46:21 DEBUG ----- output ----- 06:46:21 DEBUG 06:46:21 DEBUG --- status code --- 06:46:21 DEBUG { 06:46:21 DEBUG ----- output ----- 06:46:21 DEBUG --- status code --- 06:46:21 DEBUG "status": "success", 06:46:21 DEBUG { 06:46:21 DEBUG 06:46:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:21 DEBUG "status": "success", 06:46:21 DEBUG { 06:46:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:21 DEBUG "data": { 06:46:21 DEBUG "status": "success", 06:46:21 DEBUG 06:46:21 DEBUG --- http response --- 06:46:21 DEBUG "data": { 06:46:21 DEBUG "status": "success", 06:46:21 DEBUG --- http response --- 06:46:21 DEBUG "resultType": "vector", 06:46:21 DEBUG "data": { 06:46:21 DEBUG [...] 06:46:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814381" 06:46:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:46:21 DEBUG "resultType": "vector", 06:46:21 INFO [http_cmd]: http status code OK 06:46:21 DEBUG "data": { 06:46:21 DEBUG "result": [ 06:46:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:46:21 DEBUG "resultType": "vector", 06:46:21 DEBUG 06:46:21 DEBUG ----- output ----- 06:46:21 DEBUG "result": [ 06:46:21 DEBUG --- status code --- 06:46:21 DEBUG "resultType": "vector", 06:46:21 DEBUG { 06:46:21 DEBUG ----- output ----- 06:46:21 DEBUG "result": [ 06:46:21 DEBUG 06:46:21 DEBUG { 06:46:21 DEBUG { 06:46:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:21 DEBUG "result": [ 06:46:21 DEBUG "metric": { 06:46:21 DEBUG { 06:46:21 DEBUG { 06:46:21 DEBUG 06:46:21 DEBUG "status": "success", 06:46:21 DEBUG "metric": { 06:46:21 DEBUG --- http response --- 06:46:21 DEBUG { 06:46:21 DEBUG "instance": "10.142.0.107:9100" 06:46:21 DEBUG "status": "success", 06:46:21 DEBUG "metric": { 06:46:21 DEBUG "instance": "10.142.0.77:9100" 06:46:21 DEBUG "data": { 06:46:21 DEBUG "instance": "10.142.0.107:9100" 06:46:21 DEBUG "metric": { 06:46:21 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:46:21 DEBUG }, 06:46:21 DEBUG "data": { 06:46:21 DEBUG "instance": "10.142.0.107:9100" 06:46:21 DEBUG }, 06:46:21 DEBUG "resultType": "vector", 06:46:21 DEBUG }, 06:46:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:46:21 DEBUG "value": [ 06:46:21 DEBUG "resultType": "vector", 06:46:21 DEBUG }, 06:46:21 DEBUG "value": [ 06:46:21 DEBUG "result": [ 06:46:21 DEBUG "value": [ 06:46:21 DEBUG }, 06:46:21 DEBUG 06:46:21 DEBUG "result": [ 06:46:21 DEBUG "value": [ 06:46:21 DEBUG 1712814381, 06:46:21 DEBUG { 06:46:21 DEBUG 06:46:21 DEBUG "value": [ 06:46:21 DEBUG 06:46:21 DEBUG { 06:46:21 DEBUG 06:46:21 DEBUG "82568575.76666667" 06:46:21 DEBUG "metric": { 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG "metric": { 06:46:21 DEBUG 06:46:21 DEBUG ] 06:46:21 DEBUG "pod": "ds-idrepo-2" 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG [...] 06:46:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:46:21 DEBUG 06:46:21 DEBUG } 06:46:21 DEBUG }, 06:46:21 DEBUG [...] 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG }, 06:46:21 DEBUG [...] 06:46:21 DEBUG ] 06:46:21 DEBUG "value": [ 06:46:21 DEBUG 06:46:21 DEBUG [...] 06:46:21 DEBUG 06:46:21 DEBUG "value": [ 06:46:21 DEBUG 06:46:21 DEBUG } 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG } 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG "instance": "10.142.0.77:9100" 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG "instance": "10.142.0.77:9100" 06:46:21 DEBUG 06:46:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:46:21 DEBUG 06:46:21 DEBUG "instance": "10.142.0.77:9100" 06:46:21 DEBUG [...] 06:46:21 DEBUG }, 06:46:21 DEBUG }, 06:46:21 DEBUG }, 06:46:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814381" 06:46:21 DEBUG [...] 06:46:21 DEBUG }, 06:46:21 DEBUG 06:46:21 DEBUG "value": [ 06:46:21 DEBUG "value": [ 06:46:21 DEBUG "value": [ 06:46:21 DEBUG 06:46:21 DEBUG "value": [ 06:46:21 DEBUG 06:46:21 INFO [http_cmd]: http status code OK 06:46:21 DEBUG 1712814381, 06:46:21 DEBUG 1712814381, 06:46:21 DEBUG 1712814381, 06:46:21 DEBUG 06:46:21 DEBUG 1712814381, 06:46:21 DEBUG 06:46:21 DEBUG --- status code --- 06:46:21 DEBUG "0.002733333333333358" 06:46:21 DEBUG "52955552768" 06:46:21 DEBUG "1367.2333333333333" 06:46:21 DEBUG 06:46:21 DEBUG "82571215.26666667" 06:46:21 DEBUG "pod": "ds-idrepo-0" 06:46:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:21 DEBUG ] 06:46:21 DEBUG ] 06:46:21 DEBUG ] 06:46:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:46:21 DEBUG ] 06:46:21 DEBUG }, 06:46:21 DEBUG --- http response --- 06:46:21 DEBUG } 06:46:21 DEBUG } 06:46:21 DEBUG } 06:46:21 DEBUG }, 06:46:21 DEBUG } 06:46:21 DEBUG "value": [ 06:46:21 DEBUG ] 06:46:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:46:21 DEBUG ] 06:46:21 DEBUG ] 06:46:21 DEBUG "value": [ 06:46:21 DEBUG ] 06:46:21 DEBUG 1712814381, 06:46:21 DEBUG } 06:46:21 DEBUG ----- output ----- 06:46:21 DEBUG } 06:46:21 DEBUG } 06:46:21 DEBUG 1712814381, 06:46:21 DEBUG } 06:46:21 DEBUG "0" 06:46:21 DEBUG } 06:46:21 DEBUG { 06:46:21 DEBUG } 06:46:21 DEBUG } 06:46:21 DEBUG "0.021" 06:46:21 DEBUG } 06:46:21 DEBUG ] 06:46:21 DEBUG "status": "success", 06:46:21 DEBUG ] 06:46:21 DEBUG } 06:46:21 DEBUG "data": { 06:46:21 DEBUG } 06:46:21 DEBUG ] 06:46:21 DEBUG "resultType": "vector", 06:46:21 DEBUG ] 06:46:21 DEBUG } 06:46:21 DEBUG "result": [ 06:46:21 DEBUG } 06:46:21 DEBUG } 06:46:21 DEBUG { 06:46:21 DEBUG } 06:46:21 DEBUG "metric": { 06:46:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:46:21 DEBUG }, 06:46:21 DEBUG "value": [ 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG [...] 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:46:21 DEBUG }, 06:46:21 DEBUG "value": [ 06:46:21 DEBUG 1712814381, 06:46:21 DEBUG "245309.5" 06:46:21 DEBUG ] 06:46:21 DEBUG } 06:46:21 DEBUG ] 06:46:21 DEBUG } 06:46:21 DEBUG } 06:46:21 INFO [http_cmd]: http status code OK 06:46:21 DEBUG --- status code --- 06:46:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:21 DEBUG --- http response --- 06:46:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:46:21 DEBUG ----- output ----- 06:46:21 DEBUG { 06:46:21 DEBUG "status": "success", 06:46:21 DEBUG "data": { 06:46:21 DEBUG "resultType": "vector", 06:46:21 DEBUG "result": [ 06:46:21 DEBUG { 06:46:21 DEBUG "metric": { 06:46:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:46:21 DEBUG }, 06:46:21 DEBUG "value": [ 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG [...] 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG "pod": "ds-idrepo-1" 06:46:21 DEBUG }, 06:46:21 DEBUG "value": [ 06:46:21 DEBUG 1712814381, 06:46:21 DEBUG "5452.38582838419" 06:46:21 DEBUG ] 06:46:21 DEBUG } 06:46:21 DEBUG ] 06:46:21 DEBUG } 06:46:21 DEBUG } 06:46:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712814381" 06:46:21 INFO [http_cmd]: http status code OK 06:46:21 DEBUG --- status code --- 06:46:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:21 DEBUG --- http response --- 06:46:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:46:21 DEBUG ----- output ----- 06:46:21 DEBUG { 06:46:21 DEBUG "status": "success", 06:46:21 DEBUG "data": { 06:46:21 DEBUG "resultType": "vector", 06:46:21 DEBUG "result": [ 06:46:21 DEBUG { 06:46:21 DEBUG "metric": { 06:46:21 DEBUG "pod": "ds-idrepo-2" 06:46:21 DEBUG }, 06:46:21 DEBUG "value": [ 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG [...] 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG "pod": "ds-cts-2" 06:46:21 DEBUG }, 06:46:21 DEBUG "value": [ 06:46:21 DEBUG 1712814381, 06:46:21 DEBUG "0" 06:46:21 DEBUG ] 06:46:21 DEBUG } 06:46:21 DEBUG ] 06:46:21 DEBUG } 06:46:21 DEBUG } 06:46:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814381" 06:46:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814381" 06:46:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814381" 06:46:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814381" 06:46:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814381" 06:46:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814381" 06:46:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712814381" 06:46:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814381" 06:46:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814381" 06:46:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814381" 06:46:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814381" 06:46:21 INFO [http_cmd]: http status code OK 06:46:21 INFO [http_cmd]: http status code OK 06:46:21 INFO [http_cmd]: http status code OK 06:46:21 INFO [http_cmd]: http status code OK 06:46:21 DEBUG --- status code --- 06:46:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814381" 06:46:21 DEBUG --- status code --- 06:46:21 INFO [http_cmd]: http status code OK 06:46:21 DEBUG --- status code --- 06:46:21 DEBUG --- status code --- 06:46:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:21 DEBUG --- status code --- 06:46:21 INFO [http_cmd]: http status code OK 06:46:21 INFO [http_cmd]: http status code OK 06:46:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:21 INFO [http_cmd]: http status code OK 06:46:21 INFO [http_cmd]: http status code OK 06:46:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:21 DEBUG --- http response --- 06:46:21 DEBUG --- http response --- 06:46:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:21 DEBUG --- status code --- 06:46:21 DEBUG --- status code --- 06:46:21 DEBUG --- http response --- 06:46:21 DEBUG --- status code --- 06:46:21 DEBUG --- status code --- 06:46:21 DEBUG --- http response --- 06:46:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:46:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:46:21 INFO [http_cmd]: http status code OK 06:46:21 DEBUG --- http response --- 06:46:21 INFO [http_cmd]: http status code OK 06:46:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:46:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:21 DEBUG ----- output ----- 06:46:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:46:21 DEBUG ----- output ----- 06:46:21 DEBUG --- status code --- 06:46:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:46:21 DEBUG --- status code --- 06:46:21 DEBUG --- http response --- 06:46:21 DEBUG --- http response --- 06:46:21 DEBUG --- http response --- 06:46:21 DEBUG ----- output ----- 06:46:21 DEBUG --- http response --- 06:46:21 DEBUG { 06:46:21 DEBUG ----- output ----- 06:46:21 DEBUG { 06:46:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:21 DEBUG ----- output ----- 06:46:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:46:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:46:21 DEBUG { 06:46:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:46:21 DEBUG "status": "success", 06:46:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:46:21 DEBUG { 06:46:21 DEBUG "status": "success", 06:46:21 DEBUG --- http response --- 06:46:21 DEBUG { 06:46:21 DEBUG --- http response --- 06:46:21 DEBUG ----- output ----- 06:46:21 DEBUG ----- output ----- 06:46:21 DEBUG "status": "success", 06:46:21 DEBUG ----- output ----- 06:46:21 DEBUG "data": { 06:46:21 DEBUG ----- output ----- 06:46:21 DEBUG "status": "success", 06:46:21 DEBUG "data": { 06:46:21 DEBUG "status": "success", 06:46:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:46:21 DEBUG { 06:46:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:46:21 DEBUG { 06:46:21 DEBUG "data": { 06:46:21 DEBUG { 06:46:21 INFO [http_cmd]: http status code OK 06:46:21 DEBUG "resultType": "vector", 06:46:21 DEBUG { 06:46:21 DEBUG "data": { 06:46:21 DEBUG "resultType": "vector", 06:46:21 DEBUG "data": { 06:46:21 DEBUG ----- output ----- 06:46:21 DEBUG "status": "success", 06:46:21 DEBUG ----- output ----- 06:46:21 DEBUG "status": "success", 06:46:21 DEBUG "resultType": "vector", 06:46:21 DEBUG "status": "success", 06:46:21 DEBUG --- status code --- 06:46:21 DEBUG "result": [ 06:46:21 DEBUG "status": "success", 06:46:21 DEBUG "resultType": "vector", 06:46:21 DEBUG "result": [ 06:46:21 DEBUG "resultType": "vector", 06:46:21 DEBUG { 06:46:21 DEBUG "data": { 06:46:21 DEBUG { 06:46:21 DEBUG "data": { 06:46:21 DEBUG "result": [ 06:46:21 DEBUG "data": { 06:46:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:21 DEBUG { 06:46:21 DEBUG "data": { 06:46:21 DEBUG "result": [ 06:46:21 DEBUG { 06:46:21 DEBUG "result": [ 06:46:21 DEBUG "status": "success", 06:46:21 DEBUG "resultType": "vector", 06:46:21 DEBUG "status": "success", 06:46:21 DEBUG "resultType": "vector", 06:46:21 DEBUG { 06:46:21 DEBUG "resultType": "vector", 06:46:21 DEBUG --- http response --- 06:46:21 DEBUG "metric": { 06:46:21 DEBUG "resultType": "vector", 06:46:21 DEBUG { 06:46:21 DEBUG "metric": { 06:46:21 DEBUG { 06:46:21 DEBUG "data": { 06:46:21 DEBUG "result": [ 06:46:21 DEBUG "data": { 06:46:21 DEBUG "result": [ 06:46:21 DEBUG "metric": { 06:46:21 DEBUG "result": [ 06:46:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:46:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:46:21 DEBUG "result": [ 06:46:21 DEBUG "metric": { 06:46:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:46:21 DEBUG "metric": { 06:46:21 DEBUG "resultType": "vector", 06:46:21 DEBUG { 06:46:21 DEBUG "resultType": "vector", 06:46:21 DEBUG { 06:46:21 DEBUG "pod": "ds-idrepo-2" 06:46:21 DEBUG { 06:46:21 DEBUG }, 06:46:21 DEBUG ----- output ----- 06:46:21 DEBUG { 06:46:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:46:21 DEBUG }, 06:46:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:46:21 DEBUG "result": [ 06:46:21 DEBUG "metric": { 06:46:21 DEBUG "result": [ 06:46:21 DEBUG "metric": { 06:46:21 DEBUG }, 06:46:21 DEBUG "metric": { 06:46:21 DEBUG "value": [ 06:46:21 DEBUG { 06:46:21 DEBUG "metric": { 06:46:21 DEBUG }, 06:46:21 DEBUG "value": [ 06:46:21 DEBUG }, 06:46:21 DEBUG { 06:46:21 DEBUG "pod": "ds-idrepo-2" 06:46:21 DEBUG { 06:46:21 DEBUG "pod": "ds-idrepo-2" 06:46:21 DEBUG "value": [ 06:46:21 DEBUG "pod": "ds-idrepo-2" 06:46:21 DEBUG 06:46:21 DEBUG "status": "success", 06:46:21 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:46:21 DEBUG "value": [ 06:46:21 DEBUG 06:46:21 DEBUG "value": [ 06:46:21 DEBUG "metric": { 06:46:21 DEBUG }, 06:46:21 DEBUG "metric": { 06:46:21 DEBUG }, 06:46:21 DEBUG 06:46:21 DEBUG }, 06:46:21 DEBUG 06:46:21 DEBUG "data": { 06:46:21 DEBUG }, 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG "pod": "ds-idrepo-2" 06:46:21 DEBUG "value": [ 06:46:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:46:21 DEBUG "value": [ 06:46:21 DEBUG 06:46:21 DEBUG "value": [ 06:46:21 DEBUG 06:46:21 DEBUG "resultType": "vector", 06:46:21 DEBUG "value": [ 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG }, 06:46:21 DEBUG 06:46:21 DEBUG }, 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG [...] 06:46:21 DEBUG "result": [ 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG [...] 06:46:21 DEBUG 06:46:21 DEBUG "value": [ 06:46:21 DEBUG 06:46:21 DEBUG "value": [ 06:46:21 DEBUG 06:46:21 DEBUG [...] 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG { 06:46:21 DEBUG 06:46:21 DEBUG [...] 06:46:21 DEBUG 06:46:21 DEBUG [...] 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG "metric": { 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG [...] 06:46:21 DEBUG 06:46:21 DEBUG [...] 06:46:21 DEBUG 06:46:21 DEBUG [...] 06:46:21 DEBUG 06:46:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:46:21 DEBUG [...] 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:46:21 DEBUG }, 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG "pod": "ds-idrepo-1" 06:46:21 DEBUG 06:46:21 DEBUG [...] 06:46:21 DEBUG 06:46:21 DEBUG [...] 06:46:21 DEBUG 06:46:21 DEBUG "pod": "ds-idrepo-0" 06:46:21 DEBUG 06:46:21 DEBUG }, 06:46:21 DEBUG "value": [ 06:46:21 DEBUG 06:46:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:46:21 DEBUG }, 06:46:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG }, 06:46:21 DEBUG 06:46:21 DEBUG "pod": "ds-cts-2" 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG }, 06:46:21 DEBUG "value": [ 06:46:21 DEBUG }, 06:46:21 DEBUG 06:46:21 DEBUG "pod": "ds-idrepo-0" 06:46:21 DEBUG 06:46:21 DEBUG "pod": "ds-cts-2" 06:46:21 DEBUG "value": [ 06:46:21 DEBUG "value": [ 06:46:21 DEBUG 1712814381, 06:46:21 DEBUG 06:46:21 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 06:46:21 DEBUG "value": [ 06:46:21 DEBUG 1712814381, 06:46:21 DEBUG "value": [ 06:46:21 DEBUG 06:46:21 DEBUG }, 06:46:21 DEBUG 06:46:21 DEBUG }, 06:46:21 DEBUG 1712814381, 06:46:21 DEBUG }, 06:46:21 DEBUG "0" 06:46:21 DEBUG 06:46:21 DEBUG }, 06:46:21 DEBUG 1712814381, 06:46:21 DEBUG "2935.9000614376405" 06:46:21 DEBUG 1712814381, 06:46:21 DEBUG "pod": "ds-cts-2" 06:46:21 DEBUG "value": [ 06:46:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:46:21 DEBUG "value": [ 06:46:21 DEBUG 1712814381, 06:46:21 DEBUG "value": [ 06:46:21 DEBUG ] 06:46:21 DEBUG [...] 06:46:21 DEBUG "value": [ 06:46:21 DEBUG "1366.4666666666667" 06:46:21 DEBUG ] 06:46:21 DEBUG "1348.0666666666666" 06:46:21 DEBUG }, 06:46:21 DEBUG 1712814381, 06:46:21 DEBUG }, 06:46:21 DEBUG "0" 06:46:21 DEBUG "29.599999999999998" 06:46:21 DEBUG 1712814381, 06:46:21 DEBUG } 06:46:21 DEBUG 06:46:21 DEBUG 1712814381, 06:46:21 DEBUG ] 06:46:21 DEBUG } 06:46:21 DEBUG ] 06:46:21 DEBUG "value": [ 06:46:21 DEBUG "0" 06:46:21 DEBUG "value": [ 06:46:21 DEBUG ] 06:46:21 DEBUG ] 06:46:21 DEBUG "0.0004086094444444444" 06:46:21 DEBUG ] 06:46:21 DEBUG 06:46:21 DEBUG "14137614336" 06:46:21 DEBUG } 06:46:21 DEBUG ] 06:46:21 DEBUG } 06:46:21 DEBUG 1712814381, 06:46:21 DEBUG ] 06:46:21 DEBUG 1712814381, 06:46:21 DEBUG } 06:46:21 DEBUG } 06:46:21 DEBUG ] 06:46:21 DEBUG } 06:46:21 DEBUG 06:46:21 DEBUG ] 06:46:21 DEBUG ] 06:46:21 DEBUG } 06:46:21 DEBUG ] 06:46:21 DEBUG "0" 06:46:21 DEBUG } 06:46:21 DEBUG "1366.4" 06:46:21 DEBUG ] 06:46:21 DEBUG ] 06:46:21 DEBUG } 06:46:21 DEBUG } 06:46:21 DEBUG "pod": "ds-idrepo-1" 06:46:21 DEBUG } 06:46:21 DEBUG } 06:46:21 DEBUG } 06:46:21 DEBUG } 06:46:21 DEBUG } 06:46:21 DEBUG ] 06:46:21 DEBUG ] 06:46:21 DEBUG } 06:46:21 DEBUG } 06:46:21 DEBUG ] 06:46:21 DEBUG }, 06:46:21 DEBUG ] 06:46:21 DEBUG } 06:46:21 DEBUG ] 06:46:21 DEBUG } 06:46:21 DEBUG } 06:46:21 DEBUG } 06:46:21 DEBUG } 06:46:21 DEBUG "value": [ 06:46:21 DEBUG } 06:46:21 DEBUG } 06:46:21 DEBUG ] 06:46:21 DEBUG ] 06:46:21 DEBUG } 06:46:21 DEBUG } 06:46:21 DEBUG 1712814381, 06:46:21 DEBUG } 06:46:21 DEBUG } 06:46:21 DEBUG } 06:46:21 DEBUG } 06:46:21 DEBUG "0" 06:46:21 DEBUG } 06:46:21 DEBUG } 06:46:21 DEBUG ] 06:46:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814381" 06:46:21 DEBUG } 06:46:21 DEBUG ] 06:46:21 DEBUG } 06:46:21 DEBUG } 06:46:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712814381" 06:46:21 INFO [http_cmd]: http status code OK 06:46:21 DEBUG --- status code --- 06:46:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:21 DEBUG --- http response --- 06:46:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:46:21 DEBUG ----- output ----- 06:46:21 DEBUG { 06:46:21 DEBUG "status": "success", 06:46:21 DEBUG "data": { 06:46:21 DEBUG "resultType": "vector", 06:46:21 DEBUG "result": [ 06:46:21 DEBUG { 06:46:21 DEBUG "metric": { 06:46:21 DEBUG "instance": "10.142.0.107:9100" 06:46:21 DEBUG }, 06:46:21 DEBUG "value": [ 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG [...] 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG "instance": "10.142.0.77:9100" 06:46:21 DEBUG }, 06:46:21 DEBUG "value": [ 06:46:21 DEBUG 1712814381, 06:46:21 DEBUG "0" 06:46:21 DEBUG ] 06:46:21 DEBUG } 06:46:21 DEBUG ] 06:46:21 DEBUG } 06:46:21 DEBUG } 06:46:21 INFO [http_cmd]: http status code OK 06:46:21 DEBUG --- status code --- 06:46:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:46:21 DEBUG --- http response --- 06:46:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:46:21 DEBUG ----- output ----- 06:46:21 DEBUG { 06:46:21 DEBUG "status": "success", 06:46:21 DEBUG "data": { 06:46:21 DEBUG "resultType": "vector", 06:46:21 DEBUG "result": [ 06:46:21 DEBUG { 06:46:21 DEBUG "metric": { 06:46:21 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:46:21 DEBUG }, 06:46:21 DEBUG "value": [ 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG [...] 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG 06:46:21 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:46:21 DEBUG }, 06:46:21 DEBUG "value": [ 06:46:21 DEBUG 1712814381, 06:46:21 DEBUG "4.142350478861563" 06:46:21 DEBUG ] 06:46:21 DEBUG } 06:46:21 DEBUG ] 06:46:21 DEBUG } 06:46:21 DEBUG } 06:46:32 INFO 06:46:32 INFO [loop_until]: kubectl --namespace=xlou top pods 06:46:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:46:32 INFO [loop_until]: OK (rc = 0) 06:46:32 DEBUG --- stdout --- 06:46:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 8941m 8512Mi am-5b5bd8567b-8h2w4 8690m 8565Mi am-5b5bd8567b-qlkw6 7516m 8462Mi ds-cts-0 8130m 13393Mi ds-cts-1 8712m 12796Mi ds-cts-2 8499m 13394Mi ds-idrepo-0 11433m 5504Mi ds-idrepo-1 430m 5480Mi ds-idrepo-2 347m 5783Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 5m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 5m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4056m 1858Mi 06:46:32 DEBUG --- stderr --- 06:46:32 DEBUG 06:46:36 INFO 06:46:36 INFO [loop_until]: kubectl --namespace=xlou top node 06:46:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:46:36 INFO [loop_until]: OK (rc = 0) 06:46:36 DEBUG --- stdout --- 06:46:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 2733Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 61m 0% 2782Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7870m 26% 10112Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10712m 35% 2994Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9572m 32% 10100Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 143m 0% 3854Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9616m 32% 10196Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 406m 2% 6461Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11137m 70% 6205Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8447m 53% 13998Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 8420m 52% 14017Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 8758m 55% 13382Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 449m 2% 6157Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4102m 25% 5615Mi 9% 06:46:36 DEBUG --- stderr --- 06:46:36 DEBUG 06:47:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814441" 06:47:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712814441" 06:47:21 ERROR [http_cmd]: ERROR 06:47:21 INFO [http_cmd]: http status code OK 06:47:21 ERROR --- status code --- 06:47:21 DEBUG --- status code --- 06:47:21 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:47:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:21 ERROR --- http response --- 06:47:21 DEBUG --- http response --- 06:47:21 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:47:21 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:47:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:47:21 DEBUG ----- output ----- 06:47:21 DEBUG { 06:47:21 DEBUG "status": "success", 06:47:21 DEBUG "data": { 06:47:21 DEBUG "resultType": "vector", 06:47:21 DEBUG "result": [ 06:47:21 DEBUG { 06:47:21 DEBUG "metric": { 06:47:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:47:21 DEBUG }, 06:47:21 DEBUG "value": [ 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG [...] 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG "pod": "ds-cts-1" 06:47:21 DEBUG }, 06:47:21 DEBUG "value": [ 06:47:21 DEBUG 1712814441, 06:47:21 DEBUG "5.759799751159022" 06:47:21 DEBUG ] 06:47:21 DEBUG } 06:47:21 DEBUG ] 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814441" 06:47:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814441" 06:47:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712814441" 06:47:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814441" 06:47:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814441" 06:47:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814441" 06:47:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814441" 06:47:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814441" 06:47:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814441" 06:47:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814441" 06:47:21 INFO [http_cmd]: http status code OK 06:47:21 DEBUG --- status code --- 06:47:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:21 DEBUG --- http response --- 06:47:21 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:47:21 INFO [http_cmd]: http status code OK 06:47:21 DEBUG --- status code --- 06:47:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:21 DEBUG --- http response --- 06:47:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:47:21 INFO [http_cmd]: http status code OK 06:47:21 DEBUG ----- output ----- 06:47:21 DEBUG --- status code --- 06:47:21 DEBUG { 06:47:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:21 DEBUG "status": "success", 06:47:21 DEBUG --- http response --- 06:47:21 DEBUG "data": { 06:47:21 DEBUG "resultType": "vector", 06:47:21 DEBUG "result": [ 06:47:21 DEBUG { 06:47:21 DEBUG "metric": { 06:47:21 DEBUG "instance": "10.142.0.107:9100" 06:47:21 DEBUG }, 06:47:21 DEBUG "value": [ 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG [...] 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG "instance": "10.142.0.77:9100" 06:47:21 DEBUG }, 06:47:21 DEBUG "value": [ 06:47:21 DEBUG 1712814441, 06:47:21 DEBUG "82824499.43333334" 06:47:21 DEBUG ] 06:47:21 DEBUG } 06:47:21 DEBUG ] 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:47:21 DEBUG ----- output ----- 06:47:21 DEBUG { 06:47:21 DEBUG "status": "success", 06:47:21 DEBUG "data": { 06:47:21 DEBUG "resultType": "vector", 06:47:21 DEBUG "result": [ 06:47:21 DEBUG { 06:47:21 DEBUG "metric": { 06:47:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:47:21 DEBUG }, 06:47:21 DEBUG "value": [ 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG [...] 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:47:21 DEBUG }, 06:47:21 DEBUG "value": [ 06:47:21 DEBUG 1712814441, 06:47:21 DEBUG "0.021" 06:47:21 DEBUG ] 06:47:21 DEBUG } 06:47:21 DEBUG ] 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 INFO [http_cmd]: http status code OK 06:47:21 INFO [http_cmd]: http status code OK 06:47:21 INFO [http_cmd]: http status code OK 06:47:21 INFO [http_cmd]: http status code OK 06:47:21 DEBUG --- status code --- 06:47:21 DEBUG --- status code --- 06:47:21 DEBUG --- status code --- 06:47:21 DEBUG --- status code --- 06:47:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:21 INFO [http_cmd]: http status code OK 06:47:21 DEBUG --- http response --- 06:47:21 INFO [http_cmd]: http status code OK 06:47:21 DEBUG --- http response --- 06:47:21 DEBUG --- http response --- 06:47:21 DEBUG --- http response --- 06:47:21 DEBUG --- status code --- 06:47:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:47:21 DEBUG --- status code --- 06:47:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:47:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:47:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:47:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:21 INFO [http_cmd]: http status code OK 06:47:21 DEBUG ----- output ----- 06:47:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:21 DEBUG ----- output ----- 06:47:21 DEBUG ----- output ----- 06:47:21 DEBUG ----- output ----- 06:47:21 DEBUG --- http response --- 06:47:21 DEBUG --- status code --- 06:47:21 DEBUG { 06:47:21 DEBUG --- http response --- 06:47:21 DEBUG { 06:47:21 DEBUG { 06:47:21 DEBUG { 06:47:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:47:21 DEBUG "status": "success", 06:47:21 DEBUG "status": "success", 06:47:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:47:21 DEBUG "status": "success", 06:47:21 DEBUG "status": "success", 06:47:21 DEBUG --- http response --- 06:47:21 DEBUG ----- output ----- 06:47:21 DEBUG "data": { 06:47:21 DEBUG "data": { 06:47:21 DEBUG ----- output ----- 06:47:21 DEBUG "data": { 06:47:21 DEBUG "data": { 06:47:21 DEBUG { 06:47:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:47:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712814441" 06:47:21 DEBUG "resultType": "vector", 06:47:21 DEBUG "resultType": "vector", 06:47:21 DEBUG { 06:47:21 DEBUG "resultType": "vector", 06:47:21 DEBUG "resultType": "vector", 06:47:21 DEBUG "status": "success", 06:47:21 DEBUG ----- output ----- 06:47:21 DEBUG "result": [ 06:47:21 DEBUG "result": [ 06:47:21 DEBUG "status": "success", 06:47:21 DEBUG "result": [ 06:47:21 DEBUG "result": [ 06:47:21 DEBUG "data": { 06:47:21 DEBUG { 06:47:21 DEBUG { 06:47:21 DEBUG { 06:47:21 DEBUG "data": { 06:47:21 DEBUG { 06:47:21 DEBUG { 06:47:21 DEBUG "resultType": "vector", 06:47:21 DEBUG "status": "success", 06:47:21 DEBUG "metric": { 06:47:21 DEBUG "metric": { 06:47:21 DEBUG "resultType": "vector", 06:47:21 DEBUG "metric": { 06:47:21 DEBUG "metric": { 06:47:21 DEBUG "result": [ 06:47:21 DEBUG "data": { 06:47:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:47:21 DEBUG "instance": "10.142.0.107:9100" 06:47:21 DEBUG "result": [ 06:47:21 DEBUG "instance": "10.142.0.107:9100" 06:47:21 DEBUG "instance": "10.142.0.107:9100" 06:47:21 DEBUG { 06:47:21 DEBUG "resultType": "vector", 06:47:21 DEBUG }, 06:47:21 DEBUG }, 06:47:21 DEBUG { 06:47:21 DEBUG }, 06:47:21 DEBUG }, 06:47:21 DEBUG "metric": { 06:47:21 DEBUG "result": [ 06:47:21 DEBUG "value": [ 06:47:21 DEBUG "value": [ 06:47:21 DEBUG "metric": { 06:47:21 DEBUG "value": [ 06:47:21 DEBUG "value": [ 06:47:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:47:21 DEBUG { 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG "pod": "ds-idrepo-2" 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG }, 06:47:21 DEBUG "metric": { 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG }, 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG "value": [ 06:47:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG "value": [ 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG }, 06:47:21 DEBUG [...] 06:47:21 DEBUG [...] 06:47:21 DEBUG 06:47:21 DEBUG [...] 06:47:21 DEBUG [...] 06:47:21 DEBUG 06:47:21 DEBUG "value": [ 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG [...] 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG [...] 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:47:21 DEBUG "instance": "10.142.0.77:9100" 06:47:21 DEBUG 06:47:21 DEBUG "instance": "10.142.0.77:9100" 06:47:21 DEBUG "instance": "10.142.0.77:9100" 06:47:21 DEBUG 06:47:21 DEBUG [...] 06:47:21 DEBUG }, 06:47:21 DEBUG }, 06:47:21 DEBUG 06:47:21 DEBUG }, 06:47:21 DEBUG }, 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG "value": [ 06:47:21 DEBUG "value": [ 06:47:21 DEBUG 06:47:21 DEBUG "value": [ 06:47:21 DEBUG "value": [ 06:47:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:47:21 DEBUG 06:47:21 DEBUG 1712814441, 06:47:21 DEBUG 1712814441, 06:47:21 DEBUG "pod": "ds-idrepo-0" 06:47:21 DEBUG 1712814441, 06:47:21 DEBUG 1712814441, 06:47:21 DEBUG }, 06:47:21 DEBUG 06:47:21 DEBUG "1370.8" 06:47:21 DEBUG "82826892.93333334" 06:47:21 DEBUG }, 06:47:21 DEBUG "52957148160" 06:47:21 DEBUG "0.0026999999999996285" 06:47:21 DEBUG "value": [ 06:47:21 DEBUG "pod": "ds-cts-1" 06:47:21 DEBUG ] 06:47:21 DEBUG ] 06:47:21 DEBUG "value": [ 06:47:21 DEBUG ] 06:47:21 DEBUG ] 06:47:21 DEBUG 1712814441, 06:47:21 DEBUG }, 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 DEBUG 1712814441, 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 DEBUG "245177" 06:47:21 DEBUG "value": [ 06:47:21 DEBUG ] 06:47:21 DEBUG ] 06:47:21 DEBUG "0" 06:47:21 DEBUG ] 06:47:21 DEBUG ] 06:47:21 DEBUG ] 06:47:21 DEBUG 1712814441, 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 DEBUG ] 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 DEBUG "11414294.343470544" 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 DEBUG ] 06:47:21 DEBUG ] 06:47:21 DEBUG ] 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 INFO [http_cmd]: http status code OK 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 DEBUG ] 06:47:21 DEBUG --- status code --- 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:21 DEBUG } 06:47:21 DEBUG --- http response --- 06:47:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:47:21 DEBUG ----- output ----- 06:47:21 DEBUG { 06:47:21 DEBUG "status": "success", 06:47:21 DEBUG "data": { 06:47:21 DEBUG "resultType": "vector", 06:47:21 DEBUG "result": [ 06:47:21 DEBUG { 06:47:21 DEBUG "metric": { 06:47:21 DEBUG "pod": "ds-idrepo-2" 06:47:21 DEBUG }, 06:47:21 DEBUG "value": [ 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG [...] 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG "pod": "ds-cts-2" 06:47:21 DEBUG }, 06:47:21 DEBUG "value": [ 06:47:21 DEBUG 1712814441, 06:47:21 DEBUG "0" 06:47:21 DEBUG ] 06:47:21 DEBUG } 06:47:21 DEBUG ] 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814441" 06:47:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814441" 06:47:21 INFO [http_cmd]: http status code OK 06:47:21 DEBUG --- status code --- 06:47:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:21 DEBUG --- http response --- 06:47:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:47:21 DEBUG ----- output ----- 06:47:21 DEBUG { 06:47:21 DEBUG "status": "success", 06:47:21 DEBUG "data": { 06:47:21 DEBUG "resultType": "vector", 06:47:21 DEBUG "result": [ 06:47:21 DEBUG { 06:47:21 DEBUG "metric": { 06:47:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:47:21 DEBUG }, 06:47:21 DEBUG "value": [ 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG [...] 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:47:21 DEBUG }, 06:47:21 DEBUG "value": [ 06:47:21 DEBUG 1712814441, 06:47:21 DEBUG "1362.9333333333334" 06:47:21 DEBUG ] 06:47:21 DEBUG } 06:47:21 DEBUG ] 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 INFO [http_cmd]: http status code OK 06:47:21 DEBUG --- status code --- 06:47:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:21 DEBUG --- http response --- 06:47:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:47:21 DEBUG ----- output ----- 06:47:21 DEBUG { 06:47:21 DEBUG "status": "success", 06:47:21 DEBUG "data": { 06:47:21 DEBUG "resultType": "vector", 06:47:21 DEBUG "result": [ 06:47:21 DEBUG { 06:47:21 DEBUG "metric": { 06:47:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:47:21 DEBUG }, 06:47:21 DEBUG "value": [ 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG [...] 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG "pod": "ds-cts-1" 06:47:21 DEBUG }, 06:47:21 DEBUG "value": [ 06:47:21 DEBUG 1712814441, 06:47:21 DEBUG "23466418.97424281" 06:47:21 DEBUG ] 06:47:21 DEBUG } 06:47:21 DEBUG ] 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814441" 06:47:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814441" 06:47:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814441" 06:47:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814441" 06:47:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814441" 06:47:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814441" 06:47:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814441" 06:47:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712814441" 06:47:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814441" 06:47:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814441" 06:47:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814441" 06:47:21 INFO [http_cmd]: http status code OK 06:47:21 INFO [http_cmd]: http status code OK 06:47:21 INFO [http_cmd]: http status code OK 06:47:21 INFO [http_cmd]: http status code OK 06:47:21 INFO [http_cmd]: http status code OK 06:47:21 DEBUG --- status code --- 06:47:21 DEBUG --- status code --- 06:47:21 INFO [http_cmd]: http status code OK 06:47:21 INFO [http_cmd]: http status code OK 06:47:21 DEBUG --- status code --- 06:47:21 DEBUG --- status code --- 06:47:21 DEBUG --- status code --- 06:47:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:21 DEBUG --- status code --- 06:47:21 DEBUG --- status code --- 06:47:21 INFO [http_cmd]: http status code OK 06:47:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:21 DEBUG --- http response --- 06:47:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:21 DEBUG --- http response --- 06:47:21 DEBUG --- http response --- 06:47:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712814441" 06:47:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:21 INFO [http_cmd]: http status code OK 06:47:21 INFO [http_cmd]: http status code OK 06:47:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:21 DEBUG --- status code --- 06:47:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:21 DEBUG --- http response --- 06:47:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:47:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:47:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:47:21 DEBUG --- http response --- 06:47:21 DEBUG --- status code --- 06:47:21 DEBUG --- status code --- 06:47:21 DEBUG --- http response --- 06:47:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:21 DEBUG --- http response --- 06:47:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:47:21 DEBUG ----- output ----- 06:47:21 DEBUG ----- output ----- 06:47:21 DEBUG ----- output ----- 06:47:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:47:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:21 DEBUG --- http response --- 06:47:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:47:21 DEBUG ----- output ----- 06:47:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:47:21 DEBUG { 06:47:21 DEBUG { 06:47:21 DEBUG { 06:47:21 DEBUG --- http response --- 06:47:21 DEBUG ----- output ----- 06:47:21 DEBUG --- http response --- 06:47:21 DEBUG ----- output ----- 06:47:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:47:21 DEBUG { 06:47:21 DEBUG "status": "success", 06:47:21 DEBUG "status": "success", 06:47:21 DEBUG "status": "success", 06:47:21 DEBUG "status": "success", 06:47:21 DEBUG { 06:47:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:47:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:47:21 INFO [http_cmd]: http status code OK 06:47:21 DEBUG { 06:47:21 DEBUG ----- output ----- 06:47:21 DEBUG ----- output ----- 06:47:21 DEBUG "data": { 06:47:21 DEBUG "data": { 06:47:21 DEBUG "data": { 06:47:21 DEBUG "data": { 06:47:21 DEBUG "status": "success", 06:47:21 DEBUG ----- output ----- 06:47:21 DEBUG ----- output ----- 06:47:21 DEBUG --- status code --- 06:47:21 DEBUG "status": "success", 06:47:21 DEBUG { 06:47:21 DEBUG { 06:47:21 DEBUG "resultType": "vector", 06:47:21 DEBUG "resultType": "vector", 06:47:21 DEBUG "resultType": "vector", 06:47:21 DEBUG "resultType": "vector", 06:47:21 DEBUG "data": { 06:47:21 DEBUG { 06:47:21 DEBUG { 06:47:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:21 DEBUG "data": { 06:47:21 DEBUG "status": "success", 06:47:21 DEBUG "status": "success", 06:47:21 DEBUG "result": [ 06:47:21 DEBUG "result": [ 06:47:21 DEBUG "result": [ 06:47:21 DEBUG "result": [ 06:47:21 DEBUG "resultType": "vector", 06:47:21 DEBUG "status": "success", 06:47:21 DEBUG "status": "success", 06:47:21 DEBUG --- http response --- 06:47:21 DEBUG "resultType": "vector", 06:47:21 DEBUG "data": { 06:47:21 DEBUG "data": { 06:47:21 DEBUG "resultType": "vector", 06:47:21 DEBUG { 06:47:21 DEBUG { 06:47:21 DEBUG { 06:47:21 DEBUG "result": [ 06:47:21 DEBUG "data": { 06:47:21 DEBUG "data": { 06:47:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:47:21 DEBUG "result": [ 06:47:21 DEBUG "resultType": "vector", 06:47:21 DEBUG "result": [ 06:47:21 DEBUG { 06:47:21 DEBUG "metric": { 06:47:21 DEBUG "metric": { 06:47:21 DEBUG "metric": { 06:47:21 DEBUG { 06:47:21 DEBUG "resultType": "vector", 06:47:21 DEBUG "resultType": "vector", 06:47:21 DEBUG ----- output ----- 06:47:21 DEBUG { 06:47:21 DEBUG "result": [ 06:47:21 DEBUG { 06:47:21 DEBUG "metric": { 06:47:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:47:21 DEBUG "pod": "ds-idrepo-2" 06:47:21 DEBUG "pod": "ds-idrepo-2" 06:47:21 DEBUG "metric": { 06:47:21 DEBUG "result": [ 06:47:21 DEBUG "result": [ 06:47:21 DEBUG { 06:47:21 DEBUG "metric": { 06:47:21 DEBUG { 06:47:21 DEBUG "metric": { 06:47:21 DEBUG "pod": "ds-idrepo-2" 06:47:21 DEBUG }, 06:47:21 DEBUG }, 06:47:21 DEBUG }, 06:47:21 DEBUG "pod": "ds-idrepo-2" 06:47:21 DEBUG { 06:47:21 DEBUG { 06:47:21 DEBUG "status": "success", 06:47:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:47:21 DEBUG "metric": { 06:47:21 DEBUG "pod": "ds-idrepo-2" 06:47:21 DEBUG }, 06:47:21 DEBUG "value": [ 06:47:21 DEBUG "value": [ 06:47:21 DEBUG "value": [ 06:47:21 DEBUG }, 06:47:21 DEBUG "metric": { 06:47:21 DEBUG "metric": { 06:47:21 DEBUG "data": { 06:47:21 DEBUG }, 06:47:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:47:21 DEBUG }, 06:47:21 DEBUG "value": [ 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 INFO [http_cmd]: http status code OK 06:47:21 DEBUG 06:47:21 DEBUG "value": [ 06:47:21 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:47:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:47:21 DEBUG "resultType": "vector", 06:47:21 DEBUG "value": [ 06:47:21 DEBUG }, 06:47:21 DEBUG "value": [ 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG --- status code --- 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG }, 06:47:21 DEBUG }, 06:47:21 DEBUG "result": [ 06:47:21 DEBUG 06:47:21 DEBUG "value": [ 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG "value": [ 06:47:21 DEBUG "value": [ 06:47:21 DEBUG { 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG [...] 06:47:21 DEBUG [...] 06:47:21 DEBUG --- http response --- 06:47:21 DEBUG [...] 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG "metric": { 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG [...] 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:47:21 DEBUG [...] 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG "instance": "10.142.0.107:9100" 06:47:21 DEBUG [...] 06:47:21 DEBUG 06:47:21 DEBUG [...] 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG ----- output ----- 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG }, 06:47:21 DEBUG 06:47:21 DEBUG [...] 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG { 06:47:21 DEBUG 06:47:21 DEBUG [...] 06:47:21 DEBUG [...] 06:47:21 DEBUG "value": [ 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:47:21 DEBUG "pod": "ds-cts-2" 06:47:21 DEBUG "pod": "ds-idrepo-0" 06:47:21 DEBUG "status": "success", 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:47:21 DEBUG 06:47:21 DEBUG "pod": "ds-cts-2" 06:47:21 DEBUG }, 06:47:21 DEBUG }, 06:47:21 DEBUG }, 06:47:21 DEBUG "data": { 06:47:21 DEBUG "pod": "ds-cts-2" 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG }, 06:47:21 DEBUG "pod": "ds-cts-1" 06:47:21 DEBUG }, 06:47:21 DEBUG "value": [ 06:47:21 DEBUG "value": [ 06:47:21 DEBUG "value": [ 06:47:21 DEBUG "resultType": "vector", 06:47:21 DEBUG }, 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG 06:47:21 DEBUG "value": [ 06:47:21 DEBUG }, 06:47:21 DEBUG "value": [ 06:47:21 DEBUG 1712814441, 06:47:21 DEBUG 1712814441, 06:47:21 DEBUG 1712814441, 06:47:21 DEBUG "result": [ 06:47:21 DEBUG "value": [ 06:47:21 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 06:47:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:47:21 DEBUG [...] 06:47:21 DEBUG "pod": "ds-idrepo-0" 06:47:21 DEBUG 1712814441, 06:47:21 DEBUG "value": [ 06:47:21 DEBUG 1712814441, 06:47:21 DEBUG "0" 06:47:21 DEBUG "29.466666666666665" 06:47:21 DEBUG "0" 06:47:21 DEBUG { 06:47:21 DEBUG 1712814441, 06:47:21 DEBUG }, 06:47:21 DEBUG }, 06:47:21 DEBUG 06:47:21 DEBUG }, 06:47:21 DEBUG "1364.1666666666667" 06:47:21 DEBUG 1712814441, 06:47:21 DEBUG "0" 06:47:21 DEBUG ] 06:47:21 DEBUG ] 06:47:21 DEBUG ] 06:47:21 DEBUG "metric": { 06:47:21 DEBUG "0" 06:47:21 DEBUG "value": [ 06:47:21 DEBUG "value": [ 06:47:21 DEBUG 06:47:21 DEBUG "value": [ 06:47:21 DEBUG ] 06:47:21 DEBUG "22674203.605849043" 06:47:21 DEBUG ] 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:47:21 DEBUG ] 06:47:21 DEBUG 1712814441, 06:47:21 DEBUG 1712814441, 06:47:21 DEBUG 06:47:21 DEBUG 1712814441, 06:47:21 DEBUG } 06:47:21 DEBUG ] 06:47:21 DEBUG } 06:47:21 DEBUG ] 06:47:21 DEBUG ] 06:47:21 DEBUG ] 06:47:21 DEBUG }, 06:47:21 DEBUG } 06:47:21 DEBUG "13286268928" 06:47:21 DEBUG "1370.8333333333333" 06:47:21 DEBUG "instance": "10.142.0.77:9100" 06:47:21 DEBUG "0" 06:47:21 DEBUG ] 06:47:21 DEBUG } 06:47:21 DEBUG ] 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 DEBUG "value": [ 06:47:21 DEBUG ] 06:47:21 DEBUG ] 06:47:21 DEBUG ] 06:47:21 DEBUG }, 06:47:21 DEBUG ] 06:47:21 DEBUG } 06:47:21 DEBUG ] 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 DEBUG 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 DEBUG "value": [ 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 DEBUG 06:47:21 DEBUG } 06:47:21 DEBUG ] 06:47:21 DEBUG 1712814441, 06:47:21 DEBUG ] 06:47:21 DEBUG } 06:47:21 DEBUG 06:47:21 DEBUG ] 06:47:21 DEBUG } 06:47:21 DEBUG "0.000666666666666534" 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 DEBUG [...] 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 DEBUG ] 06:47:21 DEBUG 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:21 DEBUG 06:47:21 DEBUG ] 06:47:21 DEBUG 06:47:21 DEBUG } 06:47:21 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:47:21 DEBUG } 06:47:21 DEBUG }, 06:47:21 DEBUG "value": [ 06:47:21 DEBUG 1712814441, 06:47:21 DEBUG "4.159160810040618" 06:47:21 DEBUG ] 06:47:21 DEBUG } 06:47:21 DEBUG ] 06:47:21 DEBUG } 06:47:21 DEBUG } 06:47:32 INFO 06:47:32 INFO [loop_until]: kubectl --namespace=xlou top pods 06:47:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:47:32 INFO [loop_until]: OK (rc = 0) 06:47:32 DEBUG --- stdout --- 06:47:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 8917m 8511Mi am-5b5bd8567b-8h2w4 8952m 8564Mi am-5b5bd8567b-qlkw6 7604m 8459Mi ds-cts-0 8027m 12877Mi ds-cts-1 8810m 13399Mi ds-cts-2 8256m 12927Mi ds-idrepo-0 11630m 5503Mi ds-idrepo-1 357m 5479Mi ds-idrepo-2 358m 5783Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 4m 1010Mi idm-668d898cfd-st2t7 10m 1135Mi lodemon-ccc964465-p7gdm 2m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4142m 1858Mi 06:47:32 DEBUG --- stderr --- 06:47:32 DEBUG 06:47:36 INFO 06:47:36 INFO [loop_until]: kubectl --namespace=xlou top node 06:47:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:47:36 INFO [loop_until]: OK (rc = 0) 06:47:36 DEBUG --- stdout --- 06:47:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 2727Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 68m 0% 2780Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7870m 26% 10116Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10801m 36% 2982Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9617m 32% 10100Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 142m 0% 3852Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9680m 32% 10186Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 410m 2% 6464Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11616m 73% 6208Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8562m 53% 13548Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 8019m 50% 13413Mi 22% gke-xlou-cdm-ds-32e4dcb1-j2hk 8941m 56% 14028Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 418m 2% 6157Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4265m 26% 5617Mi 9% 06:47:36 DEBUG --- stderr --- 06:47:36 DEBUG 06:48:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712814501" 06:48:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814501" 06:48:21 ERROR [http_cmd]: ERROR 06:48:21 ERROR --- status code --- 06:48:21 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:48:21 ERROR --- http response --- 06:48:21 INFO [http_cmd]: http status code OK 06:48:21 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:48:21 DEBUG --- status code --- 06:48:21 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:48:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:21 DEBUG --- http response --- 06:48:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 06:48:21 DEBUG ----- output ----- 06:48:21 DEBUG { 06:48:21 DEBUG "status": "success", 06:48:21 DEBUG "data": { 06:48:21 DEBUG "resultType": "vector", 06:48:21 DEBUG "result": [ 06:48:21 DEBUG { 06:48:21 DEBUG "metric": { 06:48:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:48:21 DEBUG }, 06:48:21 DEBUG "value": [ 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG [...] 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG "pod": "ds-cts-1" 06:48:21 DEBUG }, 06:48:21 DEBUG "value": [ 06:48:21 DEBUG 1712814501, 06:48:21 DEBUG "0" 06:48:21 DEBUG ] 06:48:21 DEBUG } 06:48:21 DEBUG ] 06:48:21 DEBUG } 06:48:21 DEBUG } 06:48:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814501" 06:48:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814501" 06:48:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814501" 06:48:21 INFO [http_cmd]: http status code OK 06:48:21 DEBUG --- status code --- 06:48:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:21 DEBUG --- http response --- 06:48:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814501" 06:48:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:48:21 DEBUG ----- output ----- 06:48:21 DEBUG { 06:48:21 DEBUG "status": "success", 06:48:21 DEBUG "data": { 06:48:21 DEBUG "resultType": "vector", 06:48:21 DEBUG "result": [ 06:48:21 DEBUG { 06:48:21 DEBUG "metric": { 06:48:21 DEBUG "instance": "10.142.0.107:9100" 06:48:21 DEBUG }, 06:48:21 DEBUG "value": [ 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG [...] 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG "instance": "10.142.0.77:9100" 06:48:21 DEBUG }, 06:48:21 DEBUG "value": [ 06:48:21 DEBUG 1712814501, 06:48:21 DEBUG "82580295.16666666" 06:48:21 DEBUG ] 06:48:21 DEBUG } 06:48:21 DEBUG ] 06:48:21 DEBUG } 06:48:21 DEBUG } 06:48:21 INFO [http_cmd]: http status code OK 06:48:21 DEBUG --- status code --- 06:48:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:21 DEBUG --- http response --- 06:48:21 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:48:21 INFO [http_cmd]: http status code OK 06:48:21 DEBUG --- status code --- 06:48:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:21 DEBUG --- http response --- 06:48:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:48:21 DEBUG ----- output ----- 06:48:21 DEBUG { 06:48:21 DEBUG "status": "success", 06:48:21 DEBUG "data": { 06:48:21 DEBUG "resultType": "vector", 06:48:21 DEBUG "result": [ 06:48:21 DEBUG { 06:48:21 DEBUG "metric": { 06:48:21 DEBUG "pod": "ds-idrepo-2" 06:48:21 DEBUG }, 06:48:21 DEBUG "value": [ 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG [...] 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG "pod": "ds-idrepo-0" 06:48:21 DEBUG }, 06:48:21 DEBUG "value": [ 06:48:21 DEBUG 1712814501, 06:48:21 DEBUG "0" 06:48:21 DEBUG ] 06:48:21 DEBUG } 06:48:21 DEBUG ] 06:48:21 DEBUG } 06:48:21 DEBUG } 06:48:21 INFO [http_cmd]: http status code OK 06:48:21 DEBUG --- status code --- 06:48:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:21 DEBUG --- http response --- 06:48:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:48:21 DEBUG ----- output ----- 06:48:21 DEBUG { 06:48:21 DEBUG "status": "success", 06:48:21 DEBUG "data": { 06:48:21 DEBUG "resultType": "vector", 06:48:21 DEBUG "result": [ 06:48:21 DEBUG { 06:48:21 DEBUG "metric": { 06:48:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:48:21 DEBUG }, 06:48:21 DEBUG "value": [ 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG [...] 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:48:21 DEBUG }, 06:48:21 DEBUG "value": [ 06:48:21 DEBUG 1712814501, 06:48:21 DEBUG "245355.5" 06:48:21 DEBUG ] 06:48:21 DEBUG } 06:48:21 DEBUG ] 06:48:21 DEBUG } 06:48:21 DEBUG } 06:48:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712814501" 06:48:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814501" 06:48:21 INFO [http_cmd]: http status code OK 06:48:21 DEBUG --- status code --- 06:48:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:21 DEBUG --- http response --- 06:48:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:48:21 DEBUG ----- output ----- 06:48:21 DEBUG { 06:48:21 DEBUG "status": "success", 06:48:21 DEBUG "data": { 06:48:21 DEBUG "resultType": "vector", 06:48:21 DEBUG "result": [ 06:48:21 DEBUG { 06:48:21 DEBUG "metric": { 06:48:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:48:21 DEBUG }, 06:48:21 DEBUG "value": [ 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG [...] 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:48:21 DEBUG }, 06:48:21 DEBUG "value": [ 06:48:21 DEBUG 1712814501, 06:48:21 DEBUG "0.021" 06:48:21 DEBUG ] 06:48:21 DEBUG } 06:48:21 DEBUG ] 06:48:21 DEBUG } 06:48:21 DEBUG } 06:48:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814501" 06:48:21 INFO [http_cmd]: http status code OK 06:48:21 DEBUG --- status code --- 06:48:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:21 DEBUG --- http response --- 06:48:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814501" 06:48:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814501" 06:48:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:48:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814501" 06:48:21 DEBUG ----- output ----- 06:48:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712814501" 06:48:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814501" 06:48:21 DEBUG { 06:48:21 DEBUG "status": "success", 06:48:21 DEBUG "data": { 06:48:21 DEBUG "resultType": "vector", 06:48:21 DEBUG "result": [ 06:48:21 DEBUG { 06:48:21 DEBUG "metric": { 06:48:21 DEBUG "instance": "10.142.0.107:9100" 06:48:21 DEBUG }, 06:48:21 DEBUG "value": [ 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG [...] 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG "instance": "10.142.0.77:9100" 06:48:21 DEBUG }, 06:48:21 DEBUG "value": [ 06:48:21 DEBUG 1712814501, 06:48:21 DEBUG "52958665728" 06:48:21 DEBUG ] 06:48:21 DEBUG } 06:48:21 DEBUG ] 06:48:21 DEBUG } 06:48:21 DEBUG } 06:48:21 INFO [http_cmd]: http status code OK 06:48:21 DEBUG --- status code --- 06:48:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:21 DEBUG --- http response --- 06:48:21 INFO [http_cmd]: http status code OK 06:48:21 INFO [http_cmd]: http status code OK 06:48:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:48:21 DEBUG --- status code --- 06:48:21 DEBUG --- status code --- 06:48:21 DEBUG ----- output ----- 06:48:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:21 DEBUG { 06:48:21 DEBUG --- http response --- 06:48:21 DEBUG --- http response --- 06:48:21 DEBUG "status": "success", 06:48:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:48:21 DEBUG "data": { 06:48:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:48:21 DEBUG ----- output ----- 06:48:21 DEBUG "resultType": "vector", 06:48:21 DEBUG ----- output ----- 06:48:21 DEBUG { 06:48:21 DEBUG "result": [ 06:48:21 DEBUG { 06:48:21 DEBUG "status": "success", 06:48:21 DEBUG { 06:48:21 DEBUG "status": "success", 06:48:21 DEBUG "data": { 06:48:21 DEBUG "metric": { 06:48:21 DEBUG "data": { 06:48:21 DEBUG "resultType": "vector", 06:48:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:48:21 DEBUG "resultType": "vector", 06:48:21 DEBUG "result": [ 06:48:21 DEBUG }, 06:48:21 DEBUG "result": [ 06:48:21 DEBUG { 06:48:21 DEBUG "value": [ 06:48:21 INFO [http_cmd]: http status code OK 06:48:21 DEBUG { 06:48:21 INFO [http_cmd]: http status code OK 06:48:21 DEBUG "metric": { 06:48:21 DEBUG 06:48:21 DEBUG --- status code --- 06:48:21 DEBUG "metric": { 06:48:21 DEBUG --- status code --- 06:48:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:48:21 DEBUG 06:48:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:21 DEBUG "instance": "10.142.0.107:9100" 06:48:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:21 DEBUG }, 06:48:21 DEBUG 06:48:21 DEBUG --- http response --- 06:48:21 DEBUG }, 06:48:21 DEBUG --- http response --- 06:48:21 DEBUG "value": [ 06:48:21 DEBUG [...] 06:48:21 DEBUG "value": [ 06:48:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:48:21 DEBUG 06:48:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG ----- output ----- 06:48:21 DEBUG 06:48:21 DEBUG ----- output ----- 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG { 06:48:21 DEBUG 06:48:21 DEBUG { 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG "status": "success", 06:48:21 DEBUG [...] 06:48:21 DEBUG "status": "success", 06:48:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:48:21 DEBUG [...] 06:48:21 DEBUG "data": { 06:48:21 DEBUG 06:48:21 DEBUG "data": { 06:48:21 DEBUG }, 06:48:21 DEBUG 06:48:21 DEBUG "resultType": "vector", 06:48:21 DEBUG 06:48:21 DEBUG "resultType": "vector", 06:48:21 DEBUG "value": [ 06:48:21 DEBUG 06:48:21 DEBUG "result": [ 06:48:21 INFO [http_cmd]: http status code OK 06:48:21 DEBUG 06:48:21 DEBUG "result": [ 06:48:21 DEBUG 1712814501, 06:48:21 DEBUG 06:48:21 DEBUG { 06:48:21 DEBUG --- status code --- 06:48:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:48:21 DEBUG { 06:48:21 DEBUG "1372.2333333333333" 06:48:21 DEBUG "instance": "10.142.0.77:9100" 06:48:21 DEBUG "metric": { 06:48:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:21 DEBUG }, 06:48:21 DEBUG "metric": { 06:48:21 DEBUG ] 06:48:21 DEBUG }, 06:48:21 DEBUG "instance": "10.142.0.107:9100" 06:48:21 DEBUG --- http response --- 06:48:21 DEBUG "value": [ 06:48:21 DEBUG "pod": "ds-idrepo-2" 06:48:21 DEBUG } 06:48:21 DEBUG "value": [ 06:48:21 DEBUG }, 06:48:21 DEBUG 1712814501, 06:48:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 06:48:21 DEBUG }, 06:48:21 DEBUG ] 06:48:21 DEBUG 1712814501, 06:48:21 DEBUG "value": [ 06:48:21 DEBUG "1367.0333333333333" 06:48:21 DEBUG ----- output ----- 06:48:21 DEBUG "value": [ 06:48:21 DEBUG } 06:48:21 DEBUG "82582637.76666667" 06:48:21 DEBUG 06:48:21 DEBUG ] 06:48:21 DEBUG { 06:48:21 DEBUG 06:48:21 DEBUG } 06:48:21 DEBUG ] 06:48:21 DEBUG 06:48:21 DEBUG } 06:48:21 DEBUG "status": "success", 06:48:21 DEBUG 06:48:21 DEBUG } 06:48:21 DEBUG 06:48:21 DEBUG ] 06:48:21 DEBUG "data": { 06:48:21 DEBUG 06:48:21 DEBUG ] 06:48:21 DEBUG [...] 06:48:21 DEBUG } 06:48:21 DEBUG "resultType": "vector", 06:48:21 DEBUG [...] 06:48:21 DEBUG } 06:48:21 DEBUG 06:48:21 DEBUG } 06:48:21 DEBUG "result": [ 06:48:21 DEBUG 06:48:21 DEBUG } 06:48:21 DEBUG 06:48:21 DEBUG { 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG "metric": { 06:48:21 DEBUG 06:48:21 DEBUG "instance": "10.142.0.77:9100" 06:48:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:48:21 DEBUG "pod": "ds-cts-2" 06:48:21 DEBUG }, 06:48:21 DEBUG }, 06:48:21 DEBUG }, 06:48:21 DEBUG "value": [ 06:48:21 DEBUG "value": [ 06:48:21 DEBUG "value": [ 06:48:21 DEBUG 1712814501, 06:48:21 DEBUG 06:48:21 DEBUG 1712814501, 06:48:21 DEBUG "0.0027666666666666135" 06:48:21 DEBUG 06:48:21 DEBUG "0" 06:48:21 DEBUG ] 06:48:21 DEBUG 06:48:21 DEBUG ] 06:48:21 DEBUG } 06:48:21 DEBUG [...] 06:48:21 DEBUG } 06:48:21 DEBUG ] 06:48:21 DEBUG 06:48:21 DEBUG ] 06:48:21 DEBUG } 06:48:21 DEBUG 06:48:21 DEBUG } 06:48:21 DEBUG } 06:48:21 DEBUG 06:48:21 DEBUG } 06:48:21 DEBUG "pod": "ds-cts-1" 06:48:21 DEBUG }, 06:48:21 DEBUG "value": [ 06:48:21 DEBUG 1712814501, 06:48:21 DEBUG "0" 06:48:21 DEBUG ] 06:48:21 DEBUG } 06:48:21 DEBUG ] 06:48:21 DEBUG } 06:48:21 DEBUG } 06:48:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814501" 06:48:21 INFO [http_cmd]: http status code OK 06:48:21 DEBUG --- status code --- 06:48:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:21 DEBUG --- http response --- 06:48:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 06:48:21 DEBUG ----- output ----- 06:48:21 DEBUG { 06:48:21 DEBUG "status": "success", 06:48:21 DEBUG "data": { 06:48:21 DEBUG "resultType": "vector", 06:48:21 DEBUG "result": [ 06:48:21 DEBUG { 06:48:21 DEBUG "metric": { 06:48:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:48:21 DEBUG }, 06:48:21 DEBUG "value": [ 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG [...] 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:48:21 DEBUG }, 06:48:21 DEBUG "value": [ 06:48:21 DEBUG 1712814501, 06:48:21 DEBUG "5075418.620227729" 06:48:21 DEBUG ] 06:48:21 DEBUG } 06:48:21 DEBUG ] 06:48:21 DEBUG } 06:48:21 DEBUG } 06:48:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814501" 06:48:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814501" 06:48:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814501" 06:48:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814501" 06:48:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814501" 06:48:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814501" 06:48:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814501" 06:48:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814501" 06:48:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712814501" 06:48:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712814501" 06:48:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814501" 06:48:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814501" 06:48:21 INFO [http_cmd]: http status code OK 06:48:21 INFO [http_cmd]: http status code OK 06:48:21 INFO [http_cmd]: http status code OK 06:48:21 DEBUG --- status code --- 06:48:21 INFO [http_cmd]: http status code OK 06:48:21 INFO [http_cmd]: http status code OK 06:48:21 DEBUG --- status code --- 06:48:21 INFO [http_cmd]: http status code OK 06:48:21 INFO [http_cmd]: http status code OK 06:48:21 DEBUG --- status code --- 06:48:21 INFO [http_cmd]: http status code OK 06:48:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:21 DEBUG --- status code --- 06:48:21 DEBUG --- status code --- 06:48:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:21 DEBUG --- status code --- 06:48:21 DEBUG --- status code --- 06:48:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:21 DEBUG --- status code --- 06:48:21 DEBUG --- http response --- 06:48:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:21 DEBUG --- http response --- 06:48:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:21 DEBUG --- http response --- 06:48:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:48:21 DEBUG --- http response --- 06:48:21 DEBUG --- http response --- 06:48:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:48:21 DEBUG --- http response --- 06:48:21 INFO [http_cmd]: http status code OK 06:48:21 INFO [http_cmd]: http status code OK 06:48:21 DEBUG --- http response --- 06:48:21 INFO [http_cmd]: http status code OK 06:48:21 DEBUG --- http response --- 06:48:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:48:21 DEBUG ----- output ----- 06:48:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:48:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:48:21 DEBUG ----- output ----- 06:48:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 06:48:21 DEBUG --- status code --- 06:48:21 DEBUG --- status code --- 06:48:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:48:21 DEBUG --- status code --- 06:48:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:48:21 DEBUG ----- output ----- 06:48:21 DEBUG { 06:48:21 DEBUG ----- output ----- 06:48:21 DEBUG ----- output ----- 06:48:21 DEBUG { 06:48:21 DEBUG ----- output ----- 06:48:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:21 DEBUG ----- output ----- 06:48:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:21 DEBUG ----- output ----- 06:48:21 DEBUG { 06:48:21 DEBUG "status": "success", 06:48:21 DEBUG { 06:48:21 DEBUG { 06:48:21 DEBUG "status": "success", 06:48:21 DEBUG { 06:48:21 DEBUG --- http response --- 06:48:21 DEBUG --- http response --- 06:48:21 DEBUG { 06:48:21 DEBUG --- http response --- 06:48:21 DEBUG { 06:48:21 DEBUG "status": "success", 06:48:21 DEBUG "data": { 06:48:21 DEBUG "status": "success", 06:48:21 DEBUG "status": "success", 06:48:21 DEBUG "data": { 06:48:21 DEBUG "status": "success", 06:48:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:48:21 DEBUG "status": "success", 06:48:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:48:21 DEBUG "status": "success", 06:48:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:48:21 DEBUG "data": { 06:48:21 DEBUG "resultType": "vector", 06:48:21 DEBUG "data": { 06:48:21 DEBUG "data": { 06:48:21 INFO [http_cmd]: http status code OK 06:48:21 DEBUG "resultType": "vector", 06:48:21 DEBUG "data": { 06:48:21 DEBUG ----- output ----- 06:48:21 DEBUG "data": { 06:48:21 DEBUG ----- output ----- 06:48:21 DEBUG "data": { 06:48:21 DEBUG ----- output ----- 06:48:21 DEBUG "resultType": "vector", 06:48:21 DEBUG "result": [ 06:48:21 DEBUG "resultType": "vector", 06:48:21 DEBUG "resultType": "vector", 06:48:21 DEBUG --- status code --- 06:48:21 DEBUG "result": [ 06:48:21 DEBUG "resultType": "vector", 06:48:21 DEBUG { 06:48:21 DEBUG "resultType": "vector", 06:48:21 DEBUG { 06:48:21 DEBUG "resultType": "vector", 06:48:21 DEBUG { 06:48:21 DEBUG "result": [ 06:48:21 DEBUG { 06:48:21 DEBUG "result": [ 06:48:21 DEBUG "result": [ 06:48:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:48:21 DEBUG { 06:48:21 DEBUG "result": [ 06:48:21 DEBUG "status": "success", 06:48:21 DEBUG "result": [ 06:48:21 DEBUG "status": "success", 06:48:21 DEBUG "result": [ 06:48:21 DEBUG "status": "success", 06:48:21 DEBUG { 06:48:21 DEBUG "metric": { 06:48:21 DEBUG { 06:48:21 DEBUG { 06:48:21 DEBUG --- http response --- 06:48:21 DEBUG "metric": { 06:48:21 DEBUG { 06:48:21 DEBUG "data": { 06:48:21 DEBUG { 06:48:21 DEBUG "data": { 06:48:21 DEBUG { 06:48:21 DEBUG "data": { 06:48:21 DEBUG "metric": { 06:48:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:48:21 DEBUG "metric": { 06:48:21 DEBUG "metric": { 06:48:21 DEBUG "pod": "ds-idrepo-2" 06:48:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:48:21 DEBUG "metric": { 06:48:21 DEBUG "resultType": "vector", 06:48:21 DEBUG "metric": { 06:48:21 DEBUG "resultType": "vector", 06:48:21 DEBUG "metric": { 06:48:21 DEBUG "resultType": "vector", 06:48:21 DEBUG "pod": "ds-idrepo-2" 06:48:21 DEBUG }, 06:48:21 DEBUG "pod": "ds-idrepo-2" 06:48:21 DEBUG "pod": "ds-idrepo-2" 06:48:21 DEBUG }, 06:48:21 DEBUG ----- output ----- 06:48:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:48:21 DEBUG "result": [ 06:48:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:48:21 DEBUG "result": [ 06:48:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:48:21 DEBUG "result": [ 06:48:21 DEBUG }, 06:48:21 DEBUG "value": [ 06:48:21 DEBUG }, 06:48:21 DEBUG }, 06:48:21 DEBUG "value": [ 06:48:21 DEBUG { 06:48:21 DEBUG }, 06:48:21 DEBUG { 06:48:21 DEBUG }, 06:48:21 DEBUG { 06:48:21 DEBUG }, 06:48:21 DEBUG { 06:48:21 DEBUG "value": [ 06:48:21 DEBUG 06:48:21 DEBUG "value": [ 06:48:21 DEBUG "value": [ 06:48:21 DEBUG 06:48:21 DEBUG "status": "success", 06:48:21 DEBUG "value": [ 06:48:21 DEBUG "metric": { 06:48:21 DEBUG "value": [ 06:48:21 DEBUG "metric": { 06:48:21 DEBUG "value": [ 06:48:21 DEBUG "metric": { 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG "data": { 06:48:21 DEBUG 06:48:21 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:48:21 DEBUG 06:48:21 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:48:21 DEBUG 06:48:21 DEBUG "pod": "ds-idrepo-2" 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG "resultType": "vector", 06:48:21 DEBUG 06:48:21 DEBUG }, 06:48:21 DEBUG 06:48:21 DEBUG }, 06:48:21 DEBUG 06:48:21 DEBUG }, 06:48:21 DEBUG 06:48:21 DEBUG [...] 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG [...] 06:48:21 DEBUG "result": [ 06:48:21 DEBUG 06:48:21 DEBUG "value": [ 06:48:21 DEBUG 06:48:21 DEBUG "value": [ 06:48:21 DEBUG 06:48:21 DEBUG "value": [ 06:48:21 DEBUG [...] 06:48:21 DEBUG 06:48:21 DEBUG [...] 06:48:21 DEBUG [...] 06:48:21 DEBUG 06:48:21 DEBUG { 06:48:21 DEBUG [...] 06:48:21 DEBUG 06:48:21 DEBUG [...] 06:48:21 DEBUG 06:48:21 DEBUG [...] 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG "metric": { 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG "instance": "10.142.0.107:9100" 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG "pod": "ds-cts-2" 06:48:21 DEBUG 06:48:21 DEBUG 06:48:21 DEBUG "pod": "ds-idrepo-0" 06:48:21 DEBUG }, 06:48:21 DEBUG 06:48:21 DEBUG [...] 06:48:21 DEBUG 06:48:21 DEBUG [...] 06:48:21 DEBUG 06:48:21 DEBUG [...] 06:48:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:48:21 DEBUG }, 06:48:21 DEBUG "pod": "ds-cts-2" 06:48:21 DEBUG "pod": "ds-cts-2" 06:48:21 DEBUG }, 06:48:21 DEBUG "value": [ 06:48:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:48:21 DEBUG 06:48:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:48:21 DEBUG 06:48:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:48:21 DEBUG 06:48:21 DEBUG }, 06:48:21 DEBUG "value": [ 06:48:21 DEBUG }, 06:48:21 DEBUG }, 06:48:21 DEBUG "value": [ 06:48:21 DEBUG 06:48:21 DEBUG }, 06:48:21 DEBUG 06:48:21 DEBUG }, 06:48:21 DEBUG 06:48:21 DEBUG }, 06:48:21 DEBUG 06:48:21 DEBUG "value": [ 06:48:21 DEBUG 1712814501, 06:48:21 DEBUG "value": [ 06:48:21 DEBUG "value": [ 06:48:21 DEBUG 1712814501, 06:48:21 DEBUG 06:48:21 DEBUG "value": [ 06:48:21 DEBUG 06:48:21 DEBUG "value": [ 06:48:21 DEBUG 06:48:21 DEBUG "value": [ 06:48:21 DEBUG 06:48:21 DEBUG 1712814501, 06:48:21 DEBUG "0.0005157244444444444" 06:48:21 DEBUG 1712814501, 06:48:21 DEBUG 1712814501, 06:48:21 DEBUG "0" 06:48:21 DEBUG 06:48:21 DEBUG 1712814501, 06:48:21 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 06:48:21 DEBUG 1712814501, 06:48:21 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:48:21 DEBUG 1712814501, 06:48:21 DEBUG "pod": "ds-idrepo-0" 06:48:21 DEBUG "0" 06:48:21 DEBUG ] 06:48:21 DEBUG "29.666666666666668" 06:48:21 DEBUG "0" 06:48:21 DEBUG ] 06:48:21 DEBUG [...] 06:48:21 DEBUG "0" 06:48:21 DEBUG }, 06:48:21 DEBUG "1366.3333333333333" 06:48:21 DEBUG }, 06:48:21 DEBUG "1366.7333333333333" 06:48:21 DEBUG }, 06:48:21 DEBUG ] 06:48:21 DEBUG } 06:48:21 DEBUG ] 06:48:21 DEBUG ] 06:48:21 DEBUG } 06:48:21 DEBUG 06:48:21 DEBUG ] 06:48:21 DEBUG "value": [ 06:48:21 DEBUG ] 06:48:21 DEBUG "value": [ 06:48:21 DEBUG ] 06:48:21 DEBUG "value": [ 06:48:21 DEBUG } 06:48:21 DEBUG ] 06:48:21 DEBUG } 06:48:21 DEBUG } 06:48:21 DEBUG ] 06:48:21 DEBUG 06:48:21 DEBUG } 06:48:21 DEBUG 1712814501, 06:48:21 DEBUG } 06:48:21 DEBUG 1712814501, 06:48:21 DEBUG } 06:48:21 DEBUG 1712814501, 06:48:21 DEBUG ] 06:48:21 DEBUG } 06:48:21 DEBUG ] 06:48:21 DEBUG ] 06:48:21 DEBUG } 06:48:21 DEBUG 06:48:21 DEBUG ] 06:48:21 DEBUG "13866647552" 06:48:21 DEBUG ] 06:48:21 DEBUG "4.133072388149656" 06:48:21 DEBUG ] 06:48:21 DEBUG "0" 06:48:21 DEBUG } 06:48:21 DEBUG } 06:48:21 DEBUG } 06:48:21 DEBUG } 06:48:21 DEBUG } 06:48:21 DEBUG "instance": "10.142.0.77:9100" 06:48:21 DEBUG } 06:48:21 DEBUG ] 06:48:21 DEBUG } 06:48:21 DEBUG ] 06:48:21 DEBUG } 06:48:21 DEBUG ] 06:48:21 DEBUG } 06:48:21 DEBUG } 06:48:21 DEBUG } 06:48:21 DEBUG }, 06:48:21 DEBUG "value": [ 06:48:21 DEBUG } 06:48:21 DEBUG } 06:48:21 DEBUG } 06:48:21 DEBUG } 06:48:21 DEBUG } 06:48:21 DEBUG } 06:48:21 DEBUG 1712814501, 06:48:21 DEBUG ] 06:48:21 DEBUG } 06:48:21 DEBUG ] 06:48:21 DEBUG "0.0003333333333333854" 06:48:21 DEBUG ] 06:48:21 DEBUG } 06:48:21 DEBUG } 06:48:21 DEBUG ] 06:48:21 DEBUG } 06:48:21 DEBUG } 06:48:21 DEBUG } 06:48:21 DEBUG } 06:48:21 DEBUG ] 06:48:21 DEBUG } 06:48:21 DEBUG } 06:48:32 INFO 06:48:32 INFO [loop_until]: kubectl --namespace=xlou top pods 06:48:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:48:33 INFO [loop_until]: OK (rc = 0) 06:48:33 DEBUG --- stdout --- 06:48:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 9050m 8513Mi am-5b5bd8567b-8h2w4 9214m 8565Mi am-5b5bd8567b-qlkw6 7558m 8461Mi ds-cts-0 8000m 13394Mi ds-cts-1 8726m 12874Mi ds-cts-2 8474m 13396Mi ds-idrepo-0 11680m 5503Mi ds-idrepo-1 379m 5480Mi ds-idrepo-2 367m 5784Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 4m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 2m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4177m 1859Mi 06:48:33 DEBUG --- stderr --- 06:48:33 DEBUG 06:48:36 INFO 06:48:36 INFO [loop_until]: kubectl --namespace=xlou top node 06:48:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:48:36 INFO [loop_until]: OK (rc = 0) 06:48:36 DEBUG --- stdout --- 06:48:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 2721Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 64m 0% 2781Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7969m 26% 10115Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10837m 36% 2983Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9591m 32% 10097Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 132m 0% 3857Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9653m 32% 10186Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 409m 2% 6463Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11510m 72% 6199Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8697m 54% 13983Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 8185m 51% 13999Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 8683m 54% 13563Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 450m 2% 6158Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4242m 26% 5615Mi 9% 06:48:36 DEBUG --- stderr --- 06:48:36 DEBUG 06:49:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712814561" 06:49:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814561" 06:49:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814561" 06:49:21 ERROR [http_cmd]: ERROR 06:49:21 ERROR --- status code --- 06:49:21 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:49:21 ERROR --- http response --- 06:49:21 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:49:21 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:49:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814561" 06:49:21 INFO [http_cmd]: http status code OK 06:49:21 DEBUG --- status code --- 06:49:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:21 DEBUG --- http response --- 06:49:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:49:21 DEBUG ----- output ----- 06:49:21 DEBUG { 06:49:21 DEBUG "status": "success", 06:49:21 DEBUG "data": { 06:49:21 DEBUG "resultType": "vector", 06:49:21 DEBUG "result": [ 06:49:21 DEBUG { 06:49:21 DEBUG "metric": { 06:49:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:49:21 DEBUG }, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG [...] 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG "pod": "ds-idrepo-1" 06:49:21 DEBUG }, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 1712814561, 06:49:21 DEBUG "0.35596389166301196" 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG } 06:49:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814561" 06:49:21 INFO [http_cmd]: http status code OK 06:49:21 INFO [http_cmd]: http status code OK 06:49:21 DEBUG --- status code --- 06:49:21 DEBUG --- status code --- 06:49:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:21 DEBUG --- http response --- 06:49:21 DEBUG --- http response --- 06:49:21 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:49:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:49:21 DEBUG ----- output ----- 06:49:21 DEBUG { 06:49:21 DEBUG "status": "success", 06:49:21 DEBUG "data": { 06:49:21 DEBUG "resultType": "vector", 06:49:21 DEBUG "result": [ 06:49:21 DEBUG { 06:49:21 DEBUG "metric": { 06:49:21 DEBUG "pod": "ds-idrepo-2" 06:49:21 DEBUG }, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG [...] 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG "pod": "ds-idrepo-0" 06:49:21 DEBUG }, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 1712814561, 06:49:21 DEBUG "0" 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG } 06:49:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814561" 06:49:21 INFO [http_cmd]: http status code OK 06:49:21 DEBUG --- status code --- 06:49:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:21 DEBUG --- http response --- 06:49:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:49:21 DEBUG ----- output ----- 06:49:21 DEBUG { 06:49:21 DEBUG "status": "success", 06:49:21 DEBUG "data": { 06:49:21 DEBUG "resultType": "vector", 06:49:21 DEBUG "result": [ 06:49:21 DEBUG { 06:49:21 DEBUG "metric": { 06:49:21 DEBUG "instance": "10.142.0.107:9100" 06:49:21 DEBUG }, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG [...] 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG "instance": "10.142.0.77:9100" 06:49:21 DEBUG }, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 1712814561, 06:49:21 DEBUG "82772088.23333333" 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG } 06:49:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814561" 06:49:21 INFO [http_cmd]: http status code OK 06:49:21 DEBUG --- status code --- 06:49:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:21 DEBUG --- http response --- 06:49:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814561" 06:49:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:49:21 DEBUG ----- output ----- 06:49:21 DEBUG { 06:49:21 DEBUG "status": "success", 06:49:21 DEBUG "data": { 06:49:21 DEBUG "resultType": "vector", 06:49:21 DEBUG "result": [ 06:49:21 DEBUG { 06:49:21 DEBUG "metric": { 06:49:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:49:21 DEBUG }, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG [...] 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:49:21 DEBUG }, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 1712814561, 06:49:21 DEBUG "245067.5" 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG } 06:49:21 INFO [http_cmd]: http status code OK 06:49:21 DEBUG --- status code --- 06:49:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:21 DEBUG --- http response --- 06:49:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:49:21 DEBUG ----- output ----- 06:49:21 DEBUG { 06:49:21 DEBUG "status": "success", 06:49:21 DEBUG "data": { 06:49:21 DEBUG "resultType": "vector", 06:49:21 DEBUG "result": [ 06:49:21 DEBUG { 06:49:21 DEBUG "metric": { 06:49:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:49:21 DEBUG }, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG [...] 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG "pod": "ds-idrepo-1" 06:49:21 DEBUG }, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 1712814561, 06:49:21 DEBUG "179604518.71758014" 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG } 06:49:21 INFO [http_cmd]: http status code OK 06:49:21 DEBUG --- status code --- 06:49:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:21 DEBUG --- http response --- 06:49:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:49:21 DEBUG ----- output ----- 06:49:21 DEBUG { 06:49:21 DEBUG "status": "success", 06:49:21 DEBUG "data": { 06:49:21 DEBUG "resultType": "vector", 06:49:21 DEBUG "result": [ 06:49:21 DEBUG { 06:49:21 DEBUG "metric": { 06:49:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:49:21 DEBUG }, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG [...] 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:49:21 DEBUG }, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 1712814561, 06:49:21 DEBUG "1367.7666666666667" 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG } 06:49:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712814561" 06:49:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814561" 06:49:21 INFO [http_cmd]: http status code OK 06:49:21 DEBUG --- status code --- 06:49:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:21 DEBUG --- http response --- 06:49:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:49:21 DEBUG ----- output ----- 06:49:21 DEBUG { 06:49:21 DEBUG "status": "success", 06:49:21 DEBUG "data": { 06:49:21 DEBUG "resultType": "vector", 06:49:21 DEBUG "result": [ 06:49:21 DEBUG { 06:49:21 DEBUG "metric": { 06:49:21 DEBUG "instance": "10.142.0.107:9100" 06:49:21 DEBUG }, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG [...] 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG "instance": "10.142.0.77:9100" 06:49:21 DEBUG }, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 1712814561, 06:49:21 DEBUG "52960214016" 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG } 06:49:21 INFO [http_cmd]: http status code OK 06:49:21 DEBUG --- status code --- 06:49:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:21 DEBUG --- http response --- 06:49:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:49:21 DEBUG ----- output ----- 06:49:21 DEBUG { 06:49:21 DEBUG "status": "success", 06:49:21 DEBUG "data": { 06:49:21 DEBUG "resultType": "vector", 06:49:21 DEBUG "result": [ 06:49:21 DEBUG { 06:49:21 DEBUG "metric": { 06:49:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:49:21 DEBUG }, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG [...] 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:49:21 DEBUG }, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 1712814561, 06:49:21 DEBUG "0.021" 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG } 06:49:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712814561" 06:49:21 INFO [http_cmd]: http status code OK 06:49:21 DEBUG --- status code --- 06:49:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:21 DEBUG --- http response --- 06:49:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:49:21 DEBUG ----- output ----- 06:49:21 DEBUG { 06:49:21 DEBUG "status": "success", 06:49:21 DEBUG "data": { 06:49:21 DEBUG "resultType": "vector", 06:49:21 DEBUG "result": [ 06:49:21 DEBUG { 06:49:21 DEBUG "metric": { 06:49:21 DEBUG "pod": "ds-idrepo-2" 06:49:21 DEBUG }, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG [...] 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG "pod": "ds-cts-2" 06:49:21 DEBUG }, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 1712814561, 06:49:21 DEBUG "0" 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG } 06:49:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814561" 06:49:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814561" 06:49:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814561" 06:49:21 INFO [http_cmd]: http status code OK 06:49:21 DEBUG --- status code --- 06:49:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:21 DEBUG --- http response --- 06:49:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:49:21 DEBUG ----- output ----- 06:49:21 DEBUG { 06:49:21 DEBUG "status": "success", 06:49:21 DEBUG "data": { 06:49:21 DEBUG "resultType": "vector", 06:49:21 DEBUG "result": [ 06:49:21 DEBUG { 06:49:21 DEBUG "metric": { 06:49:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:49:21 DEBUG }, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG [...] 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:49:21 DEBUG }, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 1712814561, 06:49:21 DEBUG "1358.5666666666666" 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG } 06:49:21 INFO [http_cmd]: http status code OK 06:49:21 DEBUG --- status code --- 06:49:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:21 DEBUG --- http response --- 06:49:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:49:21 DEBUG ----- output ----- 06:49:21 DEBUG { 06:49:21 DEBUG "status": "success", 06:49:21 DEBUG "data": { 06:49:21 DEBUG "resultType": "vector", 06:49:21 DEBUG "result": [ 06:49:21 DEBUG { 06:49:21 DEBUG "metric": { 06:49:21 DEBUG "instance": "10.142.0.107:9100" 06:49:21 DEBUG }, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG [...] 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG "instance": "10.142.0.77:9100" 06:49:21 DEBUG }, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 1712814561, 06:49:21 DEBUG "82774849.23333333" 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG } 06:49:21 INFO [http_cmd]: http status code OK 06:49:21 DEBUG --- status code --- 06:49:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:21 DEBUG --- http response --- 06:49:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:49:21 DEBUG ----- output ----- 06:49:21 DEBUG { 06:49:21 DEBUG "status": "success", 06:49:21 DEBUG "data": { 06:49:21 DEBUG "resultType": "vector", 06:49:21 DEBUG "result": [ 06:49:21 DEBUG { 06:49:21 DEBUG "metric": { 06:49:21 DEBUG "instance": "10.142.0.107:9100" 06:49:21 DEBUG }, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG [...] 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG "instance": "10.142.0.77:9100" 06:49:21 DEBUG }, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 1712814561, 06:49:21 DEBUG "0.0028333333333331245" 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG } 06:49:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814561" 06:49:21 INFO [http_cmd]: http status code OK 06:49:21 DEBUG --- status code --- 06:49:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:21 DEBUG --- http response --- 06:49:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:49:21 DEBUG ----- output ----- 06:49:21 DEBUG { 06:49:21 DEBUG "status": "success", 06:49:21 DEBUG "data": { 06:49:21 DEBUG "resultType": "vector", 06:49:21 DEBUG "result": [ 06:49:21 DEBUG { 06:49:21 DEBUG "metric": { 06:49:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:49:21 DEBUG }, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG [...] 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG "pod": "ds-idrepo-1" 06:49:21 DEBUG }, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 1712814561, 06:49:21 DEBUG "3071.8080119992496" 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG } 06:49:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814561" 06:49:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814561" 06:49:21 INFO [http_cmd]: http status code OK 06:49:21 DEBUG --- status code --- 06:49:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:21 DEBUG --- http response --- 06:49:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:49:21 DEBUG ----- output ----- 06:49:21 DEBUG { 06:49:21 DEBUG "status": "success", 06:49:21 DEBUG "data": { 06:49:21 DEBUG "resultType": "vector", 06:49:21 DEBUG "result": [ 06:49:21 DEBUG { 06:49:21 DEBUG "metric": { 06:49:21 DEBUG "pod": "ds-idrepo-2" 06:49:21 DEBUG }, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG [...] 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG "pod": "ds-cts-2" 06:49:21 DEBUG }, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 1712814561, 06:49:21 DEBUG "29.333333333333332" 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG } 06:49:21 INFO [http_cmd]: http status code OK 06:49:21 DEBUG --- status code --- 06:49:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:21 DEBUG --- http response --- 06:49:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:49:21 DEBUG ----- output ----- 06:49:21 DEBUG { 06:49:21 DEBUG "status": "success", 06:49:21 DEBUG "data": { 06:49:21 DEBUG "resultType": "vector", 06:49:21 DEBUG "result": [ 06:49:21 DEBUG { 06:49:21 DEBUG "metric": { 06:49:21 DEBUG "pod": "ds-idrepo-2" 06:49:21 DEBUG }, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG [...] 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG "pod": "ds-cts-2" 06:49:21 DEBUG }, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 1712814561, 06:49:21 DEBUG "0" 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG } 06:49:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814561" 06:49:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814561" 06:49:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814561" 06:49:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814561" 06:49:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814561" 06:49:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712814561" 06:49:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814561" 06:49:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814561" 06:49:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712814561" 06:49:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814561" 06:49:21 INFO [http_cmd]: http status code OK 06:49:21 INFO [http_cmd]: http status code OK 06:49:21 INFO [http_cmd]: http status code OK 06:49:21 INFO [http_cmd]: http status code OK 06:49:21 INFO [http_cmd]: http status code OK 06:49:21 DEBUG --- status code --- 06:49:21 DEBUG --- status code --- 06:49:21 DEBUG --- status code --- 06:49:21 DEBUG --- status code --- 06:49:21 DEBUG --- status code --- 06:49:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:21 DEBUG --- http response --- 06:49:21 INFO [http_cmd]: http status code OK 06:49:21 DEBUG --- http response --- 06:49:21 INFO [http_cmd]: http status code OK 06:49:21 DEBUG --- http response --- 06:49:21 DEBUG --- http response --- 06:49:21 DEBUG --- http response --- 06:49:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:49:21 DEBUG --- status code --- 06:49:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:49:21 DEBUG --- status code --- 06:49:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:49:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:49:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:49:21 DEBUG ----- output ----- 06:49:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:21 DEBUG --- http response --- 06:49:21 INFO [http_cmd]: http status code OK 06:49:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:21 DEBUG ----- output ----- 06:49:21 DEBUG ----- output ----- 06:49:21 DEBUG ----- output ----- 06:49:21 DEBUG { 06:49:21 DEBUG ----- output ----- 06:49:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:49:21 DEBUG --- status code --- 06:49:21 DEBUG --- http response --- 06:49:21 DEBUG { 06:49:21 DEBUG { 06:49:21 DEBUG { 06:49:21 DEBUG "status": "success", 06:49:21 DEBUG "data": { 06:49:21 DEBUG ----- output ----- 06:49:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:21 DEBUG "status": "success", 06:49:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:49:21 DEBUG "status": "success", 06:49:21 DEBUG "status": "success", 06:49:21 DEBUG { 06:49:21 DEBUG "resultType": "vector", 06:49:21 DEBUG { 06:49:21 DEBUG --- http response --- 06:49:21 DEBUG "data": { 06:49:21 DEBUG ----- output ----- 06:49:21 DEBUG "data": { 06:49:21 DEBUG "data": { 06:49:21 DEBUG "status": "success", 06:49:21 DEBUG "result": [ 06:49:21 DEBUG "status": "success", 06:49:21 DEBUG "resultType": "vector", 06:49:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:49:21 DEBUG { 06:49:21 DEBUG "resultType": "vector", 06:49:21 DEBUG "resultType": "vector", 06:49:21 DEBUG "data": { 06:49:21 DEBUG { 06:49:21 DEBUG "data": { 06:49:21 DEBUG "result": [ 06:49:21 DEBUG ----- output ----- 06:49:21 DEBUG "status": "success", 06:49:21 DEBUG "result": [ 06:49:21 DEBUG "result": [ 06:49:21 DEBUG "resultType": "vector", 06:49:21 DEBUG "metric": { 06:49:21 DEBUG "resultType": "vector", 06:49:21 DEBUG { 06:49:21 DEBUG { 06:49:21 DEBUG "data": { 06:49:21 DEBUG { 06:49:21 DEBUG { 06:49:21 DEBUG "result": [ 06:49:21 DEBUG "pod": "ds-idrepo-2" 06:49:21 DEBUG "result": [ 06:49:21 DEBUG "metric": { 06:49:21 DEBUG "status": "success", 06:49:21 INFO [http_cmd]: http status code OK 06:49:21 DEBUG "resultType": "vector", 06:49:21 DEBUG "metric": { 06:49:21 DEBUG "metric": { 06:49:21 DEBUG { 06:49:21 DEBUG }, 06:49:21 DEBUG { 06:49:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:49:21 DEBUG "data": { 06:49:21 DEBUG --- status code --- 06:49:21 DEBUG "result": [ 06:49:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:49:21 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:49:21 DEBUG "metric": { 06:49:21 DEBUG "value": [ 06:49:21 DEBUG "metric": { 06:49:21 DEBUG }, 06:49:21 DEBUG "resultType": "vector", 06:49:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:21 DEBUG { 06:49:21 INFO [http_cmd]: http status code OK 06:49:21 DEBUG }, 06:49:21 DEBUG }, 06:49:21 DEBUG "pod": "ds-idrepo-2" 06:49:21 DEBUG 06:49:21 DEBUG "pod": "ds-idrepo-2" 06:49:21 DEBUG "value": [ 06:49:21 DEBUG "result": [ 06:49:21 DEBUG --- http response --- 06:49:21 DEBUG "metric": { 06:49:21 DEBUG --- status code --- 06:49:21 DEBUG "value": [ 06:49:21 DEBUG "value": [ 06:49:21 DEBUG }, 06:49:21 DEBUG 06:49:21 DEBUG }, 06:49:21 DEBUG 06:49:21 DEBUG { 06:49:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:49:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:49:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 06:49:21 DEBUG "metric": { 06:49:21 DEBUG }, 06:49:21 DEBUG ----- output ----- 06:49:21 DEBUG --- http response --- 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG [...] 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG [...] 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG "instance": "10.142.0.107:9100" 06:49:21 DEBUG 06:49:21 DEBUG { 06:49:21 DEBUG 06:49:21 DEBUG [...] 06:49:21 DEBUG ----- output ----- 06:49:21 DEBUG [...] 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG }, 06:49:21 DEBUG 06:49:21 DEBUG "status": "success", 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG { 06:49:21 DEBUG 06:49:21 DEBUG [...] 06:49:21 DEBUG 06:49:21 DEBUG [...] 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 06:49:21 DEBUG "data": { 06:49:21 DEBUG [...] 06:49:21 DEBUG 06:49:21 DEBUG "status": "success", 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG "pod": "ds-cts-2" 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:49:21 DEBUG "resultType": "vector", 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG "data": { 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG }, 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG }, 06:49:21 DEBUG "result": [ 06:49:21 DEBUG 06:49:21 DEBUG "pod": "ds-idrepo-1" 06:49:21 DEBUG "resultType": "vector", 06:49:21 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:49:21 DEBUG 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG "value": [ 06:49:21 DEBUG { 06:49:21 DEBUG 06:49:21 DEBUG }, 06:49:21 DEBUG "result": [ 06:49:21 DEBUG }, 06:49:21 DEBUG "pod": "ds-idrepo-0" 06:49:21 DEBUG 1712814561, 06:49:21 DEBUG "pod": "ds-idrepo-0" 06:49:21 DEBUG [...] 06:49:21 DEBUG 1712814561, 06:49:21 DEBUG "metric": { 06:49:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:49:21 DEBUG "value": [ 06:49:21 DEBUG { 06:49:21 DEBUG "value": [ 06:49:21 DEBUG }, 06:49:21 DEBUG "0.000030149444444444445" 06:49:21 DEBUG }, 06:49:21 DEBUG 06:49:21 DEBUG "0" 06:49:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:49:21 DEBUG }, 06:49:21 DEBUG 1712814561, 06:49:21 DEBUG "metric": { 06:49:21 DEBUG 1712814561, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG ] 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 06:49:21 DEBUG ] 06:49:21 DEBUG }, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG "0" 06:49:21 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:49:21 DEBUG "4.133172513940997" 06:49:21 DEBUG 1712814561, 06:49:21 DEBUG } 06:49:21 DEBUG 1712814561, 06:49:21 DEBUG 06:49:21 DEBUG } 06:49:21 DEBUG "value": [ 06:49:21 DEBUG 1712814561, 06:49:21 DEBUG ] 06:49:21 DEBUG }, 06:49:21 DEBUG ] 06:49:21 DEBUG "0" 06:49:21 DEBUG ] 06:49:21 DEBUG "0" 06:49:21 DEBUG "instance": "10.142.0.77:9100" 06:49:21 DEBUG ] 06:49:21 DEBUG 06:49:21 DEBUG "1357.5333333333333" 06:49:21 DEBUG } 06:49:21 DEBUG "value": [ 06:49:21 DEBUG } 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG ] 06:49:21 DEBUG }, 06:49:21 DEBUG } 06:49:21 DEBUG 06:49:21 DEBUG ] 06:49:21 DEBUG ] 06:49:21 DEBUG 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG } 06:49:21 DEBUG } 06:49:21 DEBUG "value": [ 06:49:21 DEBUG } 06:49:21 DEBUG 06:49:21 DEBUG } 06:49:21 DEBUG } 06:49:21 DEBUG 06:49:21 DEBUG } 06:49:21 DEBUG ] 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG [...] 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG 06:49:21 DEBUG } 06:49:21 DEBUG } 06:49:21 DEBUG 1712814561, 06:49:21 DEBUG } 06:49:21 DEBUG 06:49:21 DEBUG } 06:49:21 DEBUG [...] 06:49:21 DEBUG } 06:49:21 DEBUG "0" 06:49:21 DEBUG 06:49:21 DEBUG } 06:49:21 DEBUG 06:49:21 DEBUG ] 06:49:21 DEBUG 06:49:21 DEBUG 06:49:21 DEBUG } 06:49:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:49:21 DEBUG 06:49:21 DEBUG ] 06:49:21 DEBUG }, 06:49:21 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 06:49:21 DEBUG } 06:49:21 DEBUG "value": [ 06:49:21 DEBUG }, 06:49:21 DEBUG } 06:49:21 DEBUG 1712814561, 06:49:21 DEBUG "value": [ 06:49:21 DEBUG "1357.7" 06:49:21 DEBUG 1712814561, 06:49:21 DEBUG ] 06:49:21 DEBUG "14135308288" 06:49:21 DEBUG } 06:49:21 DEBUG ] 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG } 06:49:21 DEBUG ] 06:49:21 DEBUG } 06:49:21 DEBUG } 06:49:21 DEBUG } 06:49:33 INFO 06:49:33 INFO [loop_until]: kubectl --namespace=xlou top pods 06:49:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:49:33 INFO [loop_until]: OK (rc = 0) 06:49:33 DEBUG --- stdout --- 06:49:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 8911m 8526Mi am-5b5bd8567b-8h2w4 8869m 8568Mi am-5b5bd8567b-qlkw6 7331m 8460Mi ds-cts-0 8073m 13394Mi ds-cts-1 8544m 12553Mi ds-cts-2 8461m 13395Mi ds-idrepo-0 11156m 5503Mi ds-idrepo-1 333m 5479Mi ds-idrepo-2 411m 5784Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 4m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 4m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4225m 1859Mi 06:49:33 DEBUG --- stderr --- 06:49:33 DEBUG 06:49:36 INFO 06:49:36 INFO [loop_until]: kubectl --namespace=xlou top node 06:49:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:49:36 INFO [loop_until]: OK (rc = 0) 06:49:36 DEBUG --- stdout --- 06:49:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 2719Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 62m 0% 2779Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7839m 26% 10116Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10754m 36% 2986Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9721m 32% 10120Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 146m 0% 3860Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9645m 32% 10197Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 461m 2% 6463Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11539m 72% 6197Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8475m 53% 14003Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 8273m 52% 14019Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 8552m 53% 13140Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 400m 2% 6159Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4298m 27% 5613Mi 9% 06:49:36 DEBUG --- stderr --- 06:49:36 DEBUG 06:50:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814621" 06:50:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814621" 06:50:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712814621" 06:50:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814621" 06:50:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814621" 06:50:21 INFO [http_cmd]: http status code OK 06:50:21 DEBUG --- status code --- 06:50:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:21 DEBUG --- http response --- 06:50:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:50:21 DEBUG ----- output ----- 06:50:21 DEBUG { 06:50:21 DEBUG "status": "success", 06:50:21 DEBUG "data": { 06:50:21 DEBUG "resultType": "vector", 06:50:21 DEBUG "result": [ 06:50:21 DEBUG { 06:50:21 DEBUG "metric": { 06:50:21 DEBUG "pod": "ds-idrepo-2" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG [...] 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG "pod": "ds-idrepo-0" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 1712814621, 06:50:21 DEBUG "0" 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG } 06:50:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814621" 06:50:21 INFO [http_cmd]: http status code OK 06:50:21 DEBUG --- status code --- 06:50:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:21 DEBUG --- http response --- 06:50:21 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:50:21 INFO [http_cmd]: http status code OK 06:50:21 DEBUG --- status code --- 06:50:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:21 DEBUG --- http response --- 06:50:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:50:21 DEBUG ----- output ----- 06:50:21 DEBUG { 06:50:21 DEBUG "status": "success", 06:50:21 DEBUG "data": { 06:50:21 DEBUG "resultType": "vector", 06:50:21 DEBUG "result": [ 06:50:21 DEBUG { 06:50:21 DEBUG "metric": { 06:50:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG [...] 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 1712814621, 06:50:21 DEBUG "0" 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG } 06:50:21 ERROR [http_cmd]: ERROR 06:50:21 ERROR --- status code --- 06:50:21 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:50:21 ERROR --- http response --- 06:50:21 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:50:21 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:50:21 INFO [http_cmd]: http status code OK 06:50:21 DEBUG --- status code --- 06:50:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:21 DEBUG --- http response --- 06:50:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:50:21 DEBUG ----- output ----- 06:50:21 DEBUG { 06:50:21 DEBUG "status": "success", 06:50:21 DEBUG "data": { 06:50:21 DEBUG "resultType": "vector", 06:50:21 DEBUG "result": [ 06:50:21 DEBUG { 06:50:21 DEBUG "metric": { 06:50:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG [...] 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 1712814621, 06:50:21 DEBUG "0" 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG } 06:50:21 INFO [http_cmd]: http status code OK 06:50:21 DEBUG --- status code --- 06:50:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:21 DEBUG --- http response --- 06:50:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:50:21 DEBUG ----- output ----- 06:50:21 DEBUG { 06:50:21 DEBUG "status": "success", 06:50:21 DEBUG "data": { 06:50:21 DEBUG "resultType": "vector", 06:50:21 DEBUG "result": [ 06:50:21 DEBUG { 06:50:21 DEBUG "metric": { 06:50:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG [...] 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 1712814621, 06:50:21 DEBUG "244982.5" 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG } 06:50:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814621" 06:50:21 INFO [http_cmd]: http status code OK 06:50:21 DEBUG --- status code --- 06:50:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:21 DEBUG --- http response --- 06:50:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:50:21 DEBUG ----- output ----- 06:50:21 DEBUG { 06:50:21 DEBUG "status": "success", 06:50:21 DEBUG "data": { 06:50:21 DEBUG "resultType": "vector", 06:50:21 DEBUG "result": [ 06:50:21 DEBUG { 06:50:21 DEBUG "metric": { 06:50:21 DEBUG "instance": "10.142.0.107:9100" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG [...] 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG "instance": "10.142.0.77:9100" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 1712814621, 06:50:21 DEBUG "82735805.03333333" 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG } 06:50:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814621" 06:50:21 INFO [http_cmd]: http status code OK 06:50:21 DEBUG --- status code --- 06:50:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:21 DEBUG --- http response --- 06:50:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:50:21 DEBUG ----- output ----- 06:50:21 DEBUG { 06:50:21 DEBUG "status": "success", 06:50:21 DEBUG "data": { 06:50:21 DEBUG "resultType": "vector", 06:50:21 DEBUG "result": [ 06:50:21 DEBUG { 06:50:21 DEBUG "metric": { 06:50:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG [...] 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 1712814621, 06:50:21 DEBUG "1368.3666666666666" 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG } 06:50:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814621" 06:50:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712814621" 06:50:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712814621" 06:50:21 INFO [http_cmd]: http status code OK 06:50:21 DEBUG --- status code --- 06:50:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:21 DEBUG --- http response --- 06:50:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:50:21 DEBUG ----- output ----- 06:50:21 DEBUG { 06:50:21 DEBUG "status": "success", 06:50:21 DEBUG "data": { 06:50:21 DEBUG "resultType": "vector", 06:50:21 DEBUG "result": [ 06:50:21 DEBUG { 06:50:21 DEBUG "metric": { 06:50:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG [...] 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 1712814621, 06:50:21 DEBUG "0.021" 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG } 06:50:21 INFO [http_cmd]: http status code OK 06:50:21 DEBUG --- status code --- 06:50:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:21 DEBUG --- http response --- 06:50:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:50:21 DEBUG ----- output ----- 06:50:21 DEBUG { 06:50:21 DEBUG "status": "success", 06:50:21 DEBUG "data": { 06:50:21 DEBUG "resultType": "vector", 06:50:21 DEBUG "result": [ 06:50:21 DEBUG { 06:50:21 DEBUG "metric": { 06:50:21 DEBUG "instance": "10.142.0.107:9100" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG [...] 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG "instance": "10.142.0.77:9100" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 1712814621, 06:50:21 DEBUG "52961838080" 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG } 06:50:21 INFO [http_cmd]: http status code OK 06:50:21 DEBUG --- status code --- 06:50:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:21 DEBUG --- http response --- 06:50:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:50:21 DEBUG ----- output ----- 06:50:21 DEBUG { 06:50:21 DEBUG "status": "success", 06:50:21 DEBUG "data": { 06:50:21 DEBUG "resultType": "vector", 06:50:21 DEBUG "result": [ 06:50:21 DEBUG { 06:50:21 DEBUG "metric": { 06:50:21 DEBUG "pod": "ds-idrepo-2" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG [...] 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG "pod": "ds-cts-2" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 1712814621, 06:50:21 DEBUG "0" 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG } 06:50:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814621" 06:50:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814621" 06:50:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814621" 06:50:21 INFO [http_cmd]: http status code OK 06:50:21 DEBUG --- status code --- 06:50:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:21 DEBUG --- http response --- 06:50:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:50:21 DEBUG ----- output ----- 06:50:21 DEBUG { 06:50:21 DEBUG "status": "success", 06:50:21 DEBUG "data": { 06:50:21 DEBUG "resultType": "vector", 06:50:21 DEBUG "result": [ 06:50:21 DEBUG { 06:50:21 DEBUG "metric": { 06:50:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG [...] 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 1712814621, 06:50:21 DEBUG "1357.5333333333333" 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG } 06:50:21 INFO [http_cmd]: http status code OK 06:50:21 DEBUG --- status code --- 06:50:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:21 DEBUG --- http response --- 06:50:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:50:21 DEBUG ----- output ----- 06:50:21 DEBUG { 06:50:21 DEBUG "status": "success", 06:50:21 DEBUG "data": { 06:50:21 DEBUG "resultType": "vector", 06:50:21 DEBUG "result": [ 06:50:21 DEBUG { 06:50:21 INFO [http_cmd]: http status code OK 06:50:21 DEBUG "metric": { 06:50:21 DEBUG --- status code --- 06:50:21 DEBUG "instance": "10.142.0.107:9100" 06:50:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:21 DEBUG }, 06:50:21 DEBUG --- http response --- 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG [...] 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG "instance": "10.142.0.77:9100" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 1712814621, 06:50:21 DEBUG "0.0027333333333331208" 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG } 06:50:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:50:21 DEBUG ----- output ----- 06:50:21 DEBUG { 06:50:21 DEBUG "status": "success", 06:50:21 DEBUG "data": { 06:50:21 DEBUG "resultType": "vector", 06:50:21 DEBUG "result": [ 06:50:21 DEBUG { 06:50:21 DEBUG "metric": { 06:50:21 DEBUG "instance": "10.142.0.107:9100" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG [...] 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG "instance": "10.142.0.77:9100" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 1712814621, 06:50:21 DEBUG "82738286.7" 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG } 06:50:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814621" 06:50:21 INFO [http_cmd]: http status code OK 06:50:21 DEBUG --- status code --- 06:50:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:21 DEBUG --- http response --- 06:50:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 06:50:21 DEBUG ----- output ----- 06:50:21 DEBUG { 06:50:21 DEBUG "status": "success", 06:50:21 DEBUG "data": { 06:50:21 DEBUG "resultType": "vector", 06:50:21 DEBUG "result": [ 06:50:21 DEBUG { 06:50:21 DEBUG "metric": { 06:50:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG [...] 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG "pod": "ds-idrepo-1" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 1712814621, 06:50:21 DEBUG "1595.22181346029" 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG } 06:50:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814621" 06:50:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814621" 06:50:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814621" 06:50:21 INFO [http_cmd]: http status code OK 06:50:21 DEBUG --- status code --- 06:50:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:21 DEBUG --- http response --- 06:50:21 INFO [http_cmd]: http status code OK 06:50:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:50:21 DEBUG --- status code --- 06:50:21 DEBUG ----- output ----- 06:50:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:21 DEBUG { 06:50:21 DEBUG --- http response --- 06:50:21 DEBUG "status": "success", 06:50:21 DEBUG "data": { 06:50:21 DEBUG "resultType": "vector", 06:50:21 DEBUG "result": [ 06:50:21 DEBUG { 06:50:21 DEBUG "metric": { 06:50:21 DEBUG "pod": "ds-idrepo-2" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG [...] 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG "pod": "ds-idrepo-0" 06:50:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:50:21 DEBUG }, 06:50:21 DEBUG ----- output ----- 06:50:21 DEBUG "value": [ 06:50:21 DEBUG { 06:50:21 DEBUG 1712814621, 06:50:21 DEBUG "status": "success", 06:50:21 DEBUG "0" 06:50:21 DEBUG "data": { 06:50:21 DEBUG ] 06:50:21 DEBUG "resultType": "vector", 06:50:21 DEBUG } 06:50:21 DEBUG "result": [ 06:50:21 DEBUG ] 06:50:21 DEBUG { 06:50:21 DEBUG } 06:50:21 DEBUG "metric": { 06:50:21 DEBUG } 06:50:21 DEBUG "pod": "ds-idrepo-2" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG [...] 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG "pod": "ds-cts-2" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 1712814621, 06:50:21 DEBUG "29.666666666666668" 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG } 06:50:21 INFO [http_cmd]: http status code OK 06:50:21 DEBUG --- status code --- 06:50:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:21 DEBUG --- http response --- 06:50:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:50:21 DEBUG ----- output ----- 06:50:21 DEBUG { 06:50:21 DEBUG "status": "success", 06:50:21 DEBUG "data": { 06:50:21 DEBUG "resultType": "vector", 06:50:21 DEBUG "result": [ 06:50:21 DEBUG { 06:50:21 DEBUG "metric": { 06:50:21 DEBUG "pod": "ds-idrepo-2" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG [...] 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG "pod": "ds-cts-2" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 1712814621, 06:50:21 DEBUG "0" 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG } 06:50:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814621" 06:50:21 INFO [http_cmd]: http status code OK 06:50:21 DEBUG --- status code --- 06:50:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:21 DEBUG --- http response --- 06:50:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:50:21 DEBUG ----- output ----- 06:50:21 DEBUG { 06:50:21 DEBUG "status": "success", 06:50:21 DEBUG "data": { 06:50:21 DEBUG "resultType": "vector", 06:50:21 DEBUG "result": [ 06:50:21 DEBUG { 06:50:21 DEBUG "metric": { 06:50:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG [...] 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 1712814621, 06:50:21 DEBUG "1357.6" 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG } 06:50:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814621" 06:50:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814621" 06:50:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712814621" 06:50:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814621" 06:50:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814621" 06:50:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814621" 06:50:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814621" 06:50:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712814621" 06:50:21 INFO [http_cmd]: http status code OK 06:50:21 DEBUG --- status code --- 06:50:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:21 DEBUG --- http response --- 06:50:21 INFO [http_cmd]: http status code OK 06:50:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:50:21 DEBUG --- status code --- 06:50:21 DEBUG ----- output ----- 06:50:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:21 DEBUG { 06:50:21 DEBUG --- http response --- 06:50:21 DEBUG "status": "success", 06:50:21 DEBUG "data": { 06:50:21 DEBUG "resultType": "vector", 06:50:21 DEBUG "result": [ 06:50:21 DEBUG { 06:50:21 DEBUG "metric": { 06:50:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:50:21 DEBUG [...] 06:50:21 INFO [http_cmd]: http status code OK 06:50:21 INFO [http_cmd]: http status code OK 06:50:21 INFO [http_cmd]: http status code OK 06:50:21 DEBUG ----- output ----- 06:50:21 DEBUG 06:50:21 DEBUG --- status code --- 06:50:21 DEBUG --- status code --- 06:50:21 DEBUG --- status code --- 06:50:21 DEBUG { 06:50:21 DEBUG 06:50:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:21 DEBUG "status": "success", 06:50:21 DEBUG 06:50:21 DEBUG --- http response --- 06:50:21 INFO [http_cmd]: http status code OK 06:50:21 DEBUG --- http response --- 06:50:21 DEBUG --- http response --- 06:50:21 DEBUG "data": { 06:50:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:50:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:50:21 DEBUG --- status code --- 06:50:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:50:21 DEBUG "resultType": "vector", 06:50:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 06:50:21 DEBUG }, 06:50:21 DEBUG ----- output ----- 06:50:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:21 DEBUG ----- output ----- 06:50:21 DEBUG "result": [ 06:50:21 DEBUG ----- output ----- 06:50:21 DEBUG "value": [ 06:50:21 DEBUG { 06:50:21 INFO [http_cmd]: http status code OK 06:50:21 DEBUG --- http response --- 06:50:21 DEBUG { 06:50:21 DEBUG { 06:50:21 DEBUG "metric": { 06:50:21 DEBUG 1712814621, 06:50:21 DEBUG "0" 06:50:21 DEBUG --- status code --- 06:50:21 DEBUG "status": "success", 06:50:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:50:21 INFO [http_cmd]: http status code OK 06:50:21 DEBUG { 06:50:21 DEBUG "pod": "ds-idrepo-2" 06:50:21 DEBUG "status": "success", 06:50:21 DEBUG ] 06:50:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:21 DEBUG "data": { 06:50:21 DEBUG ----- output ----- 06:50:21 DEBUG --- status code --- 06:50:21 DEBUG "status": "success", 06:50:21 DEBUG }, 06:50:21 DEBUG "data": { 06:50:21 DEBUG } 06:50:21 DEBUG --- http response --- 06:50:21 DEBUG "resultType": "vector", 06:50:21 DEBUG { 06:50:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:50:21 DEBUG "data": { 06:50:21 DEBUG "value": [ 06:50:21 DEBUG "resultType": "vector", 06:50:21 DEBUG ] 06:50:21 DEBUG "result": [ 06:50:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:50:21 DEBUG "status": "success", 06:50:21 DEBUG --- http response --- 06:50:21 DEBUG "resultType": "vector", 06:50:21 DEBUG 06:50:21 DEBUG "result": [ 06:50:21 DEBUG } 06:50:21 DEBUG { 06:50:21 DEBUG ----- output ----- 06:50:21 DEBUG "data": { 06:50:21 DEBUG "result": [ 06:50:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:50:21 DEBUG 06:50:21 DEBUG { 06:50:21 DEBUG } 06:50:21 DEBUG "metric": { 06:50:21 DEBUG { 06:50:21 DEBUG "resultType": "vector", 06:50:21 DEBUG { 06:50:21 DEBUG ----- output ----- 06:50:21 DEBUG 06:50:21 DEBUG "metric": { 06:50:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:50:21 DEBUG "status": "success", 06:50:21 DEBUG "result": [ 06:50:21 DEBUG "metric": { 06:50:21 DEBUG { 06:50:21 DEBUG [...] 06:50:21 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:50:21 DEBUG }, 06:50:21 DEBUG "data": { 06:50:21 DEBUG { 06:50:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:50:21 DEBUG "status": "success", 06:50:21 DEBUG 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG "resultType": "vector", 06:50:21 DEBUG "metric": { 06:50:21 DEBUG }, 06:50:21 DEBUG "data": { 06:50:21 DEBUG 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 06:50:21 DEBUG "result": [ 06:50:21 DEBUG "pod": "ds-idrepo-2" 06:50:21 DEBUG "value": [ 06:50:21 DEBUG "resultType": "vector", 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG { 06:50:21 DEBUG }, 06:50:21 DEBUG 06:50:21 DEBUG "result": [ 06:50:21 DEBUG "pod": "ds-cts-2" 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG "metric": { 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 06:50:21 DEBUG { 06:50:21 DEBUG }, 06:50:21 DEBUG 06:50:21 DEBUG [...] 06:50:21 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG "metric": { 06:50:21 DEBUG "value": [ 06:50:21 DEBUG [...] 06:50:21 DEBUG 06:50:21 DEBUG }, 06:50:21 DEBUG 06:50:21 DEBUG [...] 06:50:21 DEBUG "instance": "10.142.0.107:9100" 06:50:21 DEBUG 1712814621, 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG }, 06:50:21 DEBUG "0" 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG [...] 06:50:21 DEBUG 06:50:21 DEBUG "value": [ 06:50:21 DEBUG ] 06:50:21 DEBUG 06:50:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG } 06:50:21 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:50:21 DEBUG }, 06:50:21 DEBUG 06:50:21 DEBUG 06:50:21 DEBUG "pod": "ds-idrepo-1" 06:50:21 DEBUG 06:50:21 DEBUG ] 06:50:21 DEBUG }, 06:50:21 DEBUG "value": [ 06:50:21 DEBUG [...] 06:50:21 DEBUG 06:50:21 DEBUG }, 06:50:21 DEBUG 06:50:21 DEBUG } 06:50:21 DEBUG "value": [ 06:50:21 DEBUG 1712814621, 06:50:21 DEBUG 06:50:21 DEBUG "pod": "ds-idrepo-0" 06:50:21 DEBUG "value": [ 06:50:21 DEBUG [...] 06:50:21 DEBUG } 06:50:21 DEBUG 1712814621, 06:50:21 DEBUG "1363.1333333333334" 06:50:21 DEBUG 06:50:21 DEBUG }, 06:50:21 DEBUG 1712814621, 06:50:21 DEBUG 06:50:21 DEBUG "4.13694975696433" 06:50:21 DEBUG ] 06:50:21 DEBUG 06:50:21 DEBUG "value": [ 06:50:21 DEBUG "0" 06:50:21 DEBUG 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 06:50:21 DEBUG 1712814621, 06:50:21 DEBUG ] 06:50:21 DEBUG 06:50:21 DEBUG } 06:50:21 DEBUG ] 06:50:21 DEBUG }, 06:50:21 DEBUG "0" 06:50:21 DEBUG } 06:50:21 DEBUG "instance": "10.142.0.77:9100" 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG "value": [ 06:50:21 DEBUG ] 06:50:21 DEBUG ] 06:50:21 DEBUG }, 06:50:21 DEBUG } 06:50:21 DEBUG } 06:50:21 DEBUG 1712814621, 06:50:21 DEBUG } 06:50:21 DEBUG } 06:50:21 DEBUG "value": [ 06:50:21 DEBUG } 06:50:21 DEBUG "13365460992" 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG 1712814621, 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG "0.000999999999999801" 06:50:21 DEBUG } 06:50:21 DEBUG } 06:50:21 DEBUG ] 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG } 06:50:21 DEBUG ] 06:50:21 DEBUG } 06:50:21 DEBUG } 06:50:21 DEBUG } 06:50:33 INFO 06:50:33 INFO [loop_until]: kubectl --namespace=xlou top pods 06:50:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:50:33 INFO [loop_until]: OK (rc = 0) 06:50:33 DEBUG --- stdout --- 06:50:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 8926m 8514Mi am-5b5bd8567b-8h2w4 8861m 8565Mi am-5b5bd8567b-qlkw6 7542m 8460Mi ds-cts-0 8061m 12827Mi ds-cts-1 8670m 13065Mi ds-cts-2 8403m 13023Mi ds-idrepo-0 11306m 5503Mi ds-idrepo-1 363m 5479Mi ds-idrepo-2 369m 5784Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 3m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 6m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4111m 1860Mi 06:50:33 DEBUG --- stderr --- 06:50:33 DEBUG 06:50:36 INFO 06:50:36 INFO [loop_until]: kubectl --namespace=xlou top node 06:50:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:50:37 INFO [loop_until]: OK (rc = 0) 06:50:37 DEBUG --- stdout --- 06:50:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 2734Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 60m 0% 2777Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7874m 26% 10123Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10739m 35% 2981Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9539m 31% 10100Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 150m 0% 3855Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9666m 32% 10188Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 434m 2% 6469Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11536m 72% 6198Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8716m 54% 13656Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 7991m 50% 13512Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 8790m 55% 13911Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 417m 2% 6163Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4105m 25% 5617Mi 9% 06:50:37 DEBUG --- stderr --- 06:50:37 DEBUG 06:51:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814681" 06:51:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712814681" 06:51:21 INFO [http_cmd]: http status code OK 06:51:21 DEBUG --- status code --- 06:51:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:21 DEBUG --- http response --- 06:51:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:51:21 DEBUG ----- output ----- 06:51:21 DEBUG { 06:51:21 DEBUG "status": "success", 06:51:21 DEBUG "data": { 06:51:21 DEBUG "resultType": "vector", 06:51:21 DEBUG "result": [ 06:51:21 DEBUG { 06:51:21 DEBUG "metric": { 06:51:21 DEBUG "pod": "ds-idrepo-2" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG [...] 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG "pod": "ds-idrepo-0" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 1712814681, 06:51:21 DEBUG "0" 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG } 06:51:21 ERROR [http_cmd]: ERROR 06:51:21 ERROR --- status code --- 06:51:21 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:51:21 ERROR --- http response --- 06:51:21 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:51:21 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:51:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814681" 06:51:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814681" 06:51:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814681" 06:51:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712814681" 06:51:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814681" 06:51:21 INFO [http_cmd]: http status code OK 06:51:21 INFO [http_cmd]: http status code OK 06:51:21 DEBUG --- status code --- 06:51:21 DEBUG --- status code --- 06:51:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:21 DEBUG --- http response --- 06:51:21 DEBUG --- http response --- 06:51:21 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:51:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:51:21 DEBUG ----- output ----- 06:51:21 DEBUG { 06:51:21 DEBUG "status": "success", 06:51:21 DEBUG "data": { 06:51:21 DEBUG "resultType": "vector", 06:51:21 DEBUG "result": [ 06:51:21 DEBUG { 06:51:21 DEBUG "metric": { 06:51:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG [...] 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG "pod": "ds-cts-0" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 1712814681, 06:51:21 DEBUG "4.385741434946065" 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG } 06:51:21 INFO [http_cmd]: http status code OK 06:51:21 DEBUG --- status code --- 06:51:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:21 DEBUG --- http response --- 06:51:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:51:21 DEBUG ----- output ----- 06:51:21 DEBUG { 06:51:21 DEBUG "status": "success", 06:51:21 DEBUG "data": { 06:51:21 DEBUG "resultType": "vector", 06:51:21 DEBUG "result": [ 06:51:21 DEBUG { 06:51:21 DEBUG "metric": { 06:51:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG [...] 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG "pod": "ds-cts-0" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 1712814681, 06:51:21 DEBUG "6271800.305227929" 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG } 06:51:21 INFO [http_cmd]: http status code OK 06:51:21 INFO [http_cmd]: http status code OK 06:51:21 DEBUG --- status code --- 06:51:21 DEBUG --- status code --- 06:51:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:21 DEBUG --- http response --- 06:51:21 DEBUG --- http response --- 06:51:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:51:21 DEBUG ----- output ----- 06:51:21 DEBUG { 06:51:21 DEBUG "status": "success", 06:51:21 DEBUG "data": { 06:51:21 DEBUG "resultType": "vector", 06:51:21 DEBUG "result": [ 06:51:21 DEBUG { 06:51:21 DEBUG "metric": { 06:51:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 06:51:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:51:21 DEBUG 06:51:21 DEBUG ----- output ----- 06:51:21 DEBUG 06:51:21 DEBUG { 06:51:21 DEBUG [...] 06:51:21 DEBUG "status": "success", 06:51:21 DEBUG 06:51:21 DEBUG "data": { 06:51:21 DEBUG 06:51:21 DEBUG "resultType": "vector", 06:51:21 DEBUG 06:51:21 DEBUG "result": [ 06:51:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:51:21 DEBUG { 06:51:21 DEBUG }, 06:51:21 DEBUG "metric": { 06:51:21 DEBUG "value": [ 06:51:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:51:21 DEBUG 1712814681, 06:51:21 DEBUG }, 06:51:21 DEBUG "244752" 06:51:21 DEBUG "value": [ 06:51:21 DEBUG ] 06:51:21 DEBUG 06:51:21 DEBUG } 06:51:21 DEBUG 06:51:21 DEBUG ] 06:51:21 DEBUG 06:51:21 DEBUG } 06:51:21 DEBUG [...] 06:51:21 DEBUG } 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 1712814681, 06:51:21 DEBUG "0.021" 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG } 06:51:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814681" 06:51:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712814681" 06:51:21 INFO [http_cmd]: http status code OK 06:51:21 DEBUG --- status code --- 06:51:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:21 DEBUG --- http response --- 06:51:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:51:21 DEBUG ----- output ----- 06:51:21 DEBUG { 06:51:21 DEBUG "status": "success", 06:51:21 DEBUG "data": { 06:51:21 DEBUG "resultType": "vector", 06:51:21 DEBUG "result": [ 06:51:21 DEBUG { 06:51:21 DEBUG "metric": { 06:51:21 DEBUG "instance": "10.142.0.107:9100" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG [...] 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG "instance": "10.142.0.77:9100" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 1712814681, 06:51:21 DEBUG "82994747.73333333" 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG } 06:51:21 INFO [http_cmd]: http status code OK 06:51:21 DEBUG --- status code --- 06:51:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:21 DEBUG --- http response --- 06:51:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:51:21 DEBUG ----- output ----- 06:51:21 DEBUG { 06:51:21 DEBUG "status": "success", 06:51:21 DEBUG "data": { 06:51:21 DEBUG "resultType": "vector", 06:51:21 DEBUG "result": [ 06:51:21 DEBUG { 06:51:21 DEBUG "metric": { 06:51:21 DEBUG "pod": "ds-idrepo-2" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG [...] 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG "pod": "ds-cts-2" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 1712814681, 06:51:21 DEBUG "0" 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG } 06:51:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814681" 06:51:21 INFO [http_cmd]: http status code OK 06:51:21 DEBUG --- status code --- 06:51:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:21 DEBUG --- http response --- 06:51:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:51:21 DEBUG ----- output ----- 06:51:21 DEBUG { 06:51:21 DEBUG "status": "success", 06:51:21 DEBUG "data": { 06:51:21 DEBUG "resultType": "vector", 06:51:21 DEBUG "result": [ 06:51:21 DEBUG { 06:51:21 DEBUG "metric": { 06:51:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG [...] 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 1712814681, 06:51:21 DEBUG "1367.7333333333333" 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG } 06:51:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814681" 06:51:21 INFO [http_cmd]: http status code OK 06:51:21 DEBUG --- status code --- 06:51:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:21 DEBUG --- http response --- 06:51:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:51:21 DEBUG ----- output ----- 06:51:21 DEBUG { 06:51:21 DEBUG "status": "success", 06:51:21 DEBUG "data": { 06:51:21 DEBUG "resultType": "vector", 06:51:21 DEBUG "result": [ 06:51:21 DEBUG { 06:51:21 DEBUG "metric": { 06:51:21 DEBUG "instance": "10.142.0.107:9100" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG [...] 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG "instance": "10.142.0.77:9100" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 1712814681, 06:51:21 DEBUG "52963374080" 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG } 06:51:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814681" 06:51:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814681" 06:51:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814681" 06:51:21 INFO [http_cmd]: http status code OK 06:51:21 DEBUG --- status code --- 06:51:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:21 DEBUG --- http response --- 06:51:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:51:21 DEBUG ----- output ----- 06:51:21 DEBUG { 06:51:21 DEBUG "status": "success", 06:51:21 DEBUG "data": { 06:51:21 DEBUG "resultType": "vector", 06:51:21 DEBUG "result": [ 06:51:21 DEBUG { 06:51:21 DEBUG "metric": { 06:51:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG [...] 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 1712814681, 06:51:21 DEBUG "1347.9" 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG } 06:51:21 INFO [http_cmd]: http status code OK 06:51:21 INFO [http_cmd]: http status code OK 06:51:21 DEBUG --- status code --- 06:51:21 DEBUG --- status code --- 06:51:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:21 DEBUG --- http response --- 06:51:21 DEBUG --- http response --- 06:51:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:51:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:51:21 DEBUG ----- output ----- 06:51:21 DEBUG ----- output ----- 06:51:21 DEBUG { 06:51:21 DEBUG { 06:51:21 DEBUG "status": "success", 06:51:21 DEBUG "status": "success", 06:51:21 DEBUG "data": { 06:51:21 DEBUG "data": { 06:51:21 DEBUG "resultType": "vector", 06:51:21 DEBUG "resultType": "vector", 06:51:21 DEBUG "result": [ 06:51:21 DEBUG "result": [ 06:51:21 DEBUG { 06:51:21 DEBUG { 06:51:21 DEBUG "metric": { 06:51:21 DEBUG "metric": { 06:51:21 DEBUG "instance": "10.142.0.107:9100" 06:51:21 DEBUG "instance": "10.142.0.107:9100" 06:51:21 DEBUG }, 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG [...] 06:51:21 DEBUG [...] 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG "instance": "10.142.0.77:9100" 06:51:21 DEBUG "instance": "10.142.0.77:9100" 06:51:21 DEBUG }, 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 1712814681, 06:51:21 DEBUG 1712814681, 06:51:21 DEBUG "82996335.86666667" 06:51:21 DEBUG "0.0025000000000000946" 06:51:21 DEBUG ] 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG } 06:51:21 DEBUG ] 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG } 06:51:21 DEBUG } 06:51:21 DEBUG } 06:51:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814681" 06:51:21 INFO [http_cmd]: http status code OK 06:51:21 DEBUG --- status code --- 06:51:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:21 DEBUG --- http response --- 06:51:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:51:21 DEBUG ----- output ----- 06:51:21 DEBUG { 06:51:21 DEBUG "status": "success", 06:51:21 DEBUG "data": { 06:51:21 DEBUG "resultType": "vector", 06:51:21 DEBUG "result": [ 06:51:21 DEBUG { 06:51:21 DEBUG "metric": { 06:51:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG [...] 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG "pod": "ds-cts-0" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 1712814681, 06:51:21 DEBUG "13416442.56387767" 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG } 06:51:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814681" 06:51:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814681" 06:51:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814681" 06:51:21 INFO [http_cmd]: http status code OK 06:51:21 DEBUG --- status code --- 06:51:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:21 DEBUG --- http response --- 06:51:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:51:21 DEBUG ----- output ----- 06:51:21 DEBUG { 06:51:21 DEBUG "status": "success", 06:51:21 DEBUG "data": { 06:51:21 DEBUG "resultType": "vector", 06:51:21 DEBUG "result": [ 06:51:21 DEBUG { 06:51:21 DEBUG "metric": { 06:51:21 DEBUG "pod": "ds-idrepo-2" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG [...] 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG "pod": "ds-idrepo-0" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 1712814681, 06:51:21 DEBUG "0" 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG } 06:51:21 INFO [http_cmd]: http status code OK 06:51:21 DEBUG --- status code --- 06:51:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:21 DEBUG --- http response --- 06:51:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:51:21 DEBUG ----- output ----- 06:51:21 DEBUG { 06:51:21 DEBUG "status": "success", 06:51:21 DEBUG "data": { 06:51:21 DEBUG "resultType": "vector", 06:51:21 DEBUG "result": [ 06:51:21 DEBUG { 06:51:21 DEBUG "metric": { 06:51:21 DEBUG "pod": "ds-idrepo-2" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG [...] 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG "pod": "ds-cts-2" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 1712814681, 06:51:21 DEBUG "29.566666666666666" 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG } 06:51:21 INFO [http_cmd]: http status code OK 06:51:21 DEBUG --- status code --- 06:51:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:21 DEBUG --- http response --- 06:51:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:51:21 DEBUG ----- output ----- 06:51:21 DEBUG { 06:51:21 DEBUG "status": "success", 06:51:21 DEBUG "data": { 06:51:21 DEBUG "resultType": "vector", 06:51:21 DEBUG "result": [ 06:51:21 DEBUG { 06:51:21 DEBUG "metric": { 06:51:21 DEBUG "pod": "ds-idrepo-2" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG [...] 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG "pod": "ds-cts-2" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 1712814681, 06:51:21 DEBUG "0" 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG } 06:51:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814681" 06:51:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814681" 06:51:21 INFO [http_cmd]: http status code OK 06:51:21 DEBUG --- status code --- 06:51:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:21 DEBUG --- http response --- 06:51:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:51:21 DEBUG ----- output ----- 06:51:21 DEBUG { 06:51:21 DEBUG "status": "success", 06:51:21 DEBUG "data": { 06:51:21 DEBUG "resultType": "vector", 06:51:21 DEBUG "result": [ 06:51:21 DEBUG { 06:51:21 DEBUG "metric": { 06:51:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG [...] 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 1712814681, 06:51:21 DEBUG "1348.4" 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG } 06:51:21 INFO [http_cmd]: http status code OK 06:51:21 DEBUG --- status code --- 06:51:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:21 DEBUG --- http response --- 06:51:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:51:21 DEBUG ----- output ----- 06:51:21 DEBUG { 06:51:21 DEBUG "status": "success", 06:51:21 DEBUG "data": { 06:51:21 DEBUG "resultType": "vector", 06:51:21 DEBUG "result": [ 06:51:21 DEBUG { 06:51:21 DEBUG "metric": { 06:51:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG [...] 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG "pod": "ds-cts-0" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 1712814681, 06:51:21 DEBUG "0" 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG } 06:51:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814681" 06:51:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814681" 06:51:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814681" 06:51:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712814681" 06:51:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814681" 06:51:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814681" 06:51:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712814681" 06:51:21 INFO [http_cmd]: http status code OK 06:51:21 DEBUG --- status code --- 06:51:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:21 DEBUG --- http response --- 06:51:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:51:21 DEBUG ----- output ----- 06:51:21 DEBUG { 06:51:21 DEBUG "status": "success", 06:51:21 DEBUG "data": { 06:51:21 DEBUG "resultType": "vector", 06:51:21 DEBUG "result": [ 06:51:21 DEBUG { 06:51:21 DEBUG "metric": { 06:51:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG [...] 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 1712814681, 06:51:21 DEBUG "0" 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG } 06:51:21 INFO [http_cmd]: http status code OK 06:51:21 INFO [http_cmd]: http status code OK 06:51:21 DEBUG --- status code --- 06:51:21 DEBUG --- status code --- 06:51:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:21 DEBUG --- http response --- 06:51:21 DEBUG --- http response --- 06:51:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:51:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:51:21 DEBUG ----- output ----- 06:51:21 DEBUG ----- output ----- 06:51:21 DEBUG { 06:51:21 DEBUG { 06:51:21 DEBUG "status": "success", 06:51:21 DEBUG "status": "success", 06:51:21 DEBUG "data": { 06:51:21 DEBUG "data": { 06:51:21 DEBUG "resultType": "vector", 06:51:21 DEBUG "resultType": "vector", 06:51:21 DEBUG "result": [ 06:51:21 DEBUG "result": [ 06:51:21 DEBUG { 06:51:21 DEBUG { 06:51:21 DEBUG "metric": { 06:51:21 INFO [http_cmd]: http status code OK 06:51:21 DEBUG "metric": { 06:51:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:51:21 DEBUG --- status code --- 06:51:21 DEBUG "pod": "ds-idrepo-2" 06:51:21 DEBUG }, 06:51:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG --- http response --- 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 INFO [http_cmd]: http status code OK 06:51:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG --- status code --- 06:51:21 DEBUG ----- output ----- 06:51:21 DEBUG 06:51:21 DEBUG [...] 06:51:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:21 DEBUG { 06:51:21 DEBUG [...] 06:51:21 DEBUG 06:51:21 DEBUG --- http response --- 06:51:21 DEBUG "status": "success", 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG "data": { 06:51:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 INFO [http_cmd]: http status code OK 06:51:21 DEBUG "resultType": "vector", 06:51:21 DEBUG ----- output ----- 06:51:21 DEBUG 06:51:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:51:21 DEBUG --- status code --- 06:51:21 DEBUG "result": [ 06:51:21 DEBUG { 06:51:21 DEBUG "pod": "ds-cts-2" 06:51:21 DEBUG }, 06:51:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:21 DEBUG { 06:51:21 DEBUG "status": "success", 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG --- http response --- 06:51:21 DEBUG "metric": { 06:51:21 DEBUG "data": { 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 1712814681, 06:51:21 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:51:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:51:21 DEBUG "resultType": "vector", 06:51:21 DEBUG 1712814681, 06:51:21 DEBUG "1357.4666666666667" 06:51:21 DEBUG }, 06:51:21 DEBUG ----- output ----- 06:51:21 DEBUG "result": [ 06:51:21 DEBUG "0.0005728394444444445" 06:51:21 DEBUG ] 06:51:21 DEBUG "value": [ 06:51:21 DEBUG { 06:51:21 DEBUG { 06:51:21 INFO [http_cmd]: http status code OK 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG 06:51:21 DEBUG "status": "success", 06:51:21 DEBUG "metric": { 06:51:21 DEBUG --- status code --- 06:51:21 DEBUG } 06:51:21 DEBUG ] 06:51:21 DEBUG 06:51:21 DEBUG "data": { 06:51:21 DEBUG "pod": "ds-idrepo-2" 06:51:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG 06:51:21 DEBUG "resultType": "vector", 06:51:21 DEBUG }, 06:51:21 DEBUG --- http response --- 06:51:21 DEBUG } 06:51:21 DEBUG } 06:51:21 DEBUG [...] 06:51:21 DEBUG "result": [ 06:51:21 DEBUG "value": [ 06:51:21 DEBUG } 06:51:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:51:21 DEBUG 06:51:21 DEBUG { 06:51:21 DEBUG 06:51:21 DEBUG ----- output ----- 06:51:21 DEBUG 06:51:21 DEBUG "metric": { 06:51:21 DEBUG 06:51:21 DEBUG { 06:51:21 DEBUG 06:51:21 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:51:21 DEBUG 06:51:21 DEBUG "status": "success", 06:51:21 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:51:21 DEBUG }, 06:51:21 DEBUG [...] 06:51:21 DEBUG "data": { 06:51:21 DEBUG }, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 06:51:21 DEBUG "resultType": "vector", 06:51:21 DEBUG "value": [ 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG "result": [ 06:51:21 DEBUG 1712814681, 06:51:21 DEBUG 06:51:21 DEBUG 06:51:21 DEBUG { 06:51:21 DEBUG "4.149000634918925" 06:51:21 DEBUG 06:51:21 DEBUG "pod": "ds-idrepo-0" 06:51:21 DEBUG "metric": { 06:51:21 DEBUG ] 06:51:21 DEBUG [...] 06:51:21 DEBUG }, 06:51:21 DEBUG "instance": "10.142.0.107:9100" 06:51:21 DEBUG } 06:51:21 DEBUG 06:51:21 DEBUG "value": [ 06:51:21 DEBUG }, 06:51:21 DEBUG ] 06:51:21 DEBUG 06:51:21 DEBUG 1712814681, 06:51:21 DEBUG "value": [ 06:51:21 DEBUG } 06:51:21 DEBUG 06:51:21 DEBUG "0" 06:51:21 DEBUG 06:51:21 DEBUG } 06:51:21 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 06:51:21 DEBUG ] 06:51:21 DEBUG 06:51:21 DEBUG }, 06:51:21 DEBUG } 06:51:21 DEBUG 06:51:21 DEBUG "value": [ 06:51:21 DEBUG ] 06:51:21 DEBUG [...] 06:51:21 DEBUG 1712814681, 06:51:21 DEBUG } 06:51:21 DEBUG 06:51:21 DEBUG "14063337472" 06:51:21 DEBUG } 06:51:21 DEBUG 06:51:21 DEBUG ] 06:51:21 DEBUG 06:51:21 DEBUG } 06:51:21 DEBUG "instance": "10.142.0.77:9100" 06:51:21 DEBUG ] 06:51:21 DEBUG }, 06:51:21 DEBUG } 06:51:21 DEBUG "value": [ 06:51:21 DEBUG } 06:51:21 DEBUG 1712814681, 06:51:21 DEBUG "0.000999999999999801" 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG ] 06:51:21 DEBUG } 06:51:21 DEBUG } 06:51:33 INFO 06:51:33 INFO [loop_until]: kubectl --namespace=xlou top pods 06:51:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:51:33 INFO [loop_until]: OK (rc = 0) 06:51:33 DEBUG --- stdout --- 06:51:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 9100m 8513Mi am-5b5bd8567b-8h2w4 8892m 8814Mi am-5b5bd8567b-qlkw6 7674m 8461Mi ds-cts-0 8040m 13394Mi ds-cts-1 8752m 12693Mi ds-cts-2 8495m 13395Mi ds-idrepo-0 11325m 5503Mi ds-idrepo-1 390m 5480Mi ds-idrepo-2 366m 5784Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 3m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 2m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4069m 1861Mi 06:51:33 DEBUG --- stderr --- 06:51:33 DEBUG 06:51:37 INFO 06:51:37 INFO [loop_until]: kubectl --namespace=xlou top node 06:51:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:51:37 INFO [loop_until]: OK (rc = 0) 06:51:37 DEBUG --- stdout --- 06:51:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 2739Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 62m 0% 2777Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7804m 26% 10119Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10752m 36% 2978Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9589m 32% 10090Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 147m 0% 3851Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9640m 32% 10434Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 409m 2% 6460Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11599m 72% 6203Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8688m 54% 13986Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 8219m 51% 14013Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 8759m 55% 13324Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 458m 2% 6159Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4256m 26% 5615Mi 9% 06:51:37 DEBUG --- stderr --- 06:51:37 DEBUG 06:52:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814741" 06:52:21 INFO [http_cmd]: http status code OK 06:52:21 DEBUG --- status code --- 06:52:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:21 DEBUG --- http response --- 06:52:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:52:21 DEBUG ----- output ----- 06:52:21 DEBUG { 06:52:21 DEBUG "status": "success", 06:52:21 DEBUG "data": { 06:52:21 DEBUG "resultType": "vector", 06:52:21 DEBUG "result": [ 06:52:21 DEBUG { 06:52:21 DEBUG "metric": { 06:52:21 DEBUG "pod": "ds-idrepo-2" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG [...] 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG "pod": "ds-idrepo-0" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 1712814741, 06:52:21 DEBUG "0" 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG } 06:52:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712814741" 06:52:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814741" 06:52:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814741" 06:52:21 ERROR [http_cmd]: ERROR 06:52:21 ERROR --- status code --- 06:52:21 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:52:21 ERROR --- http response --- 06:52:21 INFO [http_cmd]: http status code OK 06:52:21 DEBUG --- status code --- 06:52:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:21 DEBUG --- http response --- 06:52:21 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:52:21 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:52:21 INFO [http_cmd]: http status code OK 06:52:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:52:21 DEBUG --- status code --- 06:52:21 DEBUG ----- output ----- 06:52:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:21 DEBUG { 06:52:21 DEBUG --- http response --- 06:52:21 DEBUG "status": "success", 06:52:21 DEBUG "data": { 06:52:21 DEBUG "resultType": "vector", 06:52:21 DEBUG "result": [ 06:52:21 DEBUG { 06:52:21 DEBUG "metric": { 06:52:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG [...] 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 1712814741, 06:52:21 DEBUG "4.106318142480877" 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG } 06:52:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:52:21 DEBUG ----- output ----- 06:52:21 DEBUG { 06:52:21 DEBUG "status": "success", 06:52:21 DEBUG "data": { 06:52:21 DEBUG "resultType": "vector", 06:52:21 DEBUG "result": [ 06:52:21 DEBUG { 06:52:21 DEBUG "metric": { 06:52:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG [...] 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 1712814741, 06:52:21 DEBUG "83430.88771787596" 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG } 06:52:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814741" 06:52:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814741" 06:52:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712814741" 06:52:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814741" 06:52:21 INFO [http_cmd]: http status code OK 06:52:21 DEBUG --- status code --- 06:52:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:21 DEBUG --- http response --- 06:52:21 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:52:21 INFO [http_cmd]: http status code OK 06:52:21 INFO [http_cmd]: http status code OK 06:52:21 DEBUG --- status code --- 06:52:21 DEBUG --- status code --- 06:52:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:21 DEBUG --- http response --- 06:52:21 DEBUG --- http response --- 06:52:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:52:21 DEBUG ----- output ----- 06:52:21 DEBUG { 06:52:21 DEBUG "status": "success", 06:52:21 DEBUG "data": { 06:52:21 DEBUG "resultType": "vector", 06:52:21 DEBUG "result": [ 06:52:21 DEBUG { 06:52:21 DEBUG "metric": { 06:52:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG [...] 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 1712814741, 06:52:21 DEBUG "0.021" 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG } 06:52:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:52:21 DEBUG ----- output ----- 06:52:21 DEBUG { 06:52:21 DEBUG "status": "success", 06:52:21 DEBUG "data": { 06:52:21 DEBUG "resultType": "vector", 06:52:21 DEBUG "result": [ 06:52:21 DEBUG { 06:52:21 DEBUG "metric": { 06:52:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG [...] 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 1712814741, 06:52:21 DEBUG "244555" 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG } 06:52:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814741" 06:52:21 INFO [http_cmd]: http status code OK 06:52:21 DEBUG --- status code --- 06:52:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:21 DEBUG --- http response --- 06:52:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:52:21 DEBUG ----- output ----- 06:52:21 DEBUG { 06:52:21 DEBUG "status": "success", 06:52:21 DEBUG "data": { 06:52:21 DEBUG "resultType": "vector", 06:52:21 DEBUG "result": [ 06:52:21 DEBUG { 06:52:21 DEBUG "metric": { 06:52:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG [...] 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 1712814741, 06:52:21 DEBUG "1359.5666666666666" 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG } 06:52:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712814741" 06:52:21 INFO [http_cmd]: http status code OK 06:52:21 DEBUG --- status code --- 06:52:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:21 DEBUG --- http response --- 06:52:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:52:21 DEBUG ----- output ----- 06:52:21 DEBUG { 06:52:21 DEBUG "status": "success", 06:52:21 DEBUG "data": { 06:52:21 DEBUG "resultType": "vector", 06:52:21 DEBUG "result": [ 06:52:21 DEBUG { 06:52:21 DEBUG "metric": { 06:52:21 DEBUG "instance": "10.142.0.107:9100" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG [...] 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG "instance": "10.142.0.77:9100" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 1712814741, 06:52:21 DEBUG "82812683.4" 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG } 06:52:21 INFO [http_cmd]: http status code OK 06:52:21 DEBUG --- status code --- 06:52:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:21 DEBUG --- http response --- 06:52:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:52:21 DEBUG ----- output ----- 06:52:21 DEBUG { 06:52:21 DEBUG "status": "success", 06:52:21 DEBUG "data": { 06:52:21 DEBUG "resultType": "vector", 06:52:21 DEBUG "result": [ 06:52:21 DEBUG { 06:52:21 DEBUG "metric": { 06:52:21 DEBUG "pod": "ds-idrepo-2" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG [...] 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG "pod": "ds-cts-2" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 1712814741, 06:52:21 DEBUG "0" 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG } 06:52:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814741" 06:52:21 INFO [http_cmd]: http status code OK 06:52:21 DEBUG --- status code --- 06:52:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:21 DEBUG --- http response --- 06:52:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:52:21 DEBUG ----- output ----- 06:52:21 DEBUG { 06:52:21 DEBUG "status": "success", 06:52:21 DEBUG "data": { 06:52:21 DEBUG "resultType": "vector", 06:52:21 DEBUG "result": [ 06:52:21 DEBUG { 06:52:21 DEBUG "metric": { 06:52:21 DEBUG "instance": "10.142.0.107:9100" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG [...] 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG "instance": "10.142.0.77:9100" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 1712814741, 06:52:21 DEBUG "52964973568" 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG } 06:52:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814741" 06:52:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814741" 06:52:21 INFO [http_cmd]: http status code OK 06:52:21 DEBUG --- status code --- 06:52:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:21 DEBUG --- http response --- 06:52:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814741" 06:52:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:52:21 DEBUG ----- output ----- 06:52:21 DEBUG { 06:52:21 DEBUG "status": "success", 06:52:21 DEBUG "data": { 06:52:21 DEBUG "resultType": "vector", 06:52:21 DEBUG "result": [ 06:52:21 DEBUG { 06:52:21 DEBUG "metric": { 06:52:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG [...] 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 1712814741, 06:52:21 DEBUG "1361.8" 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG } 06:52:21 INFO [http_cmd]: http status code OK 06:52:21 DEBUG --- status code --- 06:52:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:21 DEBUG --- http response --- 06:52:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:52:21 INFO [http_cmd]: http status code OK 06:52:21 DEBUG ----- output ----- 06:52:21 DEBUG --- status code --- 06:52:21 DEBUG { 06:52:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:21 DEBUG "status": "success", 06:52:21 DEBUG --- http response --- 06:52:21 DEBUG "data": { 06:52:21 DEBUG "resultType": "vector", 06:52:21 DEBUG "result": [ 06:52:21 DEBUG { 06:52:21 DEBUG "metric": { 06:52:21 DEBUG "instance": "10.142.0.107:9100" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG [...] 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG "instance": "10.142.0.77:9100" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 1712814741, 06:52:21 DEBUG "82816946.9" 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG } 06:52:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:52:21 DEBUG ----- output ----- 06:52:21 DEBUG { 06:52:21 DEBUG "status": "success", 06:52:21 DEBUG "data": { 06:52:21 DEBUG "resultType": "vector", 06:52:21 DEBUG "result": [ 06:52:21 DEBUG { 06:52:21 DEBUG "metric": { 06:52:21 DEBUG "instance": "10.142.0.107:9100" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG [...] 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG "instance": "10.142.0.77:9100" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 1712814741, 06:52:21 DEBUG "0.0027000000000001025" 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG } 06:52:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814741" 06:52:21 INFO [http_cmd]: http status code OK 06:52:21 DEBUG --- status code --- 06:52:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:21 DEBUG --- http response --- 06:52:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:52:21 DEBUG ----- output ----- 06:52:21 DEBUG { 06:52:21 DEBUG "status": "success", 06:52:21 DEBUG "data": { 06:52:21 DEBUG "resultType": "vector", 06:52:21 DEBUG "result": [ 06:52:21 DEBUG { 06:52:21 DEBUG "metric": { 06:52:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG [...] 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 1712814741, 06:52:21 DEBUG "0" 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG } 06:52:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814741" 06:52:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814741" 06:52:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814741" 06:52:21 INFO [http_cmd]: http status code OK 06:52:21 DEBUG --- status code --- 06:52:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:21 DEBUG --- http response --- 06:52:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:52:21 DEBUG ----- output ----- 06:52:21 DEBUG { 06:52:21 DEBUG "status": "success", 06:52:21 DEBUG "data": { 06:52:21 DEBUG "resultType": "vector", 06:52:21 DEBUG "result": [ 06:52:21 DEBUG { 06:52:21 DEBUG "metric": { 06:52:21 DEBUG "pod": "ds-idrepo-2" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG [...] 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG "pod": "ds-idrepo-0" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 1712814741, 06:52:21 DEBUG "0" 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG } 06:52:21 INFO [http_cmd]: http status code OK 06:52:21 INFO [http_cmd]: http status code OK 06:52:21 DEBUG --- status code --- 06:52:21 DEBUG --- status code --- 06:52:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:21 DEBUG --- http response --- 06:52:21 DEBUG --- http response --- 06:52:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:52:21 DEBUG ----- output ----- 06:52:21 DEBUG { 06:52:21 DEBUG "status": "success", 06:52:21 DEBUG "data": { 06:52:21 DEBUG "resultType": "vector", 06:52:21 DEBUG "result": [ 06:52:21 DEBUG { 06:52:21 DEBUG "metric": { 06:52:21 DEBUG "pod": "ds-idrepo-2" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG [...] 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG "pod": "ds-cts-2" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 1712814741, 06:52:21 DEBUG "0" 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG } 06:52:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:52:21 DEBUG ----- output ----- 06:52:21 DEBUG { 06:52:21 DEBUG "status": "success", 06:52:21 DEBUG "data": { 06:52:21 DEBUG "resultType": "vector", 06:52:21 DEBUG "result": [ 06:52:21 DEBUG { 06:52:21 DEBUG "metric": { 06:52:21 DEBUG "pod": "ds-idrepo-2" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG [...] 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG "pod": "ds-cts-2" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 1712814741, 06:52:21 DEBUG "29.7" 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG } 06:52:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814741" 06:52:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814741" 06:52:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712814741" 06:52:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712814741" 06:52:21 INFO [http_cmd]: http status code OK 06:52:21 DEBUG --- status code --- 06:52:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:21 DEBUG --- http response --- 06:52:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:52:21 DEBUG ----- output ----- 06:52:21 DEBUG { 06:52:21 DEBUG "status": "success", 06:52:21 DEBUG "data": { 06:52:21 DEBUG "resultType": "vector", 06:52:21 DEBUG "result": [ 06:52:21 DEBUG { 06:52:21 DEBUG "metric": { 06:52:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG [...] 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 1712814741, 06:52:21 DEBUG "1362.1333333333332" 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG } 06:52:21 INFO [http_cmd]: http status code OK 06:52:21 DEBUG --- status code --- 06:52:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:21 DEBUG --- http response --- 06:52:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:52:21 DEBUG ----- output ----- 06:52:21 DEBUG { 06:52:21 DEBUG "status": "success", 06:52:21 DEBUG "data": { 06:52:21 DEBUG "resultType": "vector", 06:52:21 DEBUG "result": [ 06:52:21 DEBUG { 06:52:21 DEBUG "metric": { 06:52:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG [...] 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 1712814741, 06:52:21 DEBUG "0" 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG } 06:52:21 INFO [http_cmd]: http status code OK 06:52:21 DEBUG --- status code --- 06:52:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:21 DEBUG --- http response --- 06:52:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:52:21 DEBUG ----- output ----- 06:52:21 DEBUG { 06:52:21 DEBUG "status": "success", 06:52:21 DEBUG "data": { 06:52:21 INFO [http_cmd]: http status code OK 06:52:21 DEBUG "resultType": "vector", 06:52:21 DEBUG --- status code --- 06:52:21 DEBUG "result": [ 06:52:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:21 DEBUG { 06:52:21 DEBUG --- http response --- 06:52:21 DEBUG "metric": { 06:52:21 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG [...] 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 1712814741, 06:52:21 DEBUG "14135119872" 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:52:21 DEBUG ] 06:52:21 DEBUG ----- output ----- 06:52:21 DEBUG } 06:52:21 DEBUG { 06:52:21 DEBUG } 06:52:21 DEBUG "status": "success", 06:52:21 DEBUG "data": { 06:52:21 DEBUG "resultType": "vector", 06:52:21 DEBUG "result": [ 06:52:21 DEBUG { 06:52:21 DEBUG "metric": { 06:52:21 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG [...] 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG 06:52:21 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:52:21 DEBUG }, 06:52:21 DEBUG "value": [ 06:52:21 DEBUG 1712814741, 06:52:21 DEBUG "4.138578510146296" 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG ] 06:52:21 DEBUG } 06:52:21 DEBUG } 06:52:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814742" 06:52:22 INFO [http_cmd]: http status code OK 06:52:22 DEBUG --- status code --- 06:52:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:22 DEBUG --- http response --- 06:52:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814742" 06:52:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:52:22 DEBUG ----- output ----- 06:52:22 DEBUG { 06:52:22 DEBUG "status": "success", 06:52:22 DEBUG "data": { 06:52:22 DEBUG "resultType": "vector", 06:52:22 DEBUG "result": [ 06:52:22 DEBUG { 06:52:22 DEBUG "metric": { 06:52:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:52:22 DEBUG }, 06:52:22 DEBUG "value": [ 06:52:22 DEBUG 06:52:22 DEBUG 06:52:22 DEBUG 06:52:22 DEBUG [...] 06:52:22 DEBUG 06:52:22 DEBUG 06:52:22 DEBUG 06:52:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:52:22 DEBUG }, 06:52:22 DEBUG "value": [ 06:52:22 DEBUG 1712814742, 06:52:22 DEBUG "0" 06:52:22 DEBUG ] 06:52:22 DEBUG } 06:52:22 DEBUG ] 06:52:22 DEBUG } 06:52:22 DEBUG } 06:52:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814742" 06:52:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814742" 06:52:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814742" 06:52:22 INFO [http_cmd]: http status code OK 06:52:22 DEBUG --- status code --- 06:52:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:22 DEBUG --- http response --- 06:52:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:52:22 DEBUG ----- output ----- 06:52:22 DEBUG { 06:52:22 DEBUG "status": "success", 06:52:22 DEBUG "data": { 06:52:22 DEBUG "resultType": "vector", 06:52:22 DEBUG "result": [ 06:52:22 DEBUG { 06:52:22 DEBUG "metric": { 06:52:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:52:22 DEBUG }, 06:52:22 DEBUG "value": [ 06:52:22 DEBUG 06:52:22 DEBUG 06:52:22 DEBUG 06:52:22 DEBUG [...] 06:52:22 DEBUG 06:52:22 DEBUG 06:52:22 DEBUG 06:52:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:52:22 DEBUG }, 06:52:22 DEBUG "value": [ 06:52:22 DEBUG 1712814742, 06:52:22 DEBUG "1362.0333333333333" 06:52:22 DEBUG ] 06:52:22 DEBUG } 06:52:22 DEBUG ] 06:52:22 DEBUG } 06:52:22 DEBUG } 06:52:22 INFO [http_cmd]: http status code OK 06:52:22 INFO [http_cmd]: http status code OK 06:52:22 DEBUG --- status code --- 06:52:22 DEBUG --- status code --- 06:52:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:22 DEBUG --- http response --- 06:52:22 DEBUG --- http response --- 06:52:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:52:22 DEBUG ----- output ----- 06:52:22 DEBUG { 06:52:22 DEBUG "status": "success", 06:52:22 DEBUG "data": { 06:52:22 DEBUG "resultType": "vector", 06:52:22 DEBUG "result": [ 06:52:22 DEBUG { 06:52:22 DEBUG "metric": { 06:52:22 DEBUG "pod": "ds-idrepo-2" 06:52:22 DEBUG }, 06:52:22 DEBUG "value": [ 06:52:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:52:22 DEBUG 06:52:22 DEBUG ----- output ----- 06:52:22 DEBUG 06:52:22 DEBUG { 06:52:22 DEBUG 06:52:22 DEBUG "status": "success", 06:52:22 DEBUG [...] 06:52:22 DEBUG "data": { 06:52:22 DEBUG 06:52:22 DEBUG "resultType": "vector", 06:52:22 DEBUG 06:52:22 DEBUG "result": [ 06:52:22 DEBUG 06:52:22 DEBUG { 06:52:22 DEBUG "pod": "ds-idrepo-0" 06:52:22 DEBUG "metric": { 06:52:22 DEBUG }, 06:52:22 DEBUG "pod": "ds-idrepo-2" 06:52:22 DEBUG "value": [ 06:52:22 DEBUG }, 06:52:22 DEBUG 1712814742, 06:52:22 DEBUG "value": [ 06:52:22 DEBUG "0" 06:52:22 DEBUG 06:52:22 DEBUG ] 06:52:22 DEBUG 06:52:22 DEBUG } 06:52:22 DEBUG 06:52:22 DEBUG ] 06:52:22 DEBUG [...] 06:52:22 DEBUG } 06:52:22 DEBUG 06:52:22 DEBUG } 06:52:22 DEBUG 06:52:22 DEBUG 06:52:22 DEBUG "pod": "ds-cts-2" 06:52:22 DEBUG }, 06:52:22 DEBUG "value": [ 06:52:22 DEBUG 1712814742, 06:52:22 DEBUG "0.00006666666666666667" 06:52:22 DEBUG ] 06:52:22 DEBUG } 06:52:22 DEBUG ] 06:52:22 DEBUG } 06:52:22 DEBUG } 06:52:22 INFO [http_cmd]: http status code OK 06:52:22 DEBUG --- status code --- 06:52:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:52:22 DEBUG --- http response --- 06:52:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:52:22 DEBUG ----- output ----- 06:52:22 DEBUG { 06:52:22 DEBUG "status": "success", 06:52:22 DEBUG "data": { 06:52:22 DEBUG "resultType": "vector", 06:52:22 DEBUG "result": [ 06:52:22 DEBUG { 06:52:22 DEBUG "metric": { 06:52:22 DEBUG "instance": "10.142.0.107:9100" 06:52:22 DEBUG }, 06:52:22 DEBUG "value": [ 06:52:22 DEBUG 06:52:22 DEBUG 06:52:22 DEBUG 06:52:22 DEBUG [...] 06:52:22 DEBUG 06:52:22 DEBUG 06:52:22 DEBUG 06:52:22 DEBUG "instance": "10.142.0.77:9100" 06:52:22 DEBUG }, 06:52:22 DEBUG "value": [ 06:52:22 DEBUG 1712814742, 06:52:22 DEBUG "0.0006666666666670077" 06:52:22 DEBUG ] 06:52:22 DEBUG } 06:52:22 DEBUG ] 06:52:22 DEBUG } 06:52:22 DEBUG } 06:52:33 INFO 06:52:33 INFO [loop_until]: kubectl --namespace=xlou top pods 06:52:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:52:33 INFO [loop_until]: OK (rc = 0) 06:52:33 DEBUG --- stdout --- 06:52:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 9157m 8509Mi am-5b5bd8567b-8h2w4 8500m 8820Mi am-5b5bd8567b-qlkw6 7740m 8458Mi ds-cts-0 7830m 13392Mi ds-cts-1 8572m 13399Mi ds-cts-2 8450m 12561Mi ds-idrepo-0 11523m 5503Mi ds-idrepo-1 353m 5480Mi ds-idrepo-2 357m 5784Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 5m 1011Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 4m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4116m 1862Mi 06:52:33 DEBUG --- stderr --- 06:52:33 DEBUG 06:52:37 INFO 06:52:37 INFO [loop_until]: kubectl --namespace=xlou top node 06:52:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:52:37 INFO [loop_until]: OK (rc = 0) 06:52:37 DEBUG --- stdout --- 06:52:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 2728Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 63m 0% 2781Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7791m 26% 10109Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10778m 36% 2983Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9702m 32% 10099Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 146m 0% 3864Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9277m 31% 10449Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 406m 2% 6464Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11525m 72% 6204Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8559m 53% 13169Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 8113m 51% 14021Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 8888m 55% 14033Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 408m 2% 6157Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4247m 26% 5614Mi 9% 06:52:37 DEBUG --- stderr --- 06:52:37 DEBUG 06:53:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814801" 06:53:21 INFO [http_cmd]: http status code OK 06:53:21 DEBUG --- status code --- 06:53:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:21 DEBUG --- http response --- 06:53:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:53:21 DEBUG ----- output ----- 06:53:21 DEBUG { 06:53:21 DEBUG "status": "success", 06:53:21 DEBUG "data": { 06:53:21 DEBUG "resultType": "vector", 06:53:21 DEBUG "result": [ 06:53:21 DEBUG { 06:53:21 DEBUG "metric": { 06:53:21 DEBUG "pod": "ds-idrepo-2" 06:53:21 DEBUG }, 06:53:21 DEBUG "value": [ 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG [...] 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG "pod": "ds-idrepo-0" 06:53:21 DEBUG }, 06:53:21 DEBUG "value": [ 06:53:21 DEBUG 1712814801, 06:53:21 DEBUG "0" 06:53:21 DEBUG ] 06:53:21 DEBUG } 06:53:21 DEBUG ] 06:53:21 DEBUG } 06:53:21 DEBUG } 06:53:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712814801" 06:53:21 ERROR [http_cmd]: ERROR 06:53:21 ERROR --- status code --- 06:53:21 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:53:21 ERROR --- http response --- 06:53:21 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:53:21 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:53:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814801" 06:53:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814801" 06:53:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814801" 06:53:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814801" 06:53:21 INFO [http_cmd]: http status code OK 06:53:21 INFO [http_cmd]: http status code OK 06:53:21 DEBUG --- status code --- 06:53:21 DEBUG --- status code --- 06:53:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:21 DEBUG --- http response --- 06:53:21 DEBUG --- http response --- 06:53:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:53:21 DEBUG ----- output ----- 06:53:21 DEBUG { 06:53:21 DEBUG "status": "success", 06:53:21 DEBUG "data": { 06:53:21 DEBUG "resultType": "vector", 06:53:21 DEBUG "result": [ 06:53:21 DEBUG { 06:53:21 DEBUG "metric": { 06:53:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:53:21 DEBUG }, 06:53:21 DEBUG "value": [ 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG [...] 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG "pod": "ds-cts-0" 06:53:21 DEBUG }, 06:53:21 DEBUG "value": [ 06:53:21 DEBUG 1712814801, 06:53:21 DEBUG "7.872962124352537" 06:53:21 DEBUG ] 06:53:21 DEBUG } 06:53:21 DEBUG ] 06:53:21 DEBUG } 06:53:21 DEBUG } 06:53:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:53:21 DEBUG ----- output ----- 06:53:21 DEBUG { 06:53:21 DEBUG "status": "success", 06:53:21 DEBUG "data": { 06:53:21 DEBUG "resultType": "vector", 06:53:21 DEBUG "result": [ 06:53:21 DEBUG { 06:53:21 DEBUG "metric": { 06:53:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:53:21 DEBUG }, 06:53:21 DEBUG "value": [ 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG [...] 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG "pod": "ds-cts-0" 06:53:21 DEBUG }, 06:53:21 DEBUG "value": [ 06:53:21 DEBUG 1712814801, 06:53:21 DEBUG "11307181.3662437" 06:53:21 DEBUG ] 06:53:21 DEBUG } 06:53:21 DEBUG ] 06:53:21 DEBUG } 06:53:21 DEBUG } 06:53:21 INFO [http_cmd]: http status code OK 06:53:21 DEBUG --- status code --- 06:53:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:21 DEBUG --- http response --- 06:53:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:53:21 DEBUG ----- output ----- 06:53:21 DEBUG { 06:53:21 DEBUG "status": "success", 06:53:21 DEBUG "data": { 06:53:21 DEBUG "resultType": "vector", 06:53:21 DEBUG "result": [ 06:53:21 DEBUG { 06:53:21 DEBUG "metric": { 06:53:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:53:21 DEBUG }, 06:53:21 DEBUG "value": [ 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG [...] 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:53:21 DEBUG }, 06:53:21 DEBUG "value": [ 06:53:21 DEBUG 1712814801, 06:53:21 DEBUG "245011" 06:53:21 DEBUG ] 06:53:21 DEBUG } 06:53:21 DEBUG ] 06:53:21 DEBUG } 06:53:21 DEBUG } 06:53:21 INFO [http_cmd]: http status code OK 06:53:21 DEBUG --- status code --- 06:53:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:21 DEBUG --- http response --- 06:53:21 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:53:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814801" 06:53:21 INFO [http_cmd]: http status code OK 06:53:21 DEBUG --- status code --- 06:53:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:21 DEBUG --- http response --- 06:53:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:53:21 DEBUG ----- output ----- 06:53:21 DEBUG { 06:53:21 DEBUG "status": "success", 06:53:21 DEBUG "data": { 06:53:21 DEBUG "resultType": "vector", 06:53:21 DEBUG "result": [ 06:53:21 DEBUG { 06:53:21 DEBUG "metric": { 06:53:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:53:21 DEBUG }, 06:53:21 DEBUG "value": [ 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG [...] 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:53:21 DEBUG }, 06:53:21 DEBUG "value": [ 06:53:21 DEBUG 1712814801, 06:53:21 DEBUG "1357.9666666666667" 06:53:21 DEBUG ] 06:53:21 DEBUG } 06:53:21 DEBUG ] 06:53:21 DEBUG } 06:53:21 DEBUG } 06:53:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712814801" 06:53:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814801" 06:53:21 INFO [http_cmd]: http status code OK 06:53:21 DEBUG --- status code --- 06:53:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:21 DEBUG --- http response --- 06:53:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:53:21 DEBUG ----- output ----- 06:53:21 DEBUG { 06:53:21 DEBUG "status": "success", 06:53:21 DEBUG "data": { 06:53:21 DEBUG "resultType": "vector", 06:53:21 DEBUG "result": [ 06:53:21 DEBUG { 06:53:21 DEBUG "metric": { 06:53:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:53:21 DEBUG }, 06:53:21 DEBUG "value": [ 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG [...] 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:53:21 DEBUG }, 06:53:21 DEBUG "value": [ 06:53:21 DEBUG 1712814801, 06:53:21 DEBUG "0.021" 06:53:21 DEBUG ] 06:53:21 DEBUG } 06:53:21 DEBUG ] 06:53:21 DEBUG } 06:53:21 DEBUG } 06:53:21 INFO [http_cmd]: http status code OK 06:53:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712814801" 06:53:21 DEBUG --- status code --- 06:53:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:21 DEBUG --- http response --- 06:53:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:53:21 DEBUG ----- output ----- 06:53:21 DEBUG { 06:53:21 DEBUG "status": "success", 06:53:21 DEBUG "data": { 06:53:21 DEBUG "resultType": "vector", 06:53:21 DEBUG "result": [ 06:53:21 DEBUG { 06:53:21 DEBUG "metric": { 06:53:21 DEBUG "instance": "10.142.0.107:9100" 06:53:21 DEBUG }, 06:53:21 DEBUG "value": [ 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG [...] 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG "instance": "10.142.0.77:9100" 06:53:21 DEBUG }, 06:53:21 DEBUG "value": [ 06:53:21 DEBUG 1712814801, 06:53:21 DEBUG "82822195.03333333" 06:53:21 DEBUG ] 06:53:21 DEBUG } 06:53:21 DEBUG ] 06:53:21 DEBUG } 06:53:21 DEBUG } 06:53:21 INFO [http_cmd]: http status code OK 06:53:21 DEBUG --- status code --- 06:53:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:21 DEBUG --- http response --- 06:53:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:53:21 DEBUG ----- output ----- 06:53:21 DEBUG { 06:53:21 DEBUG "status": "success", 06:53:21 DEBUG "data": { 06:53:21 DEBUG "resultType": "vector", 06:53:21 DEBUG "result": [ 06:53:21 DEBUG { 06:53:21 DEBUG "metric": { 06:53:21 DEBUG "pod": "ds-idrepo-2" 06:53:21 DEBUG }, 06:53:21 DEBUG "value": [ 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG [...] 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG "pod": "ds-cts-2" 06:53:21 DEBUG }, 06:53:21 DEBUG "value": [ 06:53:21 DEBUG 1712814801, 06:53:21 DEBUG "0" 06:53:21 DEBUG ] 06:53:21 DEBUG } 06:53:21 DEBUG ] 06:53:21 DEBUG } 06:53:21 DEBUG } 06:53:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814801" 06:53:21 INFO [http_cmd]: http status code OK 06:53:21 DEBUG --- status code --- 06:53:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:21 DEBUG --- http response --- 06:53:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:53:21 DEBUG ----- output ----- 06:53:21 DEBUG { 06:53:21 DEBUG "status": "success", 06:53:21 DEBUG "data": { 06:53:21 DEBUG "resultType": "vector", 06:53:21 DEBUG "result": [ 06:53:21 DEBUG { 06:53:21 DEBUG "metric": { 06:53:21 DEBUG "instance": "10.142.0.107:9100" 06:53:21 DEBUG }, 06:53:21 DEBUG "value": [ 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG [...] 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG "instance": "10.142.0.77:9100" 06:53:21 DEBUG }, 06:53:21 DEBUG "value": [ 06:53:21 DEBUG 1712814801, 06:53:21 DEBUG "52966347776" 06:53:21 DEBUG ] 06:53:21 DEBUG } 06:53:21 DEBUG ] 06:53:21 DEBUG } 06:53:21 DEBUG } 06:53:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814801" 06:53:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814801" 06:53:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814801" 06:53:21 INFO [http_cmd]: http status code OK 06:53:21 DEBUG --- status code --- 06:53:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:21 DEBUG --- http response --- 06:53:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:53:21 DEBUG ----- output ----- 06:53:21 DEBUG { 06:53:21 DEBUG "status": "success", 06:53:21 DEBUG "data": { 06:53:21 DEBUG "resultType": "vector", 06:53:21 DEBUG "result": [ 06:53:21 DEBUG { 06:53:21 DEBUG "metric": { 06:53:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:53:21 DEBUG }, 06:53:21 DEBUG "value": [ 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG [...] 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:53:21 DEBUG }, 06:53:21 DEBUG "value": [ 06:53:21 DEBUG 1712814801, 06:53:21 DEBUG "1368.4333333333334" 06:53:21 DEBUG ] 06:53:21 DEBUG } 06:53:21 DEBUG ] 06:53:21 DEBUG } 06:53:21 DEBUG } 06:53:21 INFO [http_cmd]: http status code OK 06:53:21 INFO [http_cmd]: http status code OK 06:53:21 DEBUG --- status code --- 06:53:21 DEBUG --- status code --- 06:53:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:21 DEBUG --- http response --- 06:53:21 DEBUG --- http response --- 06:53:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:53:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:53:21 DEBUG ----- output ----- 06:53:21 DEBUG ----- output ----- 06:53:21 DEBUG { 06:53:21 DEBUG { 06:53:21 DEBUG "status": "success", 06:53:21 DEBUG "status": "success", 06:53:21 DEBUG "data": { 06:53:21 DEBUG "data": { 06:53:21 DEBUG "resultType": "vector", 06:53:21 DEBUG "resultType": "vector", 06:53:21 DEBUG "result": [ 06:53:21 DEBUG "result": [ 06:53:21 DEBUG { 06:53:21 DEBUG { 06:53:21 DEBUG "metric": { 06:53:21 DEBUG "metric": { 06:53:21 DEBUG "instance": "10.142.0.107:9100" 06:53:21 DEBUG "instance": "10.142.0.107:9100" 06:53:21 DEBUG }, 06:53:21 DEBUG }, 06:53:21 DEBUG "value": [ 06:53:21 DEBUG "value": [ 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG [...] 06:53:21 DEBUG [...] 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG "instance": "10.142.0.77:9100" 06:53:21 DEBUG "instance": "10.142.0.77:9100" 06:53:21 DEBUG }, 06:53:21 DEBUG }, 06:53:21 DEBUG "value": [ 06:53:21 DEBUG "value": [ 06:53:21 DEBUG 1712814801, 06:53:21 DEBUG 1712814801, 06:53:21 DEBUG "0.0023333333333333426" 06:53:21 DEBUG "82824796.73333333" 06:53:21 DEBUG ] 06:53:21 DEBUG ] 06:53:21 DEBUG } 06:53:21 DEBUG } 06:53:21 DEBUG ] 06:53:21 DEBUG ] 06:53:21 DEBUG } 06:53:21 DEBUG } 06:53:21 DEBUG } 06:53:21 DEBUG } 06:53:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814801" 06:53:21 INFO [http_cmd]: http status code OK 06:53:21 DEBUG --- status code --- 06:53:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:21 DEBUG --- http response --- 06:53:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:53:21 DEBUG ----- output ----- 06:53:21 DEBUG { 06:53:21 DEBUG "status": "success", 06:53:21 DEBUG "data": { 06:53:21 DEBUG "resultType": "vector", 06:53:21 DEBUG "result": [ 06:53:21 DEBUG { 06:53:21 DEBUG "metric": { 06:53:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:53:21 DEBUG }, 06:53:21 DEBUG "value": [ 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG [...] 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG "pod": "ds-cts-0" 06:53:21 DEBUG }, 06:53:21 DEBUG "value": [ 06:53:21 DEBUG 1712814801, 06:53:21 DEBUG "25215639.56196767" 06:53:21 DEBUG ] 06:53:21 DEBUG } 06:53:21 DEBUG ] 06:53:21 DEBUG } 06:53:21 DEBUG } 06:53:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814801" 06:53:21 INFO [http_cmd]: http status code OK 06:53:21 DEBUG --- status code --- 06:53:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:21 DEBUG --- http response --- 06:53:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:53:21 DEBUG ----- output ----- 06:53:21 DEBUG { 06:53:21 DEBUG "status": "success", 06:53:21 DEBUG "data": { 06:53:21 DEBUG "resultType": "vector", 06:53:21 DEBUG "result": [ 06:53:21 DEBUG { 06:53:21 DEBUG "metric": { 06:53:21 DEBUG "pod": "ds-idrepo-2" 06:53:21 DEBUG }, 06:53:21 DEBUG "value": [ 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG [...] 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG 06:53:21 DEBUG "pod": "ds-idrepo-0" 06:53:21 DEBUG }, 06:53:21 DEBUG "value": [ 06:53:21 DEBUG 1712814801, 06:53:21 DEBUG "0" 06:53:21 DEBUG ] 06:53:21 DEBUG } 06:53:21 DEBUG ] 06:53:21 DEBUG } 06:53:21 DEBUG } 06:53:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814802" 06:53:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814802" 06:53:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712814802" 06:53:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712814802" 06:53:22 INFO [http_cmd]: http status code OK 06:53:22 INFO [http_cmd]: http status code OK 06:53:22 DEBUG --- status code --- 06:53:22 DEBUG --- status code --- 06:53:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:22 DEBUG --- http response --- 06:53:22 DEBUG --- http response --- 06:53:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:53:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:53:22 DEBUG ----- output ----- 06:53:22 DEBUG ----- output ----- 06:53:22 DEBUG { 06:53:22 DEBUG { 06:53:22 DEBUG "status": "success", 06:53:22 DEBUG "status": "success", 06:53:22 DEBUG "data": { 06:53:22 DEBUG "data": { 06:53:22 DEBUG "resultType": "vector", 06:53:22 DEBUG "resultType": "vector", 06:53:22 DEBUG "result": [ 06:53:22 DEBUG "result": [ 06:53:22 DEBUG { 06:53:22 DEBUG { 06:53:22 DEBUG "metric": { 06:53:22 DEBUG "metric": { 06:53:22 DEBUG "pod": "ds-idrepo-2" 06:53:22 DEBUG "pod": "ds-idrepo-2" 06:53:22 DEBUG }, 06:53:22 DEBUG }, 06:53:22 DEBUG "value": [ 06:53:22 DEBUG "value": [ 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG [...] 06:53:22 DEBUG [...] 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 INFO [http_cmd]: http status code OK 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG --- status code --- 06:53:22 DEBUG "pod": "ds-cts-2" 06:53:22 DEBUG 06:53:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:22 DEBUG }, 06:53:22 DEBUG "pod": "ds-cts-2" 06:53:22 DEBUG --- http response --- 06:53:22 DEBUG "value": [ 06:53:22 DEBUG }, 06:53:22 DEBUG 1712814802, 06:53:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:53:22 DEBUG "value": [ 06:53:22 DEBUG "0" 06:53:22 DEBUG ----- output ----- 06:53:22 DEBUG 1712814802, 06:53:22 DEBUG ] 06:53:22 DEBUG { 06:53:22 DEBUG "29.9" 06:53:22 DEBUG } 06:53:22 DEBUG "status": "success", 06:53:22 DEBUG ] 06:53:22 DEBUG ] 06:53:22 INFO [http_cmd]: http status code OK 06:53:22 DEBUG "data": { 06:53:22 DEBUG } 06:53:22 DEBUG } 06:53:22 DEBUG --- status code --- 06:53:22 DEBUG "resultType": "vector", 06:53:22 DEBUG ] 06:53:22 DEBUG } 06:53:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:22 DEBUG "result": [ 06:53:22 DEBUG } 06:53:22 DEBUG --- http response --- 06:53:22 DEBUG { 06:53:22 DEBUG } 06:53:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:53:22 DEBUG "metric": { 06:53:22 DEBUG ----- output ----- 06:53:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:53:22 DEBUG { 06:53:22 DEBUG }, 06:53:22 DEBUG "status": "success", 06:53:22 DEBUG "value": [ 06:53:22 DEBUG "data": { 06:53:22 DEBUG 06:53:22 DEBUG "resultType": "vector", 06:53:22 DEBUG 06:53:22 DEBUG "result": [ 06:53:22 DEBUG 06:53:22 DEBUG { 06:53:22 DEBUG [...] 06:53:22 DEBUG "metric": { 06:53:22 DEBUG 06:53:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:53:22 DEBUG 06:53:22 DEBUG }, 06:53:22 DEBUG 06:53:22 DEBUG "value": [ 06:53:22 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:53:22 DEBUG 06:53:22 DEBUG }, 06:53:22 DEBUG 06:53:22 DEBUG "value": [ 06:53:22 DEBUG 06:53:22 DEBUG 1712814802, 06:53:22 DEBUG [...] 06:53:22 DEBUG "4.140080249284746" 06:53:22 DEBUG 06:53:22 DEBUG ] 06:53:22 DEBUG 06:53:22 DEBUG } 06:53:22 DEBUG 06:53:22 DEBUG ] 06:53:22 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 06:53:22 DEBUG } 06:53:22 DEBUG }, 06:53:22 DEBUG } 06:53:22 DEBUG "value": [ 06:53:22 DEBUG 1712814802, 06:53:22 DEBUG "13462425600" 06:53:22 DEBUG ] 06:53:22 DEBUG } 06:53:22 DEBUG ] 06:53:22 DEBUG } 06:53:22 DEBUG } 06:53:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814802" 06:53:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814802" 06:53:22 INFO [http_cmd]: http status code OK 06:53:22 DEBUG --- status code --- 06:53:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:22 DEBUG --- http response --- 06:53:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:53:22 DEBUG ----- output ----- 06:53:22 DEBUG { 06:53:22 DEBUG "status": "success", 06:53:22 DEBUG "data": { 06:53:22 DEBUG "resultType": "vector", 06:53:22 DEBUG "result": [ 06:53:22 DEBUG { 06:53:22 DEBUG "metric": { 06:53:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:53:22 DEBUG }, 06:53:22 DEBUG "value": [ 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG [...] 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:53:22 DEBUG }, 06:53:22 DEBUG "value": [ 06:53:22 DEBUG 1712814802, 06:53:22 DEBUG "1368.2" 06:53:22 DEBUG ] 06:53:22 DEBUG } 06:53:22 DEBUG ] 06:53:22 DEBUG } 06:53:22 DEBUG } 06:53:22 INFO [http_cmd]: http status code OK 06:53:22 DEBUG --- status code --- 06:53:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:22 DEBUG --- http response --- 06:53:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 06:53:22 DEBUG ----- output ----- 06:53:22 DEBUG { 06:53:22 DEBUG "status": "success", 06:53:22 DEBUG "data": { 06:53:22 DEBUG "resultType": "vector", 06:53:22 DEBUG "result": [ 06:53:22 DEBUG { 06:53:22 DEBUG "metric": { 06:53:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:53:22 DEBUG }, 06:53:22 DEBUG "value": [ 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG [...] 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG "pod": "ds-cts-0" 06:53:22 DEBUG }, 06:53:22 DEBUG "value": [ 06:53:22 DEBUG 1712814802, 06:53:22 DEBUG "1139.1621762558666" 06:53:22 DEBUG ] 06:53:22 DEBUG } 06:53:22 DEBUG ] 06:53:22 DEBUG } 06:53:22 DEBUG } 06:53:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814802" 06:53:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814802" 06:53:22 INFO [http_cmd]: http status code OK 06:53:22 DEBUG --- status code --- 06:53:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:22 DEBUG --- http response --- 06:53:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:53:22 DEBUG ----- output ----- 06:53:22 DEBUG { 06:53:22 DEBUG "status": "success", 06:53:22 DEBUG "data": { 06:53:22 DEBUG "resultType": "vector", 06:53:22 DEBUG "result": [ 06:53:22 DEBUG { 06:53:22 DEBUG "metric": { 06:53:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:53:22 DEBUG }, 06:53:22 DEBUG "value": [ 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG [...] 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:53:22 DEBUG }, 06:53:22 DEBUG "value": [ 06:53:22 DEBUG 1712814802, 06:53:22 DEBUG "0" 06:53:22 DEBUG ] 06:53:22 DEBUG } 06:53:22 DEBUG ] 06:53:22 DEBUG } 06:53:22 DEBUG } 06:53:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814802" 06:53:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814802" 06:53:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814802" 06:53:22 INFO [http_cmd]: http status code OK 06:53:22 DEBUG --- status code --- 06:53:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:22 DEBUG --- http response --- 06:53:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:53:22 DEBUG ----- output ----- 06:53:22 DEBUG { 06:53:22 DEBUG "status": "success", 06:53:22 DEBUG "data": { 06:53:22 DEBUG "resultType": "vector", 06:53:22 DEBUG "result": [ 06:53:22 DEBUG { 06:53:22 DEBUG "metric": { 06:53:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:53:22 DEBUG }, 06:53:22 DEBUG "value": [ 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG [...] 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:53:22 DEBUG }, 06:53:22 DEBUG "value": [ 06:53:22 DEBUG 1712814802, 06:53:22 DEBUG "1374.1" 06:53:22 DEBUG ] 06:53:22 DEBUG } 06:53:22 DEBUG ] 06:53:22 DEBUG } 06:53:22 DEBUG } 06:53:22 INFO [http_cmd]: http status code OK 06:53:22 DEBUG --- status code --- 06:53:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:22 DEBUG --- http response --- 06:53:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:53:22 DEBUG ----- output ----- 06:53:22 DEBUG { 06:53:22 DEBUG "status": "success", 06:53:22 DEBUG "data": { 06:53:22 DEBUG "resultType": "vector", 06:53:22 DEBUG "result": [ 06:53:22 DEBUG { 06:53:22 DEBUG "metric": { 06:53:22 DEBUG "pod": "ds-idrepo-2" 06:53:22 DEBUG }, 06:53:22 DEBUG "value": [ 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG [...] 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG "pod": "ds-cts-2" 06:53:22 DEBUG }, 06:53:22 DEBUG "value": [ 06:53:22 DEBUG 1712814802, 06:53:22 DEBUG "0.000030705" 06:53:22 DEBUG ] 06:53:22 DEBUG } 06:53:22 DEBUG ] 06:53:22 DEBUG } 06:53:22 DEBUG } 06:53:22 INFO [http_cmd]: http status code OK 06:53:22 DEBUG --- status code --- 06:53:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:22 DEBUG --- http response --- 06:53:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:53:22 DEBUG ----- output ----- 06:53:22 DEBUG { 06:53:22 DEBUG "status": "success", 06:53:22 DEBUG "data": { 06:53:22 DEBUG "resultType": "vector", 06:53:22 DEBUG "result": [ 06:53:22 DEBUG { 06:53:22 DEBUG "metric": { 06:53:22 DEBUG "pod": "ds-idrepo-2" 06:53:22 DEBUG }, 06:53:22 DEBUG "value": [ 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG [...] 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG "pod": "ds-idrepo-0" 06:53:22 DEBUG }, 06:53:22 DEBUG "value": [ 06:53:22 DEBUG 1712814802, 06:53:22 DEBUG "0" 06:53:22 DEBUG ] 06:53:22 DEBUG } 06:53:22 DEBUG ] 06:53:22 DEBUG } 06:53:22 DEBUG } 06:53:22 INFO [http_cmd]: http status code OK 06:53:22 DEBUG --- status code --- 06:53:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:53:22 DEBUG --- http response --- 06:53:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:53:22 DEBUG ----- output ----- 06:53:22 DEBUG { 06:53:22 DEBUG "status": "success", 06:53:22 DEBUG "data": { 06:53:22 DEBUG "resultType": "vector", 06:53:22 DEBUG "result": [ 06:53:22 DEBUG { 06:53:22 DEBUG "metric": { 06:53:22 DEBUG "instance": "10.142.0.107:9100" 06:53:22 DEBUG }, 06:53:22 DEBUG "value": [ 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG [...] 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG 06:53:22 DEBUG "instance": "10.142.0.77:9100" 06:53:22 DEBUG }, 06:53:22 DEBUG "value": [ 06:53:22 DEBUG 1712814802, 06:53:22 DEBUG "0.0009999999999999194" 06:53:22 DEBUG ] 06:53:22 DEBUG } 06:53:22 DEBUG ] 06:53:22 DEBUG } 06:53:22 DEBUG } 06:53:33 INFO 06:53:33 INFO [loop_until]: kubectl --namespace=xlou top pods 06:53:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:53:33 INFO [loop_until]: OK (rc = 0) 06:53:33 DEBUG --- stdout --- 06:53:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 8990m 8508Mi am-5b5bd8567b-8h2w4 8998m 8815Mi am-5b5bd8567b-qlkw6 7613m 8458Mi ds-cts-0 7625m 12963Mi ds-cts-1 8592m 12799Mi ds-cts-2 8250m 13080Mi ds-idrepo-0 11452m 5503Mi ds-idrepo-1 343m 5479Mi ds-idrepo-2 358m 5784Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 3m 1011Mi idm-668d898cfd-st2t7 5m 1135Mi lodemon-ccc964465-p7gdm 2m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 3973m 1865Mi 06:53:33 DEBUG --- stderr --- 06:53:33 DEBUG 06:53:37 INFO 06:53:37 INFO [loop_until]: kubectl --namespace=xlou top node 06:53:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:53:37 INFO [loop_until]: OK (rc = 0) 06:53:37 DEBUG --- stdout --- 06:53:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 75m 0% 2728Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 65m 0% 2773Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7841m 26% 10110Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10810m 36% 2983Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9640m 32% 10099Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 145m 0% 3857Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9690m 32% 10442Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 407m 2% 6461Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11594m 72% 6201Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8539m 53% 13781Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 8126m 51% 13612Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 8396m 52% 13511Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 401m 2% 6158Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4230m 26% 5614Mi 9% 06:53:37 DEBUG --- stderr --- 06:53:37 DEBUG 06:54:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814861" 06:54:21 INFO [http_cmd]: http status code OK 06:54:21 DEBUG --- status code --- 06:54:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:21 DEBUG --- http response --- 06:54:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:54:21 DEBUG ----- output ----- 06:54:21 DEBUG { 06:54:21 DEBUG "status": "success", 06:54:21 DEBUG "data": { 06:54:21 DEBUG "resultType": "vector", 06:54:21 DEBUG "result": [ 06:54:21 DEBUG { 06:54:21 DEBUG "metric": { 06:54:21 DEBUG "pod": "ds-idrepo-2" 06:54:21 DEBUG }, 06:54:21 DEBUG "value": [ 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG [...] 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG "pod": "ds-idrepo-0" 06:54:21 DEBUG }, 06:54:21 DEBUG "value": [ 06:54:21 DEBUG 1712814861, 06:54:21 DEBUG "0" 06:54:21 DEBUG ] 06:54:21 DEBUG } 06:54:21 DEBUG ] 06:54:21 DEBUG } 06:54:21 DEBUG } 06:54:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712814861" 06:54:21 ERROR [http_cmd]: ERROR 06:54:21 ERROR --- status code --- 06:54:21 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:54:21 ERROR --- http response --- 06:54:21 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:54:21 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:54:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814861" 06:54:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814861" 06:54:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712814861" 06:54:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814861" 06:54:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814861" 06:54:21 INFO [http_cmd]: http status code OK 06:54:21 INFO [http_cmd]: http status code OK 06:54:21 INFO [http_cmd]: http status code OK 06:54:21 INFO [http_cmd]: http status code OK 06:54:21 DEBUG --- status code --- 06:54:21 DEBUG --- status code --- 06:54:21 DEBUG --- status code --- 06:54:21 DEBUG --- status code --- 06:54:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:21 DEBUG --- http response --- 06:54:21 DEBUG --- http response --- 06:54:21 DEBUG --- http response --- 06:54:21 DEBUG --- http response --- 06:54:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:54:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:54:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:54:21 DEBUG ----- output ----- 06:54:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:54:21 DEBUG ----- output ----- 06:54:21 DEBUG ----- output ----- 06:54:21 DEBUG { 06:54:21 DEBUG ----- output ----- 06:54:21 DEBUG { 06:54:21 DEBUG { 06:54:21 DEBUG "status": "success", 06:54:21 DEBUG { 06:54:21 DEBUG "status": "success", 06:54:21 DEBUG "status": "success", 06:54:21 DEBUG "data": { 06:54:21 DEBUG "status": "success", 06:54:21 DEBUG "data": { 06:54:21 DEBUG "data": { 06:54:21 DEBUG "resultType": "vector", 06:54:21 DEBUG "data": { 06:54:21 DEBUG "resultType": "vector", 06:54:21 DEBUG "resultType": "vector", 06:54:21 DEBUG "result": [ 06:54:21 DEBUG "resultType": "vector", 06:54:21 DEBUG "result": [ 06:54:21 DEBUG "result": [ 06:54:21 DEBUG { 06:54:21 DEBUG "result": [ 06:54:21 DEBUG { 06:54:21 DEBUG { 06:54:21 DEBUG "metric": { 06:54:21 DEBUG { 06:54:21 DEBUG "metric": { 06:54:21 DEBUG "metric": { 06:54:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:54:21 DEBUG "metric": { 06:54:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:54:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:54:21 DEBUG }, 06:54:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:54:21 DEBUG }, 06:54:21 DEBUG }, 06:54:21 DEBUG "value": [ 06:54:21 DEBUG }, 06:54:21 DEBUG "value": [ 06:54:21 DEBUG "value": [ 06:54:21 DEBUG 06:54:21 DEBUG "value": [ 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG [...] 06:54:21 DEBUG 06:54:21 DEBUG [...] 06:54:21 DEBUG [...] 06:54:21 DEBUG 06:54:21 DEBUG [...] 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:54:21 DEBUG 06:54:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:54:21 DEBUG "pod": "ds-idrepo-1" 06:54:21 DEBUG }, 06:54:21 DEBUG "pod": "ds-idrepo-1" 06:54:21 DEBUG }, 06:54:21 DEBUG }, 06:54:21 DEBUG "value": [ 06:54:21 DEBUG }, 06:54:21 DEBUG "value": [ 06:54:21 DEBUG "value": [ 06:54:21 DEBUG 1712814861, 06:54:21 DEBUG "value": [ 06:54:21 DEBUG 1712814861, 06:54:21 DEBUG 1712814861, 06:54:21 DEBUG "0.021" 06:54:21 DEBUG 1712814861, 06:54:21 DEBUG "245295" 06:54:21 DEBUG "0" 06:54:21 DEBUG ] 06:54:21 DEBUG "0" 06:54:21 DEBUG ] 06:54:21 DEBUG ] 06:54:21 DEBUG } 06:54:21 DEBUG ] 06:54:21 DEBUG } 06:54:21 DEBUG } 06:54:21 DEBUG ] 06:54:21 DEBUG } 06:54:21 DEBUG ] 06:54:21 DEBUG ] 06:54:21 DEBUG } 06:54:21 DEBUG ] 06:54:21 DEBUG } 06:54:21 DEBUG } 06:54:21 DEBUG } 06:54:21 DEBUG } 06:54:21 DEBUG } 06:54:21 DEBUG } 06:54:21 DEBUG } 06:54:21 INFO [http_cmd]: http status code OK 06:54:21 DEBUG --- status code --- 06:54:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:21 DEBUG --- http response --- 06:54:21 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:54:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814861" 06:54:21 INFO [http_cmd]: http status code OK 06:54:21 DEBUG --- status code --- 06:54:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:21 DEBUG --- http response --- 06:54:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:54:21 DEBUG ----- output ----- 06:54:21 DEBUG { 06:54:21 DEBUG "status": "success", 06:54:21 DEBUG "data": { 06:54:21 DEBUG "resultType": "vector", 06:54:21 DEBUG "result": [ 06:54:21 DEBUG { 06:54:21 DEBUG "metric": { 06:54:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:54:21 DEBUG }, 06:54:21 DEBUG "value": [ 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG [...] 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:54:21 DEBUG }, 06:54:21 DEBUG "value": [ 06:54:21 DEBUG 1712814861, 06:54:21 DEBUG "1347.7666666666667" 06:54:21 DEBUG ] 06:54:21 DEBUG } 06:54:21 DEBUG ] 06:54:21 DEBUG } 06:54:21 DEBUG } 06:54:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814861" 06:54:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814861" 06:54:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712814861" 06:54:21 INFO [http_cmd]: http status code OK 06:54:21 DEBUG --- status code --- 06:54:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:21 DEBUG --- http response --- 06:54:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:54:21 DEBUG ----- output ----- 06:54:21 DEBUG { 06:54:21 DEBUG "status": "success", 06:54:21 DEBUG "data": { 06:54:21 DEBUG "resultType": "vector", 06:54:21 DEBUG "result": [ 06:54:21 DEBUG { 06:54:21 DEBUG "metric": { 06:54:21 DEBUG "instance": "10.142.0.107:9100" 06:54:21 DEBUG }, 06:54:21 DEBUG "value": [ 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG [...] 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG "instance": "10.142.0.77:9100" 06:54:21 DEBUG }, 06:54:21 DEBUG "value": [ 06:54:21 DEBUG 1712814861, 06:54:21 DEBUG "82754244.4" 06:54:21 DEBUG ] 06:54:21 DEBUG } 06:54:21 DEBUG ] 06:54:21 DEBUG } 06:54:21 DEBUG } 06:54:21 INFO [http_cmd]: http status code OK 06:54:21 DEBUG --- status code --- 06:54:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:21 DEBUG --- http response --- 06:54:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:54:21 DEBUG ----- output ----- 06:54:21 DEBUG { 06:54:21 DEBUG "status": "success", 06:54:21 DEBUG "data": { 06:54:21 DEBUG "resultType": "vector", 06:54:21 DEBUG "result": [ 06:54:21 DEBUG { 06:54:21 DEBUG "metric": { 06:54:21 DEBUG "pod": "ds-idrepo-2" 06:54:21 DEBUG }, 06:54:21 DEBUG "value": [ 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG [...] 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG "pod": "ds-cts-2" 06:54:21 DEBUG }, 06:54:21 DEBUG "value": [ 06:54:21 DEBUG 1712814861, 06:54:21 DEBUG "0" 06:54:21 DEBUG ] 06:54:21 DEBUG } 06:54:21 DEBUG ] 06:54:21 DEBUG } 06:54:21 DEBUG } 06:54:21 INFO [http_cmd]: http status code OK 06:54:21 DEBUG --- status code --- 06:54:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:21 DEBUG --- http response --- 06:54:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:54:21 DEBUG ----- output ----- 06:54:21 DEBUG { 06:54:21 DEBUG "status": "success", 06:54:21 DEBUG "data": { 06:54:21 DEBUG "resultType": "vector", 06:54:21 DEBUG "result": [ 06:54:21 DEBUG { 06:54:21 DEBUG "metric": { 06:54:21 DEBUG "instance": "10.142.0.107:9100" 06:54:21 DEBUG }, 06:54:21 DEBUG "value": [ 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG [...] 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG "instance": "10.142.0.77:9100" 06:54:21 DEBUG }, 06:54:21 DEBUG "value": [ 06:54:21 DEBUG 1712814861, 06:54:21 DEBUG "0.002300000000000087" 06:54:21 DEBUG ] 06:54:21 DEBUG } 06:54:21 DEBUG ] 06:54:21 DEBUG } 06:54:21 DEBUG } 06:54:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814861" 06:54:21 INFO [http_cmd]: http status code OK 06:54:21 DEBUG --- status code --- 06:54:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:21 DEBUG --- http response --- 06:54:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:54:21 DEBUG ----- output ----- 06:54:21 DEBUG { 06:54:21 DEBUG "status": "success", 06:54:21 DEBUG "data": { 06:54:21 DEBUG "resultType": "vector", 06:54:21 DEBUG "result": [ 06:54:21 DEBUG { 06:54:21 DEBUG "metric": { 06:54:21 DEBUG "instance": "10.142.0.107:9100" 06:54:21 DEBUG }, 06:54:21 DEBUG "value": [ 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG [...] 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG 06:54:21 DEBUG "instance": "10.142.0.77:9100" 06:54:21 DEBUG }, 06:54:21 DEBUG "value": [ 06:54:21 DEBUG 1712814861, 06:54:21 DEBUG "52967656448" 06:54:21 DEBUG ] 06:54:21 DEBUG } 06:54:21 DEBUG ] 06:54:21 DEBUG } 06:54:21 DEBUG } 06:54:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814861" 06:54:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814861" 06:54:22 INFO [http_cmd]: http status code OK 06:54:22 DEBUG --- status code --- 06:54:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:22 DEBUG --- http response --- 06:54:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:54:22 DEBUG ----- output ----- 06:54:22 DEBUG { 06:54:22 DEBUG "status": "success", 06:54:22 DEBUG "data": { 06:54:22 DEBUG "resultType": "vector", 06:54:22 DEBUG "result": [ 06:54:22 DEBUG { 06:54:22 DEBUG "metric": { 06:54:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:54:22 DEBUG }, 06:54:22 DEBUG "value": [ 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG [...] 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:54:22 DEBUG }, 06:54:22 DEBUG "value": [ 06:54:22 DEBUG 1712814861, 06:54:22 DEBUG "1360.9" 06:54:22 DEBUG ] 06:54:22 DEBUG } 06:54:22 DEBUG ] 06:54:22 DEBUG } 06:54:22 DEBUG } 06:54:22 INFO [http_cmd]: http status code OK 06:54:22 DEBUG --- status code --- 06:54:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:22 DEBUG --- http response --- 06:54:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:54:22 DEBUG ----- output ----- 06:54:22 DEBUG { 06:54:22 DEBUG "status": "success", 06:54:22 DEBUG "data": { 06:54:22 DEBUG "resultType": "vector", 06:54:22 DEBUG "result": [ 06:54:22 DEBUG { 06:54:22 DEBUG "metric": { 06:54:22 DEBUG "instance": "10.142.0.107:9100" 06:54:22 DEBUG }, 06:54:22 DEBUG "value": [ 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG [...] 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG "instance": "10.142.0.77:9100" 06:54:22 DEBUG }, 06:54:22 DEBUG "value": [ 06:54:22 DEBUG 1712814861, 06:54:22 DEBUG "82756683.96666667" 06:54:22 DEBUG ] 06:54:22 DEBUG } 06:54:22 DEBUG ] 06:54:22 DEBUG } 06:54:22 DEBUG } 06:54:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814862" 06:54:22 INFO [http_cmd]: http status code OK 06:54:22 DEBUG --- status code --- 06:54:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:22 DEBUG --- http response --- 06:54:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:54:22 DEBUG ----- output ----- 06:54:22 DEBUG { 06:54:22 DEBUG "status": "success", 06:54:22 DEBUG "data": { 06:54:22 DEBUG "resultType": "vector", 06:54:22 DEBUG "result": [ 06:54:22 DEBUG { 06:54:22 DEBUG "metric": { 06:54:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:54:22 DEBUG }, 06:54:22 DEBUG "value": [ 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG [...] 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG "pod": "ds-cts-1" 06:54:22 DEBUG }, 06:54:22 DEBUG "value": [ 06:54:22 DEBUG 1712814862, 06:54:22 DEBUG "24010477.872998673" 06:54:22 DEBUG ] 06:54:22 DEBUG } 06:54:22 DEBUG ] 06:54:22 DEBUG } 06:54:22 DEBUG } 06:54:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814862" 06:54:22 INFO [http_cmd]: http status code OK 06:54:22 DEBUG --- status code --- 06:54:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:22 DEBUG --- http response --- 06:54:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:54:22 DEBUG ----- output ----- 06:54:22 DEBUG { 06:54:22 DEBUG "status": "success", 06:54:22 DEBUG "data": { 06:54:22 DEBUG "resultType": "vector", 06:54:22 DEBUG "result": [ 06:54:22 DEBUG { 06:54:22 DEBUG "metric": { 06:54:22 DEBUG "pod": "ds-idrepo-2" 06:54:22 DEBUG }, 06:54:22 DEBUG "value": [ 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG [...] 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG "pod": "ds-idrepo-0" 06:54:22 DEBUG }, 06:54:22 DEBUG "value": [ 06:54:22 DEBUG 1712814862, 06:54:22 DEBUG "0" 06:54:22 DEBUG ] 06:54:22 DEBUG } 06:54:22 DEBUG ] 06:54:22 DEBUG } 06:54:22 DEBUG } 06:54:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712814862" 06:54:22 INFO [http_cmd]: http status code OK 06:54:22 DEBUG --- status code --- 06:54:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:22 DEBUG --- http response --- 06:54:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:54:22 DEBUG ----- output ----- 06:54:22 DEBUG { 06:54:22 DEBUG "status": "success", 06:54:22 DEBUG "data": { 06:54:22 DEBUG "resultType": "vector", 06:54:22 DEBUG "result": [ 06:54:22 DEBUG { 06:54:22 DEBUG "metric": { 06:54:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:54:22 DEBUG }, 06:54:22 DEBUG "value": [ 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG [...] 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:54:22 DEBUG }, 06:54:22 DEBUG "value": [ 06:54:22 DEBUG 1712814862, 06:54:22 DEBUG "4.151305265342253" 06:54:22 DEBUG ] 06:54:22 DEBUG } 06:54:22 DEBUG ] 06:54:22 DEBUG } 06:54:22 DEBUG } 06:54:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814862" 06:54:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814862" 06:54:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814862" 06:54:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712814862" 06:54:22 INFO [http_cmd]: http status code OK 06:54:22 DEBUG --- status code --- 06:54:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:22 DEBUG --- http response --- 06:54:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:54:22 DEBUG ----- output ----- 06:54:22 DEBUG { 06:54:22 DEBUG "status": "success", 06:54:22 DEBUG "data": { 06:54:22 DEBUG "resultType": "vector", 06:54:22 DEBUG "result": [ 06:54:22 DEBUG { 06:54:22 DEBUG "metric": { 06:54:22 DEBUG "pod": "ds-idrepo-2" 06:54:22 DEBUG }, 06:54:22 DEBUG "value": [ 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG [...] 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG "pod": "ds-cts-2" 06:54:22 DEBUG }, 06:54:22 DEBUG "value": [ 06:54:22 DEBUG 1712814862, 06:54:22 DEBUG "29.933333333333334" 06:54:22 DEBUG ] 06:54:22 DEBUG } 06:54:22 DEBUG ] 06:54:22 DEBUG } 06:54:22 DEBUG } 06:54:22 INFO [http_cmd]: http status code OK 06:54:22 DEBUG --- status code --- 06:54:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:22 DEBUG --- http response --- 06:54:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:54:22 DEBUG ----- output ----- 06:54:22 DEBUG { 06:54:22 DEBUG "status": "success", 06:54:22 DEBUG "data": { 06:54:22 DEBUG "resultType": "vector", 06:54:22 DEBUG "result": [ 06:54:22 DEBUG { 06:54:22 DEBUG "metric": { 06:54:22 DEBUG "pod": "ds-idrepo-2" 06:54:22 DEBUG }, 06:54:22 DEBUG "value": [ 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG [...] 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG "pod": "ds-cts-2" 06:54:22 DEBUG }, 06:54:22 DEBUG "value": [ 06:54:22 DEBUG 1712814862, 06:54:22 DEBUG "0" 06:54:22 DEBUG ] 06:54:22 DEBUG } 06:54:22 DEBUG ] 06:54:22 DEBUG } 06:54:22 DEBUG } 06:54:22 INFO [http_cmd]: http status code OK 06:54:22 DEBUG --- status code --- 06:54:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:22 DEBUG --- http response --- 06:54:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:54:22 DEBUG ----- output ----- 06:54:22 DEBUG { 06:54:22 DEBUG "status": "success", 06:54:22 DEBUG "data": { 06:54:22 DEBUG "resultType": "vector", 06:54:22 DEBUG "result": [ 06:54:22 DEBUG { 06:54:22 DEBUG "metric": { 06:54:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:54:22 DEBUG }, 06:54:22 DEBUG "value": [ 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG [...] 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG "pod": "ds-cts-1" 06:54:22 DEBUG }, 06:54:22 DEBUG "value": [ 06:54:22 DEBUG 1712814862, 06:54:22 DEBUG "29530362.962918565" 06:54:22 DEBUG ] 06:54:22 DEBUG } 06:54:22 DEBUG ] 06:54:22 DEBUG } 06:54:22 DEBUG } 06:54:22 INFO [http_cmd]: http status code OK 06:54:22 DEBUG --- status code --- 06:54:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:22 DEBUG --- http response --- 06:54:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:54:22 DEBUG ----- output ----- 06:54:22 DEBUG { 06:54:22 DEBUG "status": "success", 06:54:22 DEBUG "data": { 06:54:22 DEBUG "resultType": "vector", 06:54:22 DEBUG "result": [ 06:54:22 DEBUG { 06:54:22 DEBUG "metric": { 06:54:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:54:22 DEBUG }, 06:54:22 DEBUG "value": [ 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG [...] 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 06:54:22 DEBUG }, 06:54:22 DEBUG "value": [ 06:54:22 DEBUG 1712814862, 06:54:22 DEBUG "14136741888" 06:54:22 DEBUG ] 06:54:22 DEBUG } 06:54:22 DEBUG ] 06:54:22 DEBUG } 06:54:22 DEBUG } 06:54:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814862" 06:54:22 INFO [http_cmd]: http status code OK 06:54:22 DEBUG --- status code --- 06:54:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:22 DEBUG --- http response --- 06:54:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:54:22 DEBUG ----- output ----- 06:54:22 DEBUG { 06:54:22 DEBUG "status": "success", 06:54:22 DEBUG "data": { 06:54:22 DEBUG "resultType": "vector", 06:54:22 DEBUG "result": [ 06:54:22 DEBUG { 06:54:22 DEBUG "metric": { 06:54:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:54:22 DEBUG }, 06:54:22 DEBUG "value": [ 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG [...] 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:54:22 DEBUG }, 06:54:22 DEBUG "value": [ 06:54:22 DEBUG 1712814862, 06:54:22 DEBUG "1360.9" 06:54:22 DEBUG ] 06:54:22 DEBUG } 06:54:22 DEBUG ] 06:54:22 DEBUG } 06:54:22 DEBUG } 06:54:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814862" 06:54:22 INFO [http_cmd]: http status code OK 06:54:22 DEBUG --- status code --- 06:54:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:22 DEBUG --- http response --- 06:54:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:54:22 DEBUG ----- output ----- 06:54:22 DEBUG { 06:54:22 DEBUG "status": "success", 06:54:22 DEBUG "data": { 06:54:22 DEBUG "resultType": "vector", 06:54:22 DEBUG "result": [ 06:54:22 DEBUG { 06:54:22 DEBUG "metric": { 06:54:22 DEBUG "pod": "ds-idrepo-2" 06:54:22 DEBUG }, 06:54:22 DEBUG "value": [ 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG [...] 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG "pod": "ds-cts-2" 06:54:22 DEBUG }, 06:54:22 DEBUG "value": [ 06:54:22 DEBUG 1712814862, 06:54:22 DEBUG "0.0006614716666666666" 06:54:22 DEBUG ] 06:54:22 DEBUG } 06:54:22 DEBUG ] 06:54:22 DEBUG } 06:54:22 DEBUG } 06:54:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814862" 06:54:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814862" 06:54:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814862" 06:54:22 INFO [http_cmd]: http status code OK 06:54:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814862" 06:54:22 DEBUG --- status code --- 06:54:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:22 DEBUG --- http response --- 06:54:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:54:22 DEBUG ----- output ----- 06:54:22 DEBUG { 06:54:22 DEBUG "status": "success", 06:54:22 DEBUG "data": { 06:54:22 DEBUG "resultType": "vector", 06:54:22 DEBUG "result": [ 06:54:22 DEBUG { 06:54:22 DEBUG "metric": { 06:54:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:54:22 DEBUG }, 06:54:22 DEBUG "value": [ 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG [...] 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:54:22 DEBUG }, 06:54:22 DEBUG "value": [ 06:54:22 DEBUG 1712814862, 06:54:22 DEBUG "0" 06:54:22 DEBUG ] 06:54:22 DEBUG } 06:54:22 DEBUG ] 06:54:22 DEBUG } 06:54:22 DEBUG } 06:54:22 INFO [http_cmd]: http status code OK 06:54:22 DEBUG --- status code --- 06:54:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:22 DEBUG --- http response --- 06:54:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:54:22 DEBUG ----- output ----- 06:54:22 DEBUG { 06:54:22 DEBUG "status": "success", 06:54:22 DEBUG "data": { 06:54:22 DEBUG "resultType": "vector", 06:54:22 DEBUG "result": [ 06:54:22 DEBUG { 06:54:22 DEBUG "metric": { 06:54:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:54:22 DEBUG }, 06:54:22 DEBUG "value": [ 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG [...] 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:54:22 DEBUG }, 06:54:22 DEBUG "value": [ 06:54:22 DEBUG 1712814862, 06:54:22 DEBUG "1360.7333333333333" 06:54:22 DEBUG ] 06:54:22 DEBUG } 06:54:22 DEBUG ] 06:54:22 DEBUG } 06:54:22 DEBUG } 06:54:22 INFO [http_cmd]: http status code OK 06:54:22 DEBUG --- status code --- 06:54:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:22 DEBUG --- http response --- 06:54:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:54:22 DEBUG ----- output ----- 06:54:22 DEBUG { 06:54:22 DEBUG "status": "success", 06:54:22 DEBUG "data": { 06:54:22 DEBUG "resultType": "vector", 06:54:22 DEBUG "result": [ 06:54:22 DEBUG { 06:54:22 DEBUG "metric": { 06:54:22 DEBUG "pod": "ds-idrepo-2" 06:54:22 DEBUG }, 06:54:22 DEBUG "value": [ 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG [...] 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG "pod": "ds-idrepo-0" 06:54:22 DEBUG }, 06:54:22 DEBUG "value": [ 06:54:22 DEBUG 1712814862, 06:54:22 DEBUG "0" 06:54:22 DEBUG ] 06:54:22 DEBUG } 06:54:22 DEBUG ] 06:54:22 DEBUG } 06:54:22 DEBUG } 06:54:22 INFO [http_cmd]: http status code OK 06:54:22 DEBUG --- status code --- 06:54:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:54:22 DEBUG --- http response --- 06:54:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:54:22 DEBUG ----- output ----- 06:54:22 DEBUG { 06:54:22 DEBUG "status": "success", 06:54:22 DEBUG "data": { 06:54:22 DEBUG "resultType": "vector", 06:54:22 DEBUG "result": [ 06:54:22 DEBUG { 06:54:22 DEBUG "metric": { 06:54:22 DEBUG "instance": "10.142.0.107:9100" 06:54:22 DEBUG }, 06:54:22 DEBUG "value": [ 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG [...] 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG 06:54:22 DEBUG "instance": "10.142.0.77:9100" 06:54:22 DEBUG }, 06:54:22 DEBUG "value": [ 06:54:22 DEBUG 1712814862, 06:54:22 DEBUG "0" 06:54:22 DEBUG ] 06:54:22 DEBUG } 06:54:22 DEBUG ] 06:54:22 DEBUG } 06:54:22 DEBUG } 06:54:33 INFO 06:54:33 INFO [loop_until]: kubectl --namespace=xlou top pods 06:54:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:54:33 INFO [loop_until]: OK (rc = 0) 06:54:33 DEBUG --- stdout --- 06:54:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 8790m 8505Mi am-5b5bd8567b-8h2w4 8484m 8821Mi am-5b5bd8567b-qlkw6 7618m 8461Mi ds-cts-0 8288m 13395Mi ds-cts-1 8560m 13399Mi ds-cts-2 8372m 13398Mi ds-idrepo-0 11540m 5504Mi ds-idrepo-1 404m 5479Mi ds-idrepo-2 386m 5784Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 4m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 4m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4131m 1864Mi 06:54:33 DEBUG --- stderr --- 06:54:33 DEBUG 06:54:37 INFO 06:54:37 INFO [loop_until]: kubectl --namespace=xlou top node 06:54:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:54:37 INFO [loop_until]: OK (rc = 0) 06:54:37 DEBUG --- stdout --- 06:54:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 2729Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 62m 0% 2783Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7822m 26% 10116Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10740m 35% 2981Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9553m 31% 10099Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 144m 0% 3839Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9596m 32% 10442Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 463m 2% 6465Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11550m 72% 6202Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8354m 52% 13981Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 8040m 50% 14007Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 8653m 54% 14034Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 454m 2% 6159Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4303m 27% 5617Mi 9% 06:54:37 DEBUG --- stderr --- 06:54:37 DEBUG 06:55:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814921" 06:55:21 INFO [http_cmd]: http status code OK 06:55:21 DEBUG --- status code --- 06:55:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:21 DEBUG --- http response --- 06:55:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:55:21 DEBUG ----- output ----- 06:55:21 DEBUG { 06:55:21 DEBUG "status": "success", 06:55:21 DEBUG "data": { 06:55:21 DEBUG "resultType": "vector", 06:55:21 DEBUG "result": [ 06:55:21 DEBUG { 06:55:21 DEBUG "metric": { 06:55:21 DEBUG "pod": "ds-idrepo-2" 06:55:21 DEBUG }, 06:55:21 DEBUG "value": [ 06:55:21 DEBUG 06:55:21 DEBUG 06:55:21 DEBUG 06:55:21 DEBUG [...] 06:55:21 DEBUG 06:55:21 DEBUG 06:55:21 DEBUG 06:55:21 DEBUG "pod": "ds-idrepo-0" 06:55:21 DEBUG }, 06:55:21 DEBUG "value": [ 06:55:21 DEBUG 1712814921, 06:55:21 DEBUG "0" 06:55:21 DEBUG ] 06:55:21 DEBUG } 06:55:21 DEBUG ] 06:55:21 DEBUG } 06:55:21 DEBUG } 06:55:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712814921" 06:55:21 ERROR [http_cmd]: ERROR 06:55:21 ERROR --- status code --- 06:55:21 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:55:21 ERROR --- http response --- 06:55:21 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:55:21 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:55:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814921" 06:55:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814921" 06:55:21 INFO [http_cmd]: http status code OK 06:55:21 DEBUG --- status code --- 06:55:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:21 DEBUG --- http response --- 06:55:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:55:21 DEBUG ----- output ----- 06:55:21 DEBUG { 06:55:21 DEBUG "status": "success", 06:55:21 DEBUG "data": { 06:55:21 DEBUG "resultType": "vector", 06:55:21 DEBUG "result": [ 06:55:21 DEBUG { 06:55:21 DEBUG "metric": { 06:55:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:55:21 DEBUG }, 06:55:21 DEBUG "value": [ 06:55:21 DEBUG 06:55:21 DEBUG 06:55:21 DEBUG 06:55:21 DEBUG [...] 06:55:21 DEBUG 06:55:21 DEBUG 06:55:21 DEBUG 06:55:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:55:21 DEBUG }, 06:55:21 DEBUG "value": [ 06:55:21 DEBUG 1712814921, 06:55:21 DEBUG "245640" 06:55:21 DEBUG ] 06:55:21 DEBUG } 06:55:21 DEBUG ] 06:55:21 DEBUG } 06:55:21 DEBUG } 06:55:21 INFO [http_cmd]: http status code OK 06:55:21 DEBUG --- status code --- 06:55:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:21 DEBUG --- http response --- 06:55:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:55:21 DEBUG ----- output ----- 06:55:21 DEBUG { 06:55:21 DEBUG "status": "success", 06:55:21 DEBUG "data": { 06:55:21 DEBUG "resultType": "vector", 06:55:21 DEBUG "result": [ 06:55:21 DEBUG { 06:55:21 DEBUG "metric": { 06:55:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:55:21 DEBUG }, 06:55:21 DEBUG "value": [ 06:55:21 DEBUG 06:55:21 DEBUG 06:55:21 DEBUG 06:55:21 DEBUG [...] 06:55:21 DEBUG 06:55:21 DEBUG 06:55:21 DEBUG 06:55:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:55:21 DEBUG }, 06:55:21 DEBUG "value": [ 06:55:21 DEBUG 1712814921, 06:55:21 DEBUG "0" 06:55:21 DEBUG ] 06:55:21 DEBUG } 06:55:21 DEBUG ] 06:55:21 DEBUG } 06:55:21 DEBUG } 06:55:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814921" 06:55:21 INFO [http_cmd]: http status code OK 06:55:21 DEBUG --- status code --- 06:55:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:21 DEBUG --- http response --- 06:55:21 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:55:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712814921" 06:55:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814921" 06:55:21 INFO [http_cmd]: http status code OK 06:55:21 DEBUG --- status code --- 06:55:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:21 DEBUG --- http response --- 06:55:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:55:21 DEBUG ----- output ----- 06:55:21 DEBUG { 06:55:21 DEBUG "status": "success", 06:55:21 DEBUG "data": { 06:55:21 DEBUG "resultType": "vector", 06:55:21 DEBUG "result": [ 06:55:21 DEBUG { 06:55:21 DEBUG "metric": { 06:55:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:55:21 DEBUG }, 06:55:21 DEBUG "value": [ 06:55:21 DEBUG 06:55:21 DEBUG 06:55:21 DEBUG 06:55:21 DEBUG [...] 06:55:21 DEBUG 06:55:21 DEBUG 06:55:21 DEBUG 06:55:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:55:21 DEBUG }, 06:55:21 DEBUG "value": [ 06:55:21 DEBUG 1712814921, 06:55:21 DEBUG "0" 06:55:21 DEBUG ] 06:55:21 DEBUG } 06:55:21 DEBUG ] 06:55:21 DEBUG } 06:55:21 DEBUG } 06:55:21 INFO [http_cmd]: http status code OK 06:55:21 DEBUG --- status code --- 06:55:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:21 DEBUG --- http response --- 06:55:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814921" 06:55:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:55:21 DEBUG ----- output ----- 06:55:21 DEBUG { 06:55:21 DEBUG "status": "success", 06:55:21 DEBUG "data": { 06:55:21 DEBUG "resultType": "vector", 06:55:21 DEBUG "result": [ 06:55:21 DEBUG { 06:55:21 DEBUG "metric": { 06:55:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:55:21 DEBUG }, 06:55:21 DEBUG "value": [ 06:55:21 DEBUG 06:55:21 DEBUG 06:55:21 DEBUG 06:55:21 DEBUG [...] 06:55:21 DEBUG 06:55:21 DEBUG 06:55:21 DEBUG 06:55:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:55:21 DEBUG }, 06:55:21 DEBUG "value": [ 06:55:21 DEBUG 1712814921, 06:55:21 DEBUG "0.021" 06:55:21 DEBUG ] 06:55:21 DEBUG } 06:55:21 DEBUG ] 06:55:21 DEBUG } 06:55:21 DEBUG } 06:55:21 INFO [http_cmd]: http status code OK 06:55:21 DEBUG --- status code --- 06:55:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:21 DEBUG --- http response --- 06:55:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:55:21 DEBUG ----- output ----- 06:55:21 DEBUG { 06:55:21 DEBUG "status": "success", 06:55:21 DEBUG "data": { 06:55:21 DEBUG "resultType": "vector", 06:55:21 DEBUG "result": [ 06:55:21 DEBUG { 06:55:21 DEBUG "metric": { 06:55:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:55:21 DEBUG }, 06:55:21 DEBUG "value": [ 06:55:21 DEBUG 06:55:21 DEBUG 06:55:21 DEBUG 06:55:21 DEBUG [...] 06:55:21 DEBUG 06:55:21 DEBUG 06:55:21 DEBUG 06:55:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:55:21 DEBUG }, 06:55:21 DEBUG "value": [ 06:55:21 DEBUG 1712814921, 06:55:21 DEBUG "1361.8333333333333" 06:55:21 DEBUG ] 06:55:21 DEBUG } 06:55:21 DEBUG ] 06:55:21 DEBUG } 06:55:21 DEBUG } 06:55:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814922" 06:55:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712814922" 06:55:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814922" 06:55:22 INFO [http_cmd]: http status code OK 06:55:22 DEBUG --- status code --- 06:55:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:22 DEBUG --- http response --- 06:55:22 INFO [http_cmd]: http status code OK 06:55:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814922" 06:55:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:55:22 DEBUG --- status code --- 06:55:22 DEBUG ----- output ----- 06:55:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:22 DEBUG { 06:55:22 DEBUG --- http response --- 06:55:22 DEBUG "status": "success", 06:55:22 DEBUG "data": { 06:55:22 DEBUG "resultType": "vector", 06:55:22 DEBUG "result": [ 06:55:22 DEBUG { 06:55:22 DEBUG "metric": { 06:55:22 DEBUG "pod": "ds-idrepo-2" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG [...] 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG "pod": "ds-cts-2" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 1712814922, 06:55:22 DEBUG "0" 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG } 06:55:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:55:22 DEBUG ----- output ----- 06:55:22 DEBUG { 06:55:22 DEBUG "status": "success", 06:55:22 DEBUG "data": { 06:55:22 DEBUG "resultType": "vector", 06:55:22 DEBUG "result": [ 06:55:22 DEBUG { 06:55:22 DEBUG "metric": { 06:55:22 DEBUG "instance": "10.142.0.107:9100" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG [...] 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG "instance": "10.142.0.77:9100" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 1712814922, 06:55:22 DEBUG "82945298.43333334" 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG } 06:55:22 INFO [http_cmd]: http status code OK 06:55:22 DEBUG --- status code --- 06:55:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:22 DEBUG --- http response --- 06:55:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:55:22 DEBUG ----- output ----- 06:55:22 DEBUG { 06:55:22 DEBUG "status": "success", 06:55:22 DEBUG "data": { 06:55:22 DEBUG "resultType": "vector", 06:55:22 DEBUG "result": [ 06:55:22 DEBUG { 06:55:22 DEBUG "metric": { 06:55:22 DEBUG "instance": "10.142.0.107:9100" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG [...] 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG "instance": "10.142.0.77:9100" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 1712814922, 06:55:22 DEBUG "0.0021666666666670647" 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG } 06:55:22 INFO [http_cmd]: http status code OK 06:55:22 DEBUG --- status code --- 06:55:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:22 DEBUG --- http response --- 06:55:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:55:22 DEBUG ----- output ----- 06:55:22 DEBUG { 06:55:22 DEBUG "status": "success", 06:55:22 DEBUG "data": { 06:55:22 DEBUG "resultType": "vector", 06:55:22 DEBUG "result": [ 06:55:22 DEBUG { 06:55:22 DEBUG "metric": { 06:55:22 DEBUG "instance": "10.142.0.107:9100" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG [...] 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG "instance": "10.142.0.77:9100" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 1712814922, 06:55:22 DEBUG "82947938.76666667" 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG } 06:55:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814922" 06:55:22 INFO [http_cmd]: http status code OK 06:55:22 DEBUG --- status code --- 06:55:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:22 DEBUG --- http response --- 06:55:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:55:22 DEBUG ----- output ----- 06:55:22 DEBUG { 06:55:22 DEBUG "status": "success", 06:55:22 DEBUG "data": { 06:55:22 DEBUG "resultType": "vector", 06:55:22 DEBUG "result": [ 06:55:22 DEBUG { 06:55:22 DEBUG "metric": { 06:55:22 DEBUG "instance": "10.142.0.107:9100" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG [...] 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG "instance": "10.142.0.77:9100" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 1712814922, 06:55:22 DEBUG "52969012224" 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG } 06:55:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814922" 06:55:22 INFO [http_cmd]: http status code OK 06:55:22 DEBUG --- status code --- 06:55:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:22 DEBUG --- http response --- 06:55:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:55:22 DEBUG ----- output ----- 06:55:22 DEBUG { 06:55:22 DEBUG "status": "success", 06:55:22 DEBUG "data": { 06:55:22 DEBUG "resultType": "vector", 06:55:22 DEBUG "result": [ 06:55:22 DEBUG { 06:55:22 DEBUG "metric": { 06:55:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG [...] 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 1712814922, 06:55:22 DEBUG "1366.9333333333334" 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG } 06:55:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814922" 06:55:22 INFO [http_cmd]: http status code OK 06:55:22 DEBUG --- status code --- 06:55:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:22 DEBUG --- http response --- 06:55:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 06:55:22 DEBUG ----- output ----- 06:55:22 DEBUG { 06:55:22 DEBUG "status": "success", 06:55:22 DEBUG "data": { 06:55:22 DEBUG "resultType": "vector", 06:55:22 DEBUG "result": [ 06:55:22 DEBUG { 06:55:22 DEBUG "metric": { 06:55:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG [...] 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 1712814922, 06:55:22 DEBUG "0" 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG } 06:55:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814922" 06:55:22 INFO [http_cmd]: http status code OK 06:55:22 DEBUG --- status code --- 06:55:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:22 DEBUG --- http response --- 06:55:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:55:22 DEBUG ----- output ----- 06:55:22 DEBUG { 06:55:22 DEBUG "status": "success", 06:55:22 DEBUG "data": { 06:55:22 DEBUG "resultType": "vector", 06:55:22 DEBUG "result": [ 06:55:22 DEBUG { 06:55:22 DEBUG "metric": { 06:55:22 DEBUG "pod": "ds-idrepo-2" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG [...] 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG "pod": "ds-idrepo-0" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 1712814922, 06:55:22 DEBUG "0" 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG } 06:55:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712814922" 06:55:22 INFO [http_cmd]: http status code OK 06:55:22 DEBUG --- status code --- 06:55:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:22 DEBUG --- http response --- 06:55:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:55:22 DEBUG ----- output ----- 06:55:22 DEBUG { 06:55:22 DEBUG "status": "success", 06:55:22 DEBUG "data": { 06:55:22 DEBUG "resultType": "vector", 06:55:22 DEBUG "result": [ 06:55:22 DEBUG { 06:55:22 DEBUG "metric": { 06:55:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG [...] 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 1712814922, 06:55:22 DEBUG "4.199705644558526" 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG } 06:55:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712814922" 06:55:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814922" 06:55:22 INFO [http_cmd]: http status code OK 06:55:22 DEBUG --- status code --- 06:55:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:22 DEBUG --- http response --- 06:55:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:55:22 DEBUG ----- output ----- 06:55:22 DEBUG { 06:55:22 DEBUG "status": "success", 06:55:22 DEBUG "data": { 06:55:22 DEBUG "resultType": "vector", 06:55:22 DEBUG "result": [ 06:55:22 DEBUG { 06:55:22 DEBUG "metric": { 06:55:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG [...] 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 1712814922, 06:55:22 DEBUG "14135705600" 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG } 06:55:22 INFO [http_cmd]: http status code OK 06:55:22 DEBUG --- status code --- 06:55:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:22 DEBUG --- http response --- 06:55:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:55:22 DEBUG ----- output ----- 06:55:22 DEBUG { 06:55:22 DEBUG "status": "success", 06:55:22 DEBUG "data": { 06:55:22 DEBUG "resultType": "vector", 06:55:22 DEBUG "result": [ 06:55:22 DEBUG { 06:55:22 DEBUG "metric": { 06:55:22 DEBUG "pod": "ds-idrepo-2" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG [...] 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG "pod": "ds-cts-2" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 1712814922, 06:55:22 DEBUG "29.933333333333334" 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG } 06:55:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814922" 06:55:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814922" 06:55:22 INFO [http_cmd]: http status code OK 06:55:22 DEBUG --- status code --- 06:55:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:22 DEBUG --- http response --- 06:55:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:55:22 DEBUG ----- output ----- 06:55:22 DEBUG { 06:55:22 DEBUG "status": "success", 06:55:22 DEBUG "data": { 06:55:22 DEBUG "resultType": "vector", 06:55:22 DEBUG "result": [ 06:55:22 DEBUG { 06:55:22 DEBUG "metric": { 06:55:22 DEBUG "pod": "ds-idrepo-2" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG [...] 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG "pod": "ds-cts-2" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 1712814922, 06:55:22 DEBUG "0" 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG } 06:55:22 INFO [http_cmd]: http status code OK 06:55:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814922" 06:55:22 DEBUG --- status code --- 06:55:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:22 DEBUG --- http response --- 06:55:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 06:55:22 DEBUG ----- output ----- 06:55:22 DEBUG { 06:55:22 DEBUG "status": "success", 06:55:22 DEBUG "data": { 06:55:22 DEBUG "resultType": "vector", 06:55:22 DEBUG "result": [ 06:55:22 DEBUG { 06:55:22 DEBUG "metric": { 06:55:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG [...] 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 1712814922, 06:55:22 DEBUG "0" 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG } 06:55:22 INFO [http_cmd]: http status code OK 06:55:22 DEBUG --- status code --- 06:55:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:22 DEBUG --- http response --- 06:55:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:55:22 DEBUG ----- output ----- 06:55:22 DEBUG { 06:55:22 DEBUG "status": "success", 06:55:22 DEBUG "data": { 06:55:22 DEBUG "resultType": "vector", 06:55:22 DEBUG "result": [ 06:55:22 DEBUG { 06:55:22 DEBUG "metric": { 06:55:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG [...] 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 1712814922, 06:55:22 DEBUG "1366.9" 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG } 06:55:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814922" 06:55:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814922" 06:55:22 INFO [http_cmd]: http status code OK 06:55:22 DEBUG --- status code --- 06:55:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:22 DEBUG --- http response --- 06:55:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:55:22 DEBUG ----- output ----- 06:55:22 DEBUG { 06:55:22 DEBUG "status": "success", 06:55:22 DEBUG "data": { 06:55:22 DEBUG "resultType": "vector", 06:55:22 DEBUG "result": [ 06:55:22 DEBUG { 06:55:22 DEBUG "metric": { 06:55:22 DEBUG "pod": "ds-idrepo-2" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG [...] 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG "pod": "ds-cts-2" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 1712814922, 06:55:22 DEBUG "0" 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG } 06:55:22 INFO [http_cmd]: http status code OK 06:55:22 DEBUG --- status code --- 06:55:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:22 DEBUG --- http response --- 06:55:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:55:22 DEBUG ----- output ----- 06:55:22 DEBUG { 06:55:22 DEBUG "status": "success", 06:55:22 DEBUG "data": { 06:55:22 DEBUG "resultType": "vector", 06:55:22 DEBUG "result": [ 06:55:22 DEBUG { 06:55:22 DEBUG "metric": { 06:55:22 DEBUG "pod": "ds-idrepo-2" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG [...] 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG "pod": "ds-idrepo-0" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 1712814922, 06:55:22 DEBUG "0" 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG } 06:55:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814922" 06:55:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814922" 06:55:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814922" 06:55:22 INFO [http_cmd]: http status code OK 06:55:22 INFO [http_cmd]: http status code OK 06:55:22 DEBUG --- status code --- 06:55:22 DEBUG --- status code --- 06:55:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:22 DEBUG --- http response --- 06:55:22 DEBUG --- http response --- 06:55:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:55:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:55:22 DEBUG ----- output ----- 06:55:22 DEBUG ----- output ----- 06:55:22 DEBUG { 06:55:22 DEBUG { 06:55:22 DEBUG "status": "success", 06:55:22 DEBUG "status": "success", 06:55:22 DEBUG "data": { 06:55:22 DEBUG "data": { 06:55:22 DEBUG "resultType": "vector", 06:55:22 DEBUG "resultType": "vector", 06:55:22 DEBUG "result": [ 06:55:22 DEBUG "result": [ 06:55:22 DEBUG { 06:55:22 DEBUG { 06:55:22 DEBUG "metric": { 06:55:22 DEBUG "metric": { 06:55:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:55:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:55:22 DEBUG }, 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG [...] 06:55:22 DEBUG [...] 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:55:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:55:22 DEBUG }, 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 1712814922, 06:55:22 DEBUG 1712814922, 06:55:22 DEBUG "0" 06:55:22 DEBUG "1365.5" 06:55:22 DEBUG ] 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG } 06:55:22 DEBUG ] 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG } 06:55:22 DEBUG } 06:55:22 DEBUG } 06:55:22 INFO [http_cmd]: http status code OK 06:55:22 DEBUG --- status code --- 06:55:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:55:22 DEBUG --- http response --- 06:55:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:55:22 DEBUG ----- output ----- 06:55:22 DEBUG { 06:55:22 DEBUG "status": "success", 06:55:22 DEBUG "data": { 06:55:22 DEBUG "resultType": "vector", 06:55:22 DEBUG "result": [ 06:55:22 DEBUG { 06:55:22 DEBUG "metric": { 06:55:22 DEBUG "instance": "10.142.0.107:9100" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG [...] 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG 06:55:22 DEBUG "instance": "10.142.0.77:9100" 06:55:22 DEBUG }, 06:55:22 DEBUG "value": [ 06:55:22 DEBUG 1712814922, 06:55:22 DEBUG "0.0003333333333333854" 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG ] 06:55:22 DEBUG } 06:55:22 DEBUG } 06:55:33 INFO 06:55:33 INFO [loop_until]: kubectl --namespace=xlou top pods 06:55:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:55:33 INFO [loop_until]: OK (rc = 0) 06:55:33 DEBUG --- stdout --- 06:55:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 9354m 8510Mi am-5b5bd8567b-8h2w4 8549m 8818Mi am-5b5bd8567b-qlkw6 7595m 8457Mi ds-cts-0 7858m 13393Mi ds-cts-1 8420m 13091Mi ds-cts-2 8586m 12574Mi ds-idrepo-0 11455m 5503Mi ds-idrepo-1 357m 5480Mi ds-idrepo-2 366m 5784Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 3m 1010Mi idm-668d898cfd-st2t7 4m 1136Mi lodemon-ccc964465-p7gdm 2m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4169m 1867Mi 06:55:33 DEBUG --- stderr --- 06:55:33 DEBUG 06:55:37 INFO 06:55:37 INFO [loop_until]: kubectl --namespace=xlou top node 06:55:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:55:37 INFO [loop_until]: OK (rc = 0) 06:55:37 DEBUG --- stdout --- 06:55:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 75m 0% 2725Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 63m 0% 2791Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7921m 26% 10116Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10849m 36% 2972Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9678m 32% 10095Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 145m 0% 3837Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9497m 31% 10435Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 419m 2% 6479Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11257m 70% 6203Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8422m 53% 13165Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 8160m 51% 13101Mi 22% gke-xlou-cdm-ds-32e4dcb1-j2hk 8641m 54% 13692Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 416m 2% 6159Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4322m 27% 5617Mi 9% 06:55:37 DEBUG --- stderr --- 06:55:37 DEBUG 06:56:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814981" 06:56:21 INFO [http_cmd]: http status code OK 06:56:21 DEBUG --- status code --- 06:56:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:21 DEBUG --- http response --- 06:56:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:56:21 DEBUG ----- output ----- 06:56:21 DEBUG { 06:56:21 DEBUG "status": "success", 06:56:21 DEBUG "data": { 06:56:21 DEBUG "resultType": "vector", 06:56:21 DEBUG "result": [ 06:56:21 DEBUG { 06:56:21 DEBUG "metric": { 06:56:21 DEBUG "pod": "ds-idrepo-2" 06:56:21 DEBUG }, 06:56:21 DEBUG "value": [ 06:56:21 DEBUG 06:56:21 DEBUG 06:56:21 DEBUG 06:56:21 DEBUG [...] 06:56:21 DEBUG 06:56:21 DEBUG 06:56:21 DEBUG 06:56:21 DEBUG "pod": "ds-idrepo-0" 06:56:21 DEBUG }, 06:56:21 DEBUG "value": [ 06:56:21 DEBUG 1712814981, 06:56:21 DEBUG "0" 06:56:21 DEBUG ] 06:56:21 DEBUG } 06:56:21 DEBUG ] 06:56:21 DEBUG } 06:56:21 DEBUG } 06:56:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712814981" 06:56:21 ERROR [http_cmd]: ERROR 06:56:21 ERROR --- status code --- 06:56:21 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:56:21 ERROR --- http response --- 06:56:21 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:56:21 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:56:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814981" 06:56:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814981" 06:56:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814981" 06:56:21 INFO [http_cmd]: http status code OK 06:56:21 DEBUG --- status code --- 06:56:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:21 DEBUG --- http response --- 06:56:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:56:21 DEBUG ----- output ----- 06:56:21 DEBUG { 06:56:21 DEBUG "status": "success", 06:56:21 DEBUG "data": { 06:56:21 DEBUG "resultType": "vector", 06:56:21 DEBUG "result": [ 06:56:21 DEBUG { 06:56:21 DEBUG "metric": { 06:56:21 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:56:21 DEBUG }, 06:56:21 DEBUG "value": [ 06:56:21 DEBUG 06:56:21 DEBUG 06:56:21 DEBUG 06:56:21 DEBUG [...] 06:56:21 DEBUG 06:56:21 DEBUG 06:56:21 DEBUG 06:56:21 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:56:21 DEBUG }, 06:56:21 DEBUG "value": [ 06:56:21 DEBUG 1712814981, 06:56:21 DEBUG "245449.5" 06:56:21 DEBUG ] 06:56:21 DEBUG } 06:56:21 DEBUG ] 06:56:21 DEBUG } 06:56:21 DEBUG } 06:56:21 INFO [http_cmd]: http status code OK 06:56:21 DEBUG --- status code --- 06:56:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:21 DEBUG --- http response --- 06:56:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:56:21 DEBUG ----- output ----- 06:56:21 DEBUG { 06:56:21 DEBUG "status": "success", 06:56:21 DEBUG "data": { 06:56:21 DEBUG "resultType": "vector", 06:56:21 DEBUG "result": [ 06:56:21 DEBUG { 06:56:21 DEBUG "metric": { 06:56:21 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:56:21 DEBUG }, 06:56:21 DEBUG "value": [ 06:56:21 DEBUG 06:56:21 DEBUG 06:56:21 DEBUG 06:56:21 DEBUG [...] 06:56:21 DEBUG 06:56:21 DEBUG 06:56:21 DEBUG 06:56:21 DEBUG "pod": "ds-cts-0" 06:56:21 DEBUG }, 06:56:21 DEBUG "value": [ 06:56:21 DEBUG 1712814981, 06:56:21 DEBUG "8.014206761768854" 06:56:21 DEBUG ] 06:56:21 DEBUG } 06:56:21 DEBUG ] 06:56:21 DEBUG } 06:56:21 DEBUG } 06:56:21 INFO [http_cmd]: http status code OK 06:56:21 DEBUG --- status code --- 06:56:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:21 DEBUG --- http response --- 06:56:21 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:56:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814981" 06:56:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814981" 06:56:22 INFO [http_cmd]: http status code OK 06:56:22 DEBUG --- status code --- 06:56:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:22 DEBUG --- http response --- 06:56:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:56:22 DEBUG ----- output ----- 06:56:22 DEBUG { 06:56:22 DEBUG "status": "success", 06:56:22 DEBUG "data": { 06:56:22 DEBUG "resultType": "vector", 06:56:22 DEBUG "result": [ 06:56:22 DEBUG { 06:56:22 DEBUG "metric": { 06:56:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG [...] 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 1712814981, 06:56:22 DEBUG "1369.0333333333333" 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG } 06:56:22 INFO [http_cmd]: http status code OK 06:56:22 DEBUG --- status code --- 06:56:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:22 DEBUG --- http response --- 06:56:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:56:22 DEBUG ----- output ----- 06:56:22 DEBUG { 06:56:22 DEBUG "status": "success", 06:56:22 DEBUG "data": { 06:56:22 DEBUG "resultType": "vector", 06:56:22 DEBUG "result": [ 06:56:22 DEBUG { 06:56:22 DEBUG "metric": { 06:56:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG [...] 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG "pod": "ds-cts-0" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 1712814981, 06:56:22 DEBUG "11178927.51851097" 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG } 06:56:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712814982" 06:56:22 INFO [http_cmd]: http status code OK 06:56:22 DEBUG --- status code --- 06:56:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:22 DEBUG --- http response --- 06:56:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:56:22 DEBUG ----- output ----- 06:56:22 DEBUG { 06:56:22 DEBUG "status": "success", 06:56:22 DEBUG "data": { 06:56:22 DEBUG "resultType": "vector", 06:56:22 DEBUG "result": [ 06:56:22 DEBUG { 06:56:22 DEBUG "metric": { 06:56:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG [...] 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 1712814982, 06:56:22 DEBUG "0.021" 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG } 06:56:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712814982" 06:56:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814982" 06:56:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814982" 06:56:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814982" 06:56:22 INFO [http_cmd]: http status code OK 06:56:22 DEBUG --- status code --- 06:56:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:22 DEBUG --- http response --- 06:56:22 INFO [http_cmd]: http status code OK 06:56:22 INFO [http_cmd]: http status code OK 06:56:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:56:22 DEBUG --- status code --- 06:56:22 DEBUG --- status code --- 06:56:22 DEBUG ----- output ----- 06:56:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:22 DEBUG { 06:56:22 DEBUG --- http response --- 06:56:22 DEBUG --- http response --- 06:56:22 DEBUG "status": "success", 06:56:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:56:22 DEBUG "data": { 06:56:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:56:22 DEBUG ----- output ----- 06:56:22 DEBUG "resultType": "vector", 06:56:22 DEBUG ----- output ----- 06:56:22 DEBUG { 06:56:22 DEBUG "result": [ 06:56:22 DEBUG { 06:56:22 DEBUG "status": "success", 06:56:22 DEBUG { 06:56:22 DEBUG "status": "success", 06:56:22 DEBUG "data": { 06:56:22 DEBUG "metric": { 06:56:22 DEBUG "data": { 06:56:22 DEBUG "resultType": "vector", 06:56:22 DEBUG "pod": "ds-idrepo-2" 06:56:22 DEBUG "resultType": "vector", 06:56:22 DEBUG "result": [ 06:56:22 DEBUG }, 06:56:22 DEBUG "result": [ 06:56:22 DEBUG { 06:56:22 DEBUG "value": [ 06:56:22 DEBUG { 06:56:22 DEBUG "metric": { 06:56:22 DEBUG 06:56:22 DEBUG "metric": { 06:56:22 DEBUG "instance": "10.142.0.107:9100" 06:56:22 DEBUG 06:56:22 DEBUG "instance": "10.142.0.107:9100" 06:56:22 DEBUG }, 06:56:22 DEBUG 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG [...] 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG [...] 06:56:22 DEBUG "pod": "ds-cts-2" 06:56:22 DEBUG [...] 06:56:22 DEBUG 06:56:22 DEBUG }, 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 1712814982, 06:56:22 DEBUG 06:56:22 DEBUG "instance": "10.142.0.77:9100" 06:56:22 DEBUG "0" 06:56:22 DEBUG "instance": "10.142.0.77:9100" 06:56:22 DEBUG }, 06:56:22 DEBUG ] 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG } 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 1712814982, 06:56:22 DEBUG ] 06:56:22 DEBUG 1712814982, 06:56:22 DEBUG "82989680.16666666" 06:56:22 DEBUG } 06:56:22 DEBUG "0.0021000000000000797" 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG } 06:56:22 DEBUG ] 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG } 06:56:22 DEBUG } 06:56:22 DEBUG } 06:56:22 INFO [http_cmd]: http status code OK 06:56:22 DEBUG --- status code --- 06:56:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:22 DEBUG --- http response --- 06:56:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:56:22 DEBUG ----- output ----- 06:56:22 DEBUG { 06:56:22 DEBUG "status": "success", 06:56:22 DEBUG "data": { 06:56:22 DEBUG "resultType": "vector", 06:56:22 DEBUG "result": [ 06:56:22 DEBUG { 06:56:22 DEBUG "metric": { 06:56:22 DEBUG "instance": "10.142.0.107:9100" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG [...] 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG "instance": "10.142.0.77:9100" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 1712814982, 06:56:22 DEBUG "82992027.06666666" 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG } 06:56:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814982" 06:56:22 INFO [http_cmd]: http status code OK 06:56:22 DEBUG --- status code --- 06:56:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:22 DEBUG --- http response --- 06:56:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:56:22 DEBUG ----- output ----- 06:56:22 DEBUG { 06:56:22 DEBUG "status": "success", 06:56:22 DEBUG "data": { 06:56:22 DEBUG "resultType": "vector", 06:56:22 DEBUG "result": [ 06:56:22 DEBUG { 06:56:22 DEBUG "metric": { 06:56:22 DEBUG "instance": "10.142.0.107:9100" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG [...] 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG "instance": "10.142.0.77:9100" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 1712814982, 06:56:22 DEBUG "52970431488" 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG } 06:56:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814982" 06:56:22 INFO [http_cmd]: http status code OK 06:56:22 DEBUG --- status code --- 06:56:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:22 DEBUG --- http response --- 06:56:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:56:22 DEBUG ----- output ----- 06:56:22 DEBUG { 06:56:22 DEBUG "status": "success", 06:56:22 DEBUG "data": { 06:56:22 DEBUG "resultType": "vector", 06:56:22 DEBUG "result": [ 06:56:22 DEBUG { 06:56:22 DEBUG "metric": { 06:56:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG [...] 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 1712814982, 06:56:22 DEBUG "1361.7333333333333" 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG } 06:56:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814982" 06:56:22 INFO [http_cmd]: http status code OK 06:56:22 DEBUG --- status code --- 06:56:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:22 DEBUG --- http response --- 06:56:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:56:22 DEBUG ----- output ----- 06:56:22 DEBUG { 06:56:22 DEBUG "status": "success", 06:56:22 DEBUG "data": { 06:56:22 DEBUG "resultType": "vector", 06:56:22 DEBUG "result": [ 06:56:22 DEBUG { 06:56:22 DEBUG "metric": { 06:56:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG [...] 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG "pod": "ds-cts-0" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 1712814982, 06:56:22 DEBUG "24034123.768765707" 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG } 06:56:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814982" 06:56:22 INFO [http_cmd]: http status code OK 06:56:22 DEBUG --- status code --- 06:56:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:22 DEBUG --- http response --- 06:56:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:56:22 DEBUG ----- output ----- 06:56:22 DEBUG { 06:56:22 DEBUG "status": "success", 06:56:22 DEBUG "data": { 06:56:22 DEBUG "resultType": "vector", 06:56:22 DEBUG "result": [ 06:56:22 DEBUG { 06:56:22 DEBUG "metric": { 06:56:22 DEBUG "pod": "ds-idrepo-2" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG [...] 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG "pod": "ds-idrepo-0" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 1712814982, 06:56:22 DEBUG "0" 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG } 06:56:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712814982" 06:56:22 INFO [http_cmd]: http status code OK 06:56:22 DEBUG --- status code --- 06:56:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:22 DEBUG --- http response --- 06:56:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:56:22 DEBUG ----- output ----- 06:56:22 DEBUG { 06:56:22 DEBUG "status": "success", 06:56:22 DEBUG "data": { 06:56:22 DEBUG "resultType": "vector", 06:56:22 DEBUG "result": [ 06:56:22 DEBUG { 06:56:22 DEBUG "metric": { 06:56:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG [...] 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 1712814982, 06:56:22 DEBUG "4.2158102494191985" 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG } 06:56:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712814982" 06:56:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814982" 06:56:22 INFO [http_cmd]: http status code OK 06:56:22 DEBUG --- status code --- 06:56:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:22 DEBUG --- http response --- 06:56:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:56:22 DEBUG ----- output ----- 06:56:22 DEBUG { 06:56:22 DEBUG "status": "success", 06:56:22 DEBUG "data": { 06:56:22 DEBUG "resultType": "vector", 06:56:22 DEBUG "result": [ 06:56:22 DEBUG { 06:56:22 DEBUG "metric": { 06:56:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG [...] 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 1712814982, 06:56:22 DEBUG "13640585216" 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG } 06:56:22 INFO [http_cmd]: http status code OK 06:56:22 DEBUG --- status code --- 06:56:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:22 DEBUG --- http response --- 06:56:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:56:22 DEBUG ----- output ----- 06:56:22 DEBUG { 06:56:22 DEBUG "status": "success", 06:56:22 DEBUG "data": { 06:56:22 DEBUG "resultType": "vector", 06:56:22 DEBUG "result": [ 06:56:22 DEBUG { 06:56:22 DEBUG "metric": { 06:56:22 DEBUG "pod": "ds-idrepo-2" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG [...] 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG "pod": "ds-cts-2" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 1712814982, 06:56:22 DEBUG "29.03333333333333" 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG } 06:56:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814982" 06:56:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814982" 06:56:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814982" 06:56:22 INFO [http_cmd]: http status code OK 06:56:22 DEBUG --- status code --- 06:56:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:22 DEBUG --- http response --- 06:56:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:56:22 DEBUG ----- output ----- 06:56:22 DEBUG { 06:56:22 DEBUG "status": "success", 06:56:22 DEBUG "data": { 06:56:22 DEBUG "resultType": "vector", 06:56:22 DEBUG "result": [ 06:56:22 DEBUG { 06:56:22 DEBUG "metric": { 06:56:22 DEBUG "pod": "ds-idrepo-2" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG [...] 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG "pod": "ds-idrepo-0" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 1712814982, 06:56:22 DEBUG "0" 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG } 06:56:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814982" 06:56:22 INFO [http_cmd]: http status code OK 06:56:22 DEBUG --- status code --- 06:56:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:22 DEBUG --- http response --- 06:56:22 INFO [http_cmd]: http status code OK 06:56:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:56:22 DEBUG --- status code --- 06:56:22 DEBUG ----- output ----- 06:56:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:22 DEBUG { 06:56:22 DEBUG --- http response --- 06:56:22 DEBUG "status": "success", 06:56:22 DEBUG "data": { 06:56:22 DEBUG "resultType": "vector", 06:56:22 DEBUG "result": [ 06:56:22 DEBUG { 06:56:22 DEBUG "metric": { 06:56:22 DEBUG "pod": "ds-idrepo-2" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG [...] 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG "pod": "ds-cts-2" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 1712814982, 06:56:22 DEBUG "0" 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG } 06:56:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:56:22 DEBUG ----- output ----- 06:56:22 DEBUG { 06:56:22 DEBUG "status": "success", 06:56:22 DEBUG "data": { 06:56:22 DEBUG "resultType": "vector", 06:56:22 DEBUG "result": [ 06:56:22 DEBUG { 06:56:22 DEBUG "metric": { 06:56:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG [...] 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG "pod": "ds-cts-0" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 1712814982, 06:56:22 DEBUG "4878138.169961279" 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG } 06:56:22 INFO [http_cmd]: http status code OK 06:56:22 DEBUG --- status code --- 06:56:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:22 DEBUG --- http response --- 06:56:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:56:22 DEBUG ----- output ----- 06:56:22 DEBUG { 06:56:22 DEBUG "status": "success", 06:56:22 DEBUG "data": { 06:56:22 DEBUG "resultType": "vector", 06:56:22 DEBUG "result": [ 06:56:22 DEBUG { 06:56:22 DEBUG "metric": { 06:56:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG [...] 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 1712814982, 06:56:22 DEBUG "1361.6666666666667" 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG } 06:56:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814982" 06:56:22 INFO [http_cmd]: http status code OK 06:56:22 DEBUG --- status code --- 06:56:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:22 DEBUG --- http response --- 06:56:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:56:22 DEBUG ----- output ----- 06:56:22 DEBUG { 06:56:22 DEBUG "status": "success", 06:56:22 DEBUG "data": { 06:56:22 DEBUG "resultType": "vector", 06:56:22 DEBUG "result": [ 06:56:22 DEBUG { 06:56:22 DEBUG "metric": { 06:56:22 DEBUG "pod": "ds-idrepo-2" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG [...] 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG "pod": "ds-cts-2" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 1712814982, 06:56:22 DEBUG "0" 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG } 06:56:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814982" 06:56:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712814982" 06:56:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712814982" 06:56:22 INFO [http_cmd]: http status code OK 06:56:22 DEBUG --- status code --- 06:56:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:22 DEBUG --- http response --- 06:56:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:56:22 DEBUG ----- output ----- 06:56:22 DEBUG { 06:56:22 DEBUG "status": "success", 06:56:22 DEBUG "data": { 06:56:22 DEBUG "resultType": "vector", 06:56:22 DEBUG "result": [ 06:56:22 DEBUG { 06:56:22 DEBUG "metric": { 06:56:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG [...] 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 1712814982, 06:56:22 DEBUG "0" 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG } 06:56:22 INFO [http_cmd]: http status code OK 06:56:22 DEBUG --- status code --- 06:56:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:22 DEBUG --- http response --- 06:56:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:56:22 DEBUG ----- output ----- 06:56:22 DEBUG { 06:56:22 DEBUG "status": "success", 06:56:22 DEBUG "data": { 06:56:22 DEBUG "resultType": "vector", 06:56:22 DEBUG "result": [ 06:56:22 DEBUG { 06:56:22 DEBUG "metric": { 06:56:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG [...] 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 1712814982, 06:56:22 DEBUG "1361.6" 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG } 06:56:22 INFO [http_cmd]: http status code OK 06:56:22 DEBUG --- status code --- 06:56:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:56:22 DEBUG --- http response --- 06:56:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:56:22 DEBUG ----- output ----- 06:56:22 DEBUG { 06:56:22 DEBUG "status": "success", 06:56:22 DEBUG "data": { 06:56:22 DEBUG "resultType": "vector", 06:56:22 DEBUG "result": [ 06:56:22 DEBUG { 06:56:22 DEBUG "metric": { 06:56:22 DEBUG "instance": "10.142.0.107:9100" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG [...] 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG 06:56:22 DEBUG "instance": "10.142.0.77:9100" 06:56:22 DEBUG }, 06:56:22 DEBUG "value": [ 06:56:22 DEBUG 1712814982, 06:56:22 DEBUG "0.0006666666666662973" 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG ] 06:56:22 DEBUG } 06:56:22 DEBUG } 06:56:33 INFO 06:56:33 INFO [loop_until]: kubectl --namespace=xlou top pods 06:56:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:56:33 INFO [loop_until]: OK (rc = 0) 06:56:33 DEBUG --- stdout --- 06:56:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 9204m 8504Mi am-5b5bd8567b-8h2w4 8770m 8821Mi am-5b5bd8567b-qlkw6 7416m 8458Mi ds-cts-0 8060m 13008Mi ds-cts-1 8767m 12577Mi ds-cts-2 8963m 13139Mi ds-idrepo-0 11385m 5503Mi ds-idrepo-1 348m 5479Mi ds-idrepo-2 364m 5784Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 3m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 2m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4167m 1866Mi 06:56:33 DEBUG --- stderr --- 06:56:33 DEBUG 06:56:37 INFO 06:56:37 INFO [loop_until]: kubectl --namespace=xlou top node 06:56:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:56:37 INFO [loop_until]: OK (rc = 0) 06:56:37 DEBUG --- stdout --- 06:56:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 2714Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 61m 0% 2779Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7946m 26% 10122Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10758m 36% 2978Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9633m 32% 10094Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 131m 0% 3857Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9658m 32% 10442Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 426m 2% 6464Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11549m 72% 6203Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8911m 56% 13770Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 8312m 52% 13715Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 8761m 55% 13269Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 402m 2% 6157Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4330m 27% 5617Mi 9% 06:56:37 DEBUG --- stderr --- 06:56:37 DEBUG 06:57:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815041" 06:57:21 INFO [http_cmd]: http status code OK 06:57:21 DEBUG --- status code --- 06:57:21 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:21 DEBUG --- http response --- 06:57:21 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:57:21 DEBUG ----- output ----- 06:57:21 DEBUG { 06:57:21 DEBUG "status": "success", 06:57:21 DEBUG "data": { 06:57:21 DEBUG "resultType": "vector", 06:57:21 DEBUG "result": [ 06:57:21 DEBUG { 06:57:21 DEBUG "metric": { 06:57:21 DEBUG "pod": "ds-idrepo-2" 06:57:21 DEBUG }, 06:57:21 DEBUG "value": [ 06:57:21 DEBUG 06:57:21 DEBUG 06:57:21 DEBUG 06:57:21 DEBUG [...] 06:57:21 DEBUG 06:57:21 DEBUG 06:57:21 DEBUG 06:57:21 DEBUG "pod": "ds-idrepo-0" 06:57:21 DEBUG }, 06:57:21 DEBUG "value": [ 06:57:21 DEBUG 1712815041, 06:57:21 DEBUG "0" 06:57:21 DEBUG ] 06:57:21 DEBUG } 06:57:21 DEBUG ] 06:57:21 DEBUG } 06:57:21 DEBUG } 06:57:21 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815041" 06:57:22 INFO [http_cmd]: http status code OK 06:57:22 DEBUG --- status code --- 06:57:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:22 DEBUG --- http response --- 06:57:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:57:22 DEBUG ----- output ----- 06:57:22 DEBUG { 06:57:22 DEBUG "status": "success", 06:57:22 DEBUG "data": { 06:57:22 DEBUG "resultType": "vector", 06:57:22 DEBUG "result": [ 06:57:22 DEBUG { 06:57:22 DEBUG "metric": { 06:57:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG [...] 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 1712815041, 06:57:22 DEBUG "244940" 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG } 06:57:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712815042" 06:57:22 ERROR [http_cmd]: ERROR 06:57:22 ERROR --- status code --- 06:57:22 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:57:22 ERROR --- http response --- 06:57:22 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:57:22 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:57:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815042" 06:57:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815042" 06:57:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712815042" 06:57:22 INFO [http_cmd]: http status code OK 06:57:22 DEBUG --- status code --- 06:57:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:22 DEBUG --- http response --- 06:57:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 06:57:22 DEBUG ----- output ----- 06:57:22 DEBUG { 06:57:22 DEBUG "status": "success", 06:57:22 DEBUG "data": { 06:57:22 DEBUG "resultType": "vector", 06:57:22 DEBUG "result": [ 06:57:22 DEBUG { 06:57:22 DEBUG "metric": { 06:57:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG [...] 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG "pod": "idm-668d898cfd-st2t7" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 1712815042, 06:57:22 DEBUG "0.0021066142753997158" 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG } 06:57:22 INFO [http_cmd]: http status code OK 06:57:22 INFO [http_cmd]: http status code OK 06:57:22 DEBUG --- status code --- 06:57:22 DEBUG --- status code --- 06:57:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:22 DEBUG --- http response --- 06:57:22 DEBUG --- http response --- 06:57:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:57:22 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:57:22 DEBUG ----- output ----- 06:57:22 DEBUG { 06:57:22 DEBUG "status": "success", 06:57:22 DEBUG "data": { 06:57:22 DEBUG "resultType": "vector", 06:57:22 DEBUG "result": [ 06:57:22 DEBUG { 06:57:22 DEBUG "metric": { 06:57:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG [...] 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 1712815042, 06:57:22 DEBUG "0.021" 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG } 06:57:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815042" 06:57:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815042" 06:57:22 INFO [http_cmd]: http status code OK 06:57:22 DEBUG --- status code --- 06:57:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:22 DEBUG --- http response --- 06:57:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:57:22 DEBUG ----- output ----- 06:57:22 DEBUG { 06:57:22 DEBUG "status": "success", 06:57:22 DEBUG "data": { 06:57:22 DEBUG "resultType": "vector", 06:57:22 DEBUG "result": [ 06:57:22 DEBUG { 06:57:22 DEBUG "metric": { 06:57:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG [...] 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 1712815042, 06:57:22 DEBUG "1359.5333333333333" 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG } 06:57:22 INFO [http_cmd]: http status code OK 06:57:22 DEBUG --- status code --- 06:57:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:22 DEBUG --- http response --- 06:57:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 06:57:22 DEBUG ----- output ----- 06:57:22 DEBUG { 06:57:22 DEBUG "status": "success", 06:57:22 DEBUG "data": { 06:57:22 DEBUG "resultType": "vector", 06:57:22 DEBUG "result": [ 06:57:22 DEBUG { 06:57:22 DEBUG "metric": { 06:57:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG [...] 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG "pod": "idm-668d898cfd-st2t7" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 1712815042, 06:57:22 DEBUG "31365761.301315524" 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG } 06:57:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712815042" 06:57:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815042" 06:57:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815042" 06:57:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815042" 06:57:22 INFO [http_cmd]: http status code OK 06:57:22 DEBUG --- status code --- 06:57:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:22 DEBUG --- http response --- 06:57:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:57:22 DEBUG ----- output ----- 06:57:22 DEBUG { 06:57:22 DEBUG "status": "success", 06:57:22 DEBUG "data": { 06:57:22 DEBUG "resultType": "vector", 06:57:22 DEBUG "result": [ 06:57:22 DEBUG { 06:57:22 DEBUG "metric": { 06:57:22 DEBUG "pod": "ds-idrepo-2" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG [...] 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG "pod": "ds-cts-2" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 1712815042, 06:57:22 DEBUG "0" 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG } 06:57:22 INFO [http_cmd]: http status code OK 06:57:22 INFO [http_cmd]: http status code OK 06:57:22 DEBUG --- status code --- 06:57:22 DEBUG --- status code --- 06:57:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:22 DEBUG --- http response --- 06:57:22 DEBUG --- http response --- 06:57:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:57:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:57:22 DEBUG ----- output ----- 06:57:22 DEBUG ----- output ----- 06:57:22 DEBUG { 06:57:22 INFO [http_cmd]: http status code OK 06:57:22 DEBUG { 06:57:22 DEBUG "status": "success", 06:57:22 DEBUG --- status code --- 06:57:22 DEBUG "status": "success", 06:57:22 DEBUG "data": { 06:57:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:22 DEBUG "data": { 06:57:22 DEBUG "resultType": "vector", 06:57:22 DEBUG --- http response --- 06:57:22 DEBUG "resultType": "vector", 06:57:22 DEBUG "result": [ 06:57:22 DEBUG "result": [ 06:57:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:57:22 DEBUG { 06:57:22 DEBUG { 06:57:22 DEBUG ----- output ----- 06:57:22 DEBUG "metric": { 06:57:22 DEBUG "metric": { 06:57:22 DEBUG { 06:57:22 DEBUG "instance": "10.142.0.107:9100" 06:57:22 DEBUG "instance": "10.142.0.107:9100" 06:57:22 DEBUG "status": "success", 06:57:22 DEBUG }, 06:57:22 DEBUG }, 06:57:22 DEBUG "data": { 06:57:22 DEBUG "value": [ 06:57:22 DEBUG "value": [ 06:57:22 DEBUG "resultType": "vector", 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG "result": [ 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG { 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG "metric": { 06:57:22 DEBUG [...] 06:57:22 DEBUG [...] 06:57:22 DEBUG "instance": "10.142.0.107:9100" 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG }, 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG "instance": "10.142.0.77:9100" 06:57:22 DEBUG "instance": "10.142.0.77:9100" 06:57:22 DEBUG 06:57:22 DEBUG }, 06:57:22 DEBUG }, 06:57:22 DEBUG 06:57:22 DEBUG "value": [ 06:57:22 DEBUG "value": [ 06:57:22 DEBUG [...] 06:57:22 DEBUG 1712815042, 06:57:22 DEBUG 1712815042, 06:57:22 DEBUG 06:57:22 DEBUG "82784325.9" 06:57:22 DEBUG "82788785.86666667" 06:57:22 DEBUG 06:57:22 DEBUG ] 06:57:22 DEBUG ] 06:57:22 DEBUG 06:57:22 DEBUG } 06:57:22 DEBUG } 06:57:22 DEBUG "instance": "10.142.0.77:9100" 06:57:22 DEBUG ] 06:57:22 DEBUG ] 06:57:22 DEBUG }, 06:57:22 DEBUG } 06:57:22 DEBUG } 06:57:22 DEBUG "value": [ 06:57:22 DEBUG } 06:57:22 DEBUG } 06:57:22 DEBUG 1712815042, 06:57:22 DEBUG "0.00220000000000032" 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG } 06:57:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815042" 06:57:22 INFO [http_cmd]: http status code OK 06:57:22 DEBUG --- status code --- 06:57:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:22 DEBUG --- http response --- 06:57:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:57:22 DEBUG ----- output ----- 06:57:22 DEBUG { 06:57:22 DEBUG "status": "success", 06:57:22 DEBUG "data": { 06:57:22 DEBUG "resultType": "vector", 06:57:22 DEBUG "result": [ 06:57:22 DEBUG { 06:57:22 DEBUG "metric": { 06:57:22 DEBUG "instance": "10.142.0.107:9100" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG [...] 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG "instance": "10.142.0.77:9100" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 1712815042, 06:57:22 DEBUG "52971820032" 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG } 06:57:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815042" 06:57:22 INFO [http_cmd]: http status code OK 06:57:22 DEBUG --- status code --- 06:57:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:22 DEBUG --- http response --- 06:57:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:57:22 DEBUG ----- output ----- 06:57:22 DEBUG { 06:57:22 DEBUG "status": "success", 06:57:22 DEBUG "data": { 06:57:22 DEBUG "resultType": "vector", 06:57:22 DEBUG "result": [ 06:57:22 DEBUG { 06:57:22 DEBUG "metric": { 06:57:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG [...] 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 1712815042, 06:57:22 DEBUG "1371.1333333333332" 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG } 06:57:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815042" 06:57:22 INFO [http_cmd]: http status code OK 06:57:22 DEBUG --- status code --- 06:57:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:22 DEBUG --- http response --- 06:57:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 06:57:22 DEBUG ----- output ----- 06:57:22 DEBUG { 06:57:22 DEBUG "status": "success", 06:57:22 DEBUG "data": { 06:57:22 DEBUG "resultType": "vector", 06:57:22 DEBUG "result": [ 06:57:22 DEBUG { 06:57:22 DEBUG "metric": { 06:57:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG [...] 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG "pod": "idm-668d898cfd-st2t7" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 1712815042, 06:57:22 DEBUG "0" 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG } 06:57:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815042" 06:57:22 INFO [http_cmd]: http status code OK 06:57:22 DEBUG --- status code --- 06:57:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:22 DEBUG --- http response --- 06:57:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:57:22 DEBUG ----- output ----- 06:57:22 DEBUG { 06:57:22 DEBUG "status": "success", 06:57:22 DEBUG "data": { 06:57:22 DEBUG "resultType": "vector", 06:57:22 DEBUG "result": [ 06:57:22 DEBUG { 06:57:22 DEBUG "metric": { 06:57:22 DEBUG "pod": "ds-idrepo-2" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG [...] 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG "pod": "ds-idrepo-0" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 1712815042, 06:57:22 DEBUG "0" 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG } 06:57:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712815042" 06:57:22 INFO [http_cmd]: http status code OK 06:57:22 DEBUG --- status code --- 06:57:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:22 DEBUG --- http response --- 06:57:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:57:22 DEBUG ----- output ----- 06:57:22 DEBUG { 06:57:22 DEBUG "status": "success", 06:57:22 DEBUG "data": { 06:57:22 DEBUG "resultType": "vector", 06:57:22 DEBUG "result": [ 06:57:22 DEBUG { 06:57:22 DEBUG "metric": { 06:57:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG [...] 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 1712815042, 06:57:22 DEBUG "4.198550707060475" 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG } 06:57:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815042" 06:57:22 INFO [http_cmd]: http status code OK 06:57:22 DEBUG --- status code --- 06:57:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:22 DEBUG --- http response --- 06:57:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:57:22 DEBUG ----- output ----- 06:57:22 DEBUG { 06:57:22 DEBUG "status": "success", 06:57:22 DEBUG "data": { 06:57:22 DEBUG "resultType": "vector", 06:57:22 DEBUG "result": [ 06:57:22 DEBUG { 06:57:22 DEBUG "metric": { 06:57:22 DEBUG "pod": "ds-idrepo-2" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG [...] 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG "pod": "ds-cts-2" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 1712815042, 06:57:22 DEBUG "0" 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG } 06:57:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712815042" 06:57:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815042" 06:57:22 INFO [http_cmd]: http status code OK 06:57:22 DEBUG --- status code --- 06:57:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:22 DEBUG --- http response --- 06:57:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:57:22 DEBUG ----- output ----- 06:57:22 DEBUG { 06:57:22 DEBUG "status": "success", 06:57:22 DEBUG "data": { 06:57:22 DEBUG "resultType": "vector", 06:57:22 DEBUG "result": [ 06:57:22 DEBUG { 06:57:22 DEBUG "metric": { 06:57:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG [...] 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 1712815042, 06:57:22 DEBUG "14136954880" 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG } 06:57:22 INFO [http_cmd]: http status code OK 06:57:22 DEBUG --- status code --- 06:57:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:22 DEBUG --- http response --- 06:57:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:57:22 DEBUG ----- output ----- 06:57:22 DEBUG { 06:57:22 DEBUG "status": "success", 06:57:22 DEBUG "data": { 06:57:22 DEBUG "resultType": "vector", 06:57:22 DEBUG "result": [ 06:57:22 DEBUG { 06:57:22 DEBUG "metric": { 06:57:22 DEBUG "pod": "ds-idrepo-2" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG [...] 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG "pod": "ds-cts-2" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 1712815042, 06:57:22 DEBUG "29.966666666666665" 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG } 06:57:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815042" 06:57:22 INFO [http_cmd]: http status code OK 06:57:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815042" 06:57:22 DEBUG --- status code --- 06:57:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:22 DEBUG --- http response --- 06:57:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:57:22 DEBUG ----- output ----- 06:57:22 DEBUG { 06:57:22 DEBUG "status": "success", 06:57:22 DEBUG "data": { 06:57:22 DEBUG "resultType": "vector", 06:57:22 DEBUG "result": [ 06:57:22 DEBUG { 06:57:22 DEBUG "metric": { 06:57:22 DEBUG "pod": "ds-idrepo-2" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG [...] 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG "pod": "ds-idrepo-0" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 1712815042, 06:57:22 DEBUG "0" 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG } 06:57:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815042" 06:57:22 INFO [http_cmd]: http status code OK 06:57:22 DEBUG --- status code --- 06:57:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:22 DEBUG --- http response --- 06:57:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 06:57:22 DEBUG ----- output ----- 06:57:22 DEBUG { 06:57:22 DEBUG "status": "success", 06:57:22 DEBUG "data": { 06:57:22 DEBUG "resultType": "vector", 06:57:22 DEBUG "result": [ 06:57:22 DEBUG { 06:57:22 DEBUG "metric": { 06:57:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG [...] 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG "pod": "idm-668d898cfd-st2t7" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 1712815042, 06:57:22 DEBUG "0" 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG ] 06:57:22 INFO [http_cmd]: http status code OK 06:57:22 DEBUG } 06:57:22 DEBUG --- status code --- 06:57:22 DEBUG } 06:57:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:22 DEBUG --- http response --- 06:57:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:57:22 DEBUG ----- output ----- 06:57:22 DEBUG { 06:57:22 DEBUG "status": "success", 06:57:22 DEBUG "data": { 06:57:22 DEBUG "resultType": "vector", 06:57:22 DEBUG "result": [ 06:57:22 DEBUG { 06:57:22 DEBUG "metric": { 06:57:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG [...] 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 1712815042, 06:57:22 DEBUG "1370.8333333333333" 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG } 06:57:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815042" 06:57:22 INFO [http_cmd]: http status code OK 06:57:22 DEBUG --- status code --- 06:57:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:22 DEBUG --- http response --- 06:57:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:57:22 DEBUG ----- output ----- 06:57:22 DEBUG { 06:57:22 DEBUG "status": "success", 06:57:22 DEBUG "data": { 06:57:22 DEBUG "resultType": "vector", 06:57:22 DEBUG "result": [ 06:57:22 DEBUG { 06:57:22 DEBUG "metric": { 06:57:22 DEBUG "pod": "ds-idrepo-2" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG [...] 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG "pod": "ds-cts-2" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 1712815042, 06:57:22 DEBUG "0.000033333333333333335" 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG } 06:57:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815042" 06:57:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815042" 06:57:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815042" 06:57:22 INFO [http_cmd]: http status code OK 06:57:22 INFO [http_cmd]: http status code OK 06:57:22 DEBUG --- status code --- 06:57:22 DEBUG --- status code --- 06:57:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:22 DEBUG --- http response --- 06:57:22 DEBUG --- http response --- 06:57:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:57:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:57:22 DEBUG ----- output ----- 06:57:22 DEBUG ----- output ----- 06:57:22 DEBUG { 06:57:22 DEBUG { 06:57:22 DEBUG "status": "success", 06:57:22 DEBUG "status": "success", 06:57:22 DEBUG "data": { 06:57:22 DEBUG "data": { 06:57:22 DEBUG "resultType": "vector", 06:57:22 DEBUG "resultType": "vector", 06:57:22 DEBUG "result": [ 06:57:22 DEBUG "result": [ 06:57:22 DEBUG { 06:57:22 DEBUG { 06:57:22 DEBUG "metric": { 06:57:22 DEBUG "metric": { 06:57:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:57:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:57:22 DEBUG }, 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG [...] 06:57:22 DEBUG [...] 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:57:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:57:22 DEBUG }, 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 1712815042, 06:57:22 DEBUG 1712815042, 06:57:22 DEBUG "1361.6666666666667" 06:57:22 DEBUG "0" 06:57:22 DEBUG ] 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG } 06:57:22 DEBUG ] 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG } 06:57:22 DEBUG } 06:57:22 DEBUG } 06:57:22 INFO [http_cmd]: http status code OK 06:57:22 DEBUG --- status code --- 06:57:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:57:22 DEBUG --- http response --- 06:57:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:57:22 DEBUG ----- output ----- 06:57:22 DEBUG { 06:57:22 DEBUG "status": "success", 06:57:22 DEBUG "data": { 06:57:22 DEBUG "resultType": "vector", 06:57:22 DEBUG "result": [ 06:57:22 DEBUG { 06:57:22 DEBUG "metric": { 06:57:22 DEBUG "instance": "10.142.0.107:9100" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG [...] 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG 06:57:22 DEBUG "instance": "10.142.0.77:9100" 06:57:22 DEBUG }, 06:57:22 DEBUG "value": [ 06:57:22 DEBUG 1712815042, 06:57:22 DEBUG "0" 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG ] 06:57:22 DEBUG } 06:57:22 DEBUG } 06:57:34 INFO 06:57:34 INFO [loop_until]: kubectl --namespace=xlou top pods 06:57:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:57:34 INFO [loop_until]: OK (rc = 0) 06:57:34 DEBUG --- stdout --- 06:57:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 8746m 8507Mi am-5b5bd8567b-8h2w4 8690m 8818Mi am-5b5bd8567b-qlkw6 7683m 8458Mi ds-cts-0 7808m 13395Mi ds-cts-1 8787m 13366Mi ds-cts-2 8664m 13391Mi ds-idrepo-0 11793m 5503Mi ds-idrepo-1 332m 5480Mi ds-idrepo-2 387m 5784Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 4m 1010Mi idm-668d898cfd-st2t7 7m 1135Mi lodemon-ccc964465-p7gdm 2m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4270m 1867Mi 06:57:34 DEBUG --- stderr --- 06:57:34 DEBUG 06:57:37 INFO 06:57:37 INFO [loop_until]: kubectl --namespace=xlou top node 06:57:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:57:38 INFO [loop_until]: OK (rc = 0) 06:57:38 DEBUG --- stdout --- 06:57:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 74m 0% 2715Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 66m 0% 2777Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7889m 26% 10106Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10832m 36% 2984Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9523m 31% 10102Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 142m 0% 3840Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9493m 31% 10435Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 410m 2% 6469Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11619m 73% 6205Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8564m 53% 13964Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 8141m 51% 14004Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 8732m 54% 14036Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 400m 2% 6161Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4304m 27% 5620Mi 9% 06:57:38 DEBUG --- stderr --- 06:57:38 DEBUG 06:58:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815102" 06:58:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815102" 06:58:22 INFO [http_cmd]: http status code OK 06:58:22 INFO [http_cmd]: http status code OK 06:58:22 DEBUG --- status code --- 06:58:22 DEBUG --- status code --- 06:58:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:22 DEBUG --- http response --- 06:58:22 DEBUG --- http response --- 06:58:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:58:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:58:22 DEBUG ----- output ----- 06:58:22 DEBUG ----- output ----- 06:58:22 DEBUG { 06:58:22 DEBUG { 06:58:22 DEBUG "status": "success", 06:58:22 DEBUG "status": "success", 06:58:22 DEBUG "data": { 06:58:22 DEBUG "data": { 06:58:22 DEBUG "resultType": "vector", 06:58:22 DEBUG "resultType": "vector", 06:58:22 DEBUG "result": [ 06:58:22 DEBUG "result": [ 06:58:22 DEBUG { 06:58:22 DEBUG { 06:58:22 DEBUG "metric": { 06:58:22 DEBUG "metric": { 06:58:22 DEBUG "pod": "ds-idrepo-2" 06:58:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:58:22 DEBUG }, 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG [...] 06:58:22 DEBUG [...] 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG "pod": "ds-idrepo-0" 06:58:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:58:22 DEBUG }, 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 1712815102, 06:58:22 DEBUG 1712815102, 06:58:22 DEBUG "0" 06:58:22 DEBUG "244777.5" 06:58:22 DEBUG ] 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG } 06:58:22 DEBUG ] 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG } 06:58:22 DEBUG } 06:58:22 DEBUG } 06:58:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712815102" 06:58:22 ERROR [http_cmd]: ERROR 06:58:22 ERROR --- status code --- 06:58:22 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:58:22 ERROR --- http response --- 06:58:22 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:58:22 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:58:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815102" 06:58:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815102" 06:58:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712815102" 06:58:22 INFO [http_cmd]: http status code OK 06:58:22 INFO [http_cmd]: http status code OK 06:58:22 DEBUG --- status code --- 06:58:22 DEBUG --- status code --- 06:58:22 INFO [http_cmd]: http status code OK 06:58:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:22 DEBUG --- status code --- 06:58:22 DEBUG --- http response --- 06:58:22 DEBUG --- http response --- 06:58:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:22 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:58:22 DEBUG --- http response --- 06:58:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:58:22 DEBUG ----- output ----- 06:58:22 DEBUG { 06:58:22 DEBUG "status": "success", 06:58:22 DEBUG "data": { 06:58:22 DEBUG "resultType": "vector", 06:58:22 DEBUG "result": [ 06:58:22 DEBUG { 06:58:22 DEBUG "metric": { 06:58:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG [...] 06:58:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:58:22 DEBUG 06:58:22 DEBUG ----- output ----- 06:58:22 DEBUG 06:58:22 DEBUG { 06:58:22 DEBUG 06:58:22 DEBUG "status": "success", 06:58:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:58:22 DEBUG "data": { 06:58:22 DEBUG }, 06:58:22 DEBUG "resultType": "vector", 06:58:22 DEBUG "value": [ 06:58:22 DEBUG "result": [ 06:58:22 DEBUG 1712815102, 06:58:22 DEBUG { 06:58:22 DEBUG "0.021" 06:58:22 DEBUG "metric": { 06:58:22 DEBUG ] 06:58:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:58:22 DEBUG } 06:58:22 DEBUG }, 06:58:22 DEBUG ] 06:58:22 DEBUG "value": [ 06:58:22 DEBUG } 06:58:22 DEBUG 06:58:22 DEBUG } 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG [...] 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 1712815102, 06:58:22 DEBUG "4.106421285578556" 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG } 06:58:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815102" 06:58:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815102" 06:58:22 INFO [http_cmd]: http status code OK 06:58:22 DEBUG --- status code --- 06:58:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:22 DEBUG --- http response --- 06:58:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:58:22 INFO [http_cmd]: http status code OK 06:58:22 DEBUG ----- output ----- 06:58:22 DEBUG --- status code --- 06:58:22 DEBUG { 06:58:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:22 DEBUG "status": "success", 06:58:22 DEBUG --- http response --- 06:58:22 DEBUG "data": { 06:58:22 DEBUG "resultType": "vector", 06:58:22 DEBUG "result": [ 06:58:22 DEBUG { 06:58:22 DEBUG "metric": { 06:58:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:58:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:58:22 DEBUG }, 06:58:22 DEBUG ----- output ----- 06:58:22 DEBUG "value": [ 06:58:22 DEBUG { 06:58:22 DEBUG 06:58:22 DEBUG "status": "success", 06:58:22 DEBUG 06:58:22 DEBUG "data": { 06:58:22 DEBUG 06:58:22 DEBUG "resultType": "vector", 06:58:22 DEBUG [...] 06:58:22 DEBUG "result": [ 06:58:22 DEBUG 06:58:22 DEBUG { 06:58:22 DEBUG 06:58:22 DEBUG "metric": { 06:58:22 DEBUG 06:58:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:58:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:58:22 DEBUG }, 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 06:58:22 DEBUG 1712815102, 06:58:22 DEBUG 06:58:22 DEBUG "1366.9666666666667" 06:58:22 DEBUG 06:58:22 DEBUG ] 06:58:22 DEBUG [...] 06:58:22 DEBUG } 06:58:22 DEBUG 06:58:22 DEBUG ] 06:58:22 DEBUG 06:58:22 DEBUG } 06:58:22 DEBUG 06:58:22 DEBUG } 06:58:22 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 1712815102, 06:58:22 DEBUG "77472044.28803037" 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG } 06:58:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815102" 06:58:22 INFO [http_cmd]: http status code OK 06:58:22 DEBUG --- status code --- 06:58:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:22 DEBUG --- http response --- 06:58:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:58:22 DEBUG ----- output ----- 06:58:22 DEBUG { 06:58:22 DEBUG "status": "success", 06:58:22 DEBUG "data": { 06:58:22 DEBUG "resultType": "vector", 06:58:22 DEBUG "result": [ 06:58:22 DEBUG { 06:58:22 DEBUG "metric": { 06:58:22 DEBUG "instance": "10.142.0.107:9100" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG [...] 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG "instance": "10.142.0.77:9100" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 1712815102, 06:58:22 DEBUG "0.0021999999999996094" 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG } 06:58:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815102" 06:58:22 INFO [http_cmd]: http status code OK 06:58:22 DEBUG --- status code --- 06:58:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:22 DEBUG --- http response --- 06:58:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:58:22 DEBUG ----- output ----- 06:58:22 DEBUG { 06:58:22 DEBUG "status": "success", 06:58:22 DEBUG "data": { 06:58:22 DEBUG "resultType": "vector", 06:58:22 DEBUG "result": [ 06:58:22 DEBUG { 06:58:22 DEBUG "metric": { 06:58:22 DEBUG "instance": "10.142.0.107:9100" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG [...] 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG "instance": "10.142.0.77:9100" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815102" 06:58:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712815102" 06:58:22 DEBUG 1712815102, 06:58:22 DEBUG "82430856.5" 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG } 06:58:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815102" 06:58:22 INFO [http_cmd]: http status code OK 06:58:22 INFO [http_cmd]: http status code OK 06:58:22 DEBUG --- status code --- 06:58:22 DEBUG --- status code --- 06:58:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:22 DEBUG --- http response --- 06:58:22 DEBUG --- http response --- 06:58:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:58:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:58:22 DEBUG ----- output ----- 06:58:22 DEBUG ----- output ----- 06:58:22 DEBUG { 06:58:22 DEBUG { 06:58:22 DEBUG "status": "success", 06:58:22 DEBUG "status": "success", 06:58:22 DEBUG "data": { 06:58:22 DEBUG "data": { 06:58:22 DEBUG "resultType": "vector", 06:58:22 DEBUG "resultType": "vector", 06:58:22 DEBUG "result": [ 06:58:22 DEBUG "result": [ 06:58:22 DEBUG { 06:58:22 DEBUG { 06:58:22 DEBUG "metric": { 06:58:22 DEBUG "metric": { 06:58:22 DEBUG "pod": "ds-idrepo-2" 06:58:22 DEBUG "instance": "10.142.0.107:9100" 06:58:22 DEBUG }, 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG [...] 06:58:22 DEBUG [...] 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG "pod": "ds-cts-2" 06:58:22 DEBUG "instance": "10.142.0.77:9100" 06:58:22 DEBUG }, 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 1712815102, 06:58:22 DEBUG 1712815102, 06:58:22 DEBUG "0" 06:58:22 DEBUG "52973100032" 06:58:22 DEBUG ] 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG } 06:58:22 DEBUG ] 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG } 06:58:22 DEBUG } 06:58:22 DEBUG } 06:58:22 INFO [http_cmd]: http status code OK 06:58:22 DEBUG --- status code --- 06:58:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:22 DEBUG --- http response --- 06:58:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:58:22 DEBUG ----- output ----- 06:58:22 DEBUG { 06:58:22 DEBUG "status": "success", 06:58:22 DEBUG "data": { 06:58:22 DEBUG "resultType": "vector", 06:58:22 DEBUG "result": [ 06:58:22 DEBUG { 06:58:22 DEBUG "metric": { 06:58:22 DEBUG "instance": "10.142.0.107:9100" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG [...] 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG "instance": "10.142.0.77:9100" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 1712815102, 06:58:22 DEBUG "82428369.96666667" 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG } 06:58:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815102" 06:58:22 INFO [http_cmd]: http status code OK 06:58:22 DEBUG --- status code --- 06:58:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:22 DEBUG --- http response --- 06:58:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:58:22 DEBUG ----- output ----- 06:58:22 DEBUG { 06:58:22 DEBUG "status": "success", 06:58:22 DEBUG "data": { 06:58:22 DEBUG "resultType": "vector", 06:58:22 DEBUG "result": [ 06:58:22 DEBUG { 06:58:22 DEBUG "metric": { 06:58:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG [...] 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 1712815102, 06:58:22 DEBUG "1364.1" 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG } 06:58:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815102" 06:58:22 INFO [http_cmd]: http status code OK 06:58:22 DEBUG --- status code --- 06:58:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:22 DEBUG --- http response --- 06:58:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:58:22 DEBUG ----- output ----- 06:58:22 DEBUG { 06:58:22 DEBUG "status": "success", 06:58:22 DEBUG "data": { 06:58:22 DEBUG "resultType": "vector", 06:58:22 DEBUG "result": [ 06:58:22 DEBUG { 06:58:22 DEBUG "metric": { 06:58:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG [...] 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 1712815102, 06:58:22 DEBUG "0" 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG } 06:58:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815102" 06:58:22 INFO [http_cmd]: http status code OK 06:58:22 DEBUG --- status code --- 06:58:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:22 DEBUG --- http response --- 06:58:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:58:22 DEBUG ----- output ----- 06:58:22 DEBUG { 06:58:22 DEBUG "status": "success", 06:58:22 DEBUG "data": { 06:58:22 DEBUG "resultType": "vector", 06:58:22 DEBUG "result": [ 06:58:22 DEBUG { 06:58:22 DEBUG "metric": { 06:58:22 DEBUG "pod": "ds-idrepo-2" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG [...] 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG "pod": "ds-idrepo-0" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 1712815102, 06:58:22 DEBUG "0" 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG } 06:58:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712815102" 06:58:22 INFO [http_cmd]: http status code OK 06:58:22 DEBUG --- status code --- 06:58:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:22 DEBUG --- http response --- 06:58:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:58:22 DEBUG ----- output ----- 06:58:22 DEBUG { 06:58:22 DEBUG "status": "success", 06:58:22 DEBUG "data": { 06:58:22 DEBUG "resultType": "vector", 06:58:22 DEBUG "result": [ 06:58:22 DEBUG { 06:58:22 DEBUG "metric": { 06:58:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG [...] 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 1712815102, 06:58:22 DEBUG "4.151964806419667" 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG } 06:58:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815102" 06:58:22 INFO [http_cmd]: http status code OK 06:58:22 DEBUG --- status code --- 06:58:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:22 DEBUG --- http response --- 06:58:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:58:22 DEBUG ----- output ----- 06:58:22 DEBUG { 06:58:22 DEBUG "status": "success", 06:58:22 DEBUG "data": { 06:58:22 DEBUG "resultType": "vector", 06:58:22 DEBUG "result": [ 06:58:22 DEBUG { 06:58:22 DEBUG "metric": { 06:58:22 DEBUG "pod": "ds-idrepo-2" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG [...] 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG "pod": "ds-cts-2" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 1712815102, 06:58:22 DEBUG "0" 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG } 06:58:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712815102" 06:58:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815102" 06:58:22 INFO [http_cmd]: http status code OK 06:58:22 DEBUG --- status code --- 06:58:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:22 DEBUG --- http response --- 06:58:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:58:22 DEBUG ----- output ----- 06:58:22 DEBUG { 06:58:22 DEBUG "status": "success", 06:58:22 DEBUG "data": { 06:58:22 DEBUG "resultType": "vector", 06:58:22 DEBUG "result": [ 06:58:22 DEBUG { 06:58:22 DEBUG "metric": { 06:58:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG [...] 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 1712815102, 06:58:22 DEBUG "14135001088" 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG } 06:58:22 INFO [http_cmd]: http status code OK 06:58:22 DEBUG --- status code --- 06:58:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:22 DEBUG --- http response --- 06:58:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:58:22 DEBUG ----- output ----- 06:58:22 DEBUG { 06:58:22 DEBUG "status": "success", 06:58:22 DEBUG "data": { 06:58:22 DEBUG "resultType": "vector", 06:58:22 DEBUG "result": [ 06:58:22 DEBUG { 06:58:22 DEBUG "metric": { 06:58:22 DEBUG "pod": "ds-idrepo-2" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG [...] 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG "pod": "ds-cts-2" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 1712815102, 06:58:22 DEBUG "29.366666666666667" 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG } 06:58:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815102" 06:58:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815102" 06:58:22 INFO [http_cmd]: http status code OK 06:58:22 DEBUG --- status code --- 06:58:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:22 DEBUG --- http response --- 06:58:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:58:22 DEBUG ----- output ----- 06:58:22 DEBUG { 06:58:22 DEBUG "status": "success", 06:58:22 DEBUG "data": { 06:58:22 DEBUG "resultType": "vector", 06:58:22 DEBUG "result": [ 06:58:22 DEBUG { 06:58:22 DEBUG "metric": { 06:58:22 DEBUG "pod": "ds-idrepo-2" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG [...] 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG "pod": "ds-idrepo-0" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 1712815102, 06:58:22 DEBUG "0" 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG } 06:58:22 INFO [http_cmd]: http status code OK 06:58:22 DEBUG --- status code --- 06:58:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:22 DEBUG --- http response --- 06:58:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815102" 06:58:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:58:22 DEBUG ----- output ----- 06:58:22 DEBUG { 06:58:22 DEBUG "status": "success", 06:58:22 DEBUG "data": { 06:58:22 DEBUG "resultType": "vector", 06:58:22 DEBUG "result": [ 06:58:22 DEBUG { 06:58:22 DEBUG "metric": { 06:58:22 DEBUG "pod": "ds-idrepo-2" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG [...] 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG "pod": "ds-cts-2" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 1712815102, 06:58:22 DEBUG "0" 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG } 06:58:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815102" 06:58:22 INFO [http_cmd]: http status code OK 06:58:22 DEBUG --- status code --- 06:58:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:22 DEBUG --- http response --- 06:58:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:58:22 DEBUG ----- output ----- 06:58:22 DEBUG { 06:58:22 DEBUG "status": "success", 06:58:22 DEBUG "data": { 06:58:22 DEBUG "resultType": "vector", 06:58:22 DEBUG "result": [ 06:58:22 DEBUG { 06:58:22 DEBUG "metric": { 06:58:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG [...] 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 1712815102, 06:58:22 DEBUG "1364.7" 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 INFO [http_cmd]: http status code OK 06:58:22 DEBUG ] 06:58:22 DEBUG --- status code --- 06:58:22 DEBUG } 06:58:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:22 DEBUG } 06:58:22 DEBUG --- http response --- 06:58:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:58:22 DEBUG ----- output ----- 06:58:22 DEBUG { 06:58:22 DEBUG "status": "success", 06:58:22 DEBUG "data": { 06:58:22 DEBUG "resultType": "vector", 06:58:22 DEBUG "result": [ 06:58:22 DEBUG { 06:58:22 DEBUG "metric": { 06:58:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG [...] 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 1712815102, 06:58:22 DEBUG "0" 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG } 06:58:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815102" 06:58:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815102" 06:58:22 INFO [http_cmd]: http status code OK 06:58:22 DEBUG --- status code --- 06:58:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:22 DEBUG --- http response --- 06:58:22 INFO [http_cmd]: http status code OK 06:58:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:58:22 DEBUG --- status code --- 06:58:22 DEBUG ----- output ----- 06:58:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:22 DEBUG { 06:58:22 DEBUG --- http response --- 06:58:22 DEBUG "status": "success", 06:58:22 DEBUG "data": { 06:58:22 DEBUG "resultType": "vector", 06:58:22 DEBUG "result": [ 06:58:22 DEBUG { 06:58:22 DEBUG "metric": { 06:58:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG [...] 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 1712815102, 06:58:22 DEBUG "0" 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG } 06:58:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:58:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815102" 06:58:22 DEBUG ----- output ----- 06:58:22 DEBUG { 06:58:22 DEBUG "status": "success", 06:58:22 DEBUG "data": { 06:58:22 DEBUG "resultType": "vector", 06:58:22 DEBUG "result": [ 06:58:22 DEBUG { 06:58:22 DEBUG "metric": { 06:58:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG [...] 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 1712815102, 06:58:22 DEBUG "1362.3" 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG } 06:58:22 INFO [http_cmd]: http status code OK 06:58:22 DEBUG --- status code --- 06:58:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:58:22 DEBUG --- http response --- 06:58:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:58:22 DEBUG ----- output ----- 06:58:22 DEBUG { 06:58:22 DEBUG "status": "success", 06:58:22 DEBUG "data": { 06:58:22 DEBUG "resultType": "vector", 06:58:22 DEBUG "result": [ 06:58:22 DEBUG { 06:58:22 DEBUG "metric": { 06:58:22 DEBUG "instance": "10.142.0.107:9100" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG [...] 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG 06:58:22 DEBUG "instance": "10.142.0.77:9100" 06:58:22 DEBUG }, 06:58:22 DEBUG "value": [ 06:58:22 DEBUG 1712815102, 06:58:22 DEBUG "0.0006666666666662973" 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG ] 06:58:22 DEBUG } 06:58:22 DEBUG } 06:58:34 INFO 06:58:34 INFO [loop_until]: kubectl --namespace=xlou top pods 06:58:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:58:34 INFO [loop_until]: OK (rc = 0) 06:58:34 DEBUG --- stdout --- 06:58:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 9117m 8507Mi am-5b5bd8567b-8h2w4 8713m 8818Mi am-5b5bd8567b-qlkw6 7351m 8460Mi ds-cts-0 8201m 12575Mi ds-cts-1 8456m 12685Mi ds-cts-2 9035m 12658Mi ds-idrepo-0 11371m 5503Mi ds-idrepo-1 360m 5480Mi ds-idrepo-2 352m 5784Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 3m 1011Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 4m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4103m 1869Mi 06:58:34 DEBUG --- stderr --- 06:58:34 DEBUG 06:58:38 INFO 06:58:38 INFO [loop_until]: kubectl --namespace=xlou top node 06:58:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:58:38 INFO [loop_until]: OK (rc = 0) 06:58:38 DEBUG --- stdout --- 06:58:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 2730Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 59m 0% 2781Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7808m 26% 10108Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10715m 35% 2988Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9583m 32% 10100Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 131m 0% 3850Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9568m 32% 10446Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 407m 2% 6467Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11525m 72% 6201Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8928m 56% 13277Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 8328m 52% 13214Mi 22% gke-xlou-cdm-ds-32e4dcb1-j2hk 8624m 54% 13456Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 403m 2% 6173Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4219m 26% 5620Mi 9% 06:58:38 DEBUG --- stderr --- 06:58:38 DEBUG 06:59:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815162" 06:59:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815162" 06:59:22 INFO [http_cmd]: http status code OK 06:59:22 DEBUG --- status code --- 06:59:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:22 DEBUG --- http response --- 06:59:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:59:22 DEBUG ----- output ----- 06:59:22 DEBUG { 06:59:22 DEBUG "status": "success", 06:59:22 DEBUG "data": { 06:59:22 DEBUG "resultType": "vector", 06:59:22 DEBUG "result": [ 06:59:22 DEBUG { 06:59:22 DEBUG "metric": { 06:59:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG [...] 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 1712815162, 06:59:22 DEBUG "244754" 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG } 06:59:22 INFO [http_cmd]: http status code OK 06:59:22 DEBUG --- status code --- 06:59:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:22 DEBUG --- http response --- 06:59:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:59:22 DEBUG ----- output ----- 06:59:22 DEBUG { 06:59:22 DEBUG "status": "success", 06:59:22 DEBUG "data": { 06:59:22 DEBUG "resultType": "vector", 06:59:22 DEBUG "result": [ 06:59:22 DEBUG { 06:59:22 DEBUG "metric": { 06:59:22 DEBUG "pod": "ds-idrepo-2" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG [...] 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG "pod": "ds-idrepo-0" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 1712815162, 06:59:22 DEBUG "0" 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG } 06:59:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712815162" 06:59:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815162" 06:59:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815162" 06:59:22 ERROR [http_cmd]: ERROR 06:59:22 ERROR --- status code --- 06:59:22 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:59:22 ERROR --- http response --- 06:59:22 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:59:22 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:59:22 INFO [http_cmd]: http status code OK 06:59:22 DEBUG --- status code --- 06:59:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:22 DEBUG --- http response --- 06:59:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:59:22 DEBUG ----- output ----- 06:59:22 DEBUG { 06:59:22 DEBUG "status": "success", 06:59:22 DEBUG "data": { 06:59:22 DEBUG "resultType": "vector", 06:59:22 DEBUG "result": [ 06:59:22 DEBUG { 06:59:22 DEBUG "metric": { 06:59:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG [...] 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 1712815162, 06:59:22 DEBUG "19281.668222844695" 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG } 06:59:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712815162" 06:59:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815162" 06:59:22 INFO [http_cmd]: http status code OK 06:59:22 DEBUG --- status code --- 06:59:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:22 DEBUG --- http response --- 06:59:22 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:59:22 INFO [http_cmd]: http status code OK 06:59:22 DEBUG --- status code --- 06:59:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:22 DEBUG --- http response --- 06:59:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:59:22 DEBUG ----- output ----- 06:59:22 DEBUG { 06:59:22 DEBUG "status": "success", 06:59:22 DEBUG "data": { 06:59:22 DEBUG "resultType": "vector", 06:59:22 DEBUG "result": [ 06:59:22 DEBUG { 06:59:22 DEBUG "metric": { 06:59:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG [...] 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 1712815162, 06:59:22 DEBUG "4.129490425809317" 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG } 06:59:22 INFO [http_cmd]: http status code OK 06:59:22 DEBUG --- status code --- 06:59:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:22 DEBUG --- http response --- 06:59:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:59:22 DEBUG ----- output ----- 06:59:22 DEBUG { 06:59:22 DEBUG "status": "success", 06:59:22 DEBUG "data": { 06:59:22 DEBUG "resultType": "vector", 06:59:22 DEBUG "result": [ 06:59:22 DEBUG { 06:59:22 DEBUG "metric": { 06:59:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG [...] 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 1712815162, 06:59:22 DEBUG "0.021" 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG } 06:59:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815162" 06:59:22 INFO [http_cmd]: http status code OK 06:59:22 DEBUG --- status code --- 06:59:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:22 DEBUG --- http response --- 06:59:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:59:22 DEBUG ----- output ----- 06:59:22 DEBUG { 06:59:22 DEBUG "status": "success", 06:59:22 DEBUG "data": { 06:59:22 DEBUG "resultType": "vector", 06:59:22 DEBUG "result": [ 06:59:22 DEBUG { 06:59:22 DEBUG "metric": { 06:59:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG [...] 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 1712815162, 06:59:22 DEBUG "1363.0333333333333" 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG } 06:59:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815162" 06:59:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815162" 06:59:22 INFO [http_cmd]: http status code OK 06:59:22 DEBUG --- status code --- 06:59:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:22 DEBUG --- http response --- 06:59:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:59:22 DEBUG ----- output ----- 06:59:22 DEBUG { 06:59:22 DEBUG "status": "success", 06:59:22 DEBUG "data": { 06:59:22 DEBUG "resultType": "vector", 06:59:22 DEBUG "result": [ 06:59:22 DEBUG { 06:59:22 DEBUG "metric": { 06:59:22 DEBUG "instance": "10.142.0.107:9100" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG [...] 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG "instance": "10.142.0.77:9100" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 1712815162, 06:59:22 DEBUG "0.00196666666666682" 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG } 06:59:22 INFO [http_cmd]: http status code OK 06:59:22 DEBUG --- status code --- 06:59:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:22 DEBUG --- http response --- 06:59:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:59:22 DEBUG ----- output ----- 06:59:22 DEBUG { 06:59:22 DEBUG "status": "success", 06:59:22 DEBUG "data": { 06:59:22 DEBUG "resultType": "vector", 06:59:22 DEBUG "result": [ 06:59:22 DEBUG { 06:59:22 DEBUG "metric": { 06:59:22 DEBUG "instance": "10.142.0.107:9100" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG [...] 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG "instance": "10.142.0.77:9100" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 1712815162, 06:59:22 DEBUG "52974423040" 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG } 06:59:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815162" 06:59:22 INFO [http_cmd]: http status code OK 06:59:22 DEBUG --- status code --- 06:59:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:22 DEBUG --- http response --- 06:59:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:59:22 DEBUG ----- output ----- 06:59:22 DEBUG { 06:59:22 DEBUG "status": "success", 06:59:22 DEBUG "data": { 06:59:22 DEBUG "resultType": "vector", 06:59:22 DEBUG "result": [ 06:59:22 DEBUG { 06:59:22 DEBUG "metric": { 06:59:22 DEBUG "instance": "10.142.0.107:9100" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG [...] 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG "instance": "10.142.0.77:9100" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 1712815162, 06:59:22 DEBUG "81392460.9" 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG } 06:59:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712815162" 06:59:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815162" 06:59:22 INFO [http_cmd]: http status code OK 06:59:22 DEBUG --- status code --- 06:59:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:22 DEBUG --- http response --- 06:59:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:59:22 DEBUG ----- output ----- 06:59:22 DEBUG { 06:59:22 DEBUG "status": "success", 06:59:22 DEBUG "data": { 06:59:22 DEBUG "resultType": "vector", 06:59:22 DEBUG "result": [ 06:59:22 DEBUG { 06:59:22 DEBUG "metric": { 06:59:22 DEBUG "pod": "ds-idrepo-2" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG [...] 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG "pod": "ds-cts-2" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 1712815162, 06:59:22 DEBUG "0" 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG } 06:59:22 INFO [http_cmd]: http status code OK 06:59:22 DEBUG --- status code --- 06:59:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:22 DEBUG --- http response --- 06:59:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:59:22 DEBUG ----- output ----- 06:59:22 DEBUG { 06:59:22 DEBUG "status": "success", 06:59:22 DEBUG "data": { 06:59:22 DEBUG "resultType": "vector", 06:59:22 DEBUG "result": [ 06:59:22 DEBUG { 06:59:22 DEBUG "metric": { 06:59:22 DEBUG "instance": "10.142.0.107:9100" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG [...] 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG "instance": "10.142.0.77:9100" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 1712815162, 06:59:22 DEBUG "81390181.8" 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG } 06:59:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815162" 06:59:22 INFO [http_cmd]: http status code OK 06:59:22 DEBUG --- status code --- 06:59:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:22 DEBUG --- http response --- 06:59:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:59:22 DEBUG ----- output ----- 06:59:22 DEBUG { 06:59:22 DEBUG "status": "success", 06:59:22 DEBUG "data": { 06:59:22 DEBUG "resultType": "vector", 06:59:22 DEBUG "result": [ 06:59:22 DEBUG { 06:59:22 DEBUG "metric": { 06:59:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG [...] 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 1712815162, 06:59:22 DEBUG "1368.9666666666667" 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG } 06:59:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815162" 06:59:22 INFO [http_cmd]: http status code OK 06:59:22 DEBUG --- status code --- 06:59:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:22 DEBUG --- http response --- 06:59:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:59:22 DEBUG ----- output ----- 06:59:22 DEBUG { 06:59:22 DEBUG "status": "success", 06:59:22 DEBUG "data": { 06:59:22 DEBUG "resultType": "vector", 06:59:22 DEBUG "result": [ 06:59:22 DEBUG { 06:59:22 DEBUG "metric": { 06:59:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG [...] 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 1712815162, 06:59:22 DEBUG "0" 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG } 06:59:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815162" 06:59:22 INFO [http_cmd]: http status code OK 06:59:22 DEBUG --- status code --- 06:59:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:22 DEBUG --- http response --- 06:59:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:59:22 DEBUG ----- output ----- 06:59:22 DEBUG { 06:59:22 DEBUG "status": "success", 06:59:22 DEBUG "data": { 06:59:22 DEBUG "resultType": "vector", 06:59:22 DEBUG "result": [ 06:59:22 DEBUG { 06:59:22 DEBUG "metric": { 06:59:22 DEBUG "pod": "ds-idrepo-2" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG [...] 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG "pod": "ds-idrepo-0" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 1712815162, 06:59:22 DEBUG "0" 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG } 06:59:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712815162" 06:59:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815162" 06:59:22 INFO [http_cmd]: http status code OK 06:59:22 DEBUG --- status code --- 06:59:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:22 DEBUG --- http response --- 06:59:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:59:22 DEBUG ----- output ----- 06:59:22 DEBUG { 06:59:22 DEBUG "status": "success", 06:59:22 DEBUG "data": { 06:59:22 DEBUG "resultType": "vector", 06:59:22 DEBUG "result": [ 06:59:22 DEBUG { 06:59:22 DEBUG "metric": { 06:59:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG [...] 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 1712815162, 06:59:22 DEBUG "4.153704248361917" 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG } 06:59:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815162" 06:59:22 INFO [http_cmd]: http status code OK 06:59:22 DEBUG --- status code --- 06:59:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:22 DEBUG --- http response --- 06:59:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:59:22 DEBUG ----- output ----- 06:59:22 DEBUG { 06:59:22 DEBUG "status": "success", 06:59:22 DEBUG "data": { 06:59:22 DEBUG "resultType": "vector", 06:59:22 DEBUG "result": [ 06:59:22 DEBUG { 06:59:22 DEBUG "metric": { 06:59:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG [...] 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 1712815162, 06:59:22 DEBUG "1369.0666666666666" 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG } 06:59:22 INFO [http_cmd]: http status code OK 06:59:22 DEBUG --- status code --- 06:59:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:22 DEBUG --- http response --- 06:59:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:59:22 DEBUG ----- output ----- 06:59:22 DEBUG { 06:59:22 DEBUG "status": "success", 06:59:22 DEBUG "data": { 06:59:22 DEBUG "resultType": "vector", 06:59:22 DEBUG "result": [ 06:59:22 DEBUG { 06:59:22 DEBUG "metric": { 06:59:22 DEBUG "pod": "ds-idrepo-2" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG [...] 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG "pod": "ds-cts-2" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 1712815162, 06:59:22 DEBUG "0" 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG } 06:59:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712815162" 06:59:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815162" 06:59:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815162" 06:59:22 INFO [http_cmd]: http status code OK 06:59:22 DEBUG --- status code --- 06:59:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:22 DEBUG --- http response --- 06:59:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:59:22 DEBUG ----- output ----- 06:59:22 DEBUG { 06:59:22 DEBUG "status": "success", 06:59:22 DEBUG "data": { 06:59:22 DEBUG "resultType": "vector", 06:59:22 DEBUG "result": [ 06:59:22 DEBUG { 06:59:22 DEBUG "metric": { 06:59:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG [...] 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 1712815162, 06:59:22 DEBUG "13719953408" 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG } 06:59:22 INFO [http_cmd]: http status code OK 06:59:22 DEBUG --- status code --- 06:59:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:22 DEBUG --- http response --- 06:59:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:59:22 DEBUG ----- output ----- 06:59:22 DEBUG { 06:59:22 DEBUG "status": "success", 06:59:22 DEBUG "data": { 06:59:22 DEBUG "resultType": "vector", 06:59:22 DEBUG "result": [ 06:59:22 DEBUG { 06:59:22 DEBUG "metric": { 06:59:22 DEBUG "pod": "ds-idrepo-2" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG [...] 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG "pod": "ds-cts-2" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 1712815162, 06:59:22 DEBUG "29.53333333333333" 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG } 06:59:22 INFO [http_cmd]: http status code OK 06:59:22 DEBUG --- status code --- 06:59:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:22 DEBUG --- http response --- 06:59:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:59:22 DEBUG ----- output ----- 06:59:22 DEBUG { 06:59:22 DEBUG "status": "success", 06:59:22 DEBUG "data": { 06:59:22 DEBUG "resultType": "vector", 06:59:22 DEBUG "result": [ 06:59:22 DEBUG { 06:59:22 DEBUG "metric": { 06:59:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG [...] 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 1712815162, 06:59:22 DEBUG "0" 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG } 06:59:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815162" 06:59:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815162" 06:59:22 INFO [http_cmd]: http status code OK 06:59:22 DEBUG --- status code --- 06:59:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:22 DEBUG --- http response --- 06:59:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:59:22 DEBUG ----- output ----- 06:59:22 DEBUG { 06:59:22 DEBUG "status": "success", 06:59:22 DEBUG "data": { 06:59:22 DEBUG "resultType": "vector", 06:59:22 DEBUG "result": [ 06:59:22 DEBUG { 06:59:22 DEBUG "metric": { 06:59:22 DEBUG "pod": "ds-idrepo-2" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG [...] 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG "pod": "ds-idrepo-0" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 1712815162, 06:59:22 DEBUG "0" 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG } 06:59:22 INFO [http_cmd]: http status code OK 06:59:22 DEBUG --- status code --- 06:59:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:22 DEBUG --- http response --- 06:59:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 06:59:22 DEBUG ----- output ----- 06:59:22 DEBUG { 06:59:22 DEBUG "status": "success", 06:59:22 DEBUG "data": { 06:59:22 DEBUG "resultType": "vector", 06:59:22 DEBUG "result": [ 06:59:22 DEBUG { 06:59:22 DEBUG "metric": { 06:59:22 DEBUG "pod": "ds-idrepo-2" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG [...] 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG "pod": "ds-cts-2" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 1712815162, 06:59:22 DEBUG "0.00009474333333333333" 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG } 06:59:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815162" 06:59:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815162" 06:59:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815162" 06:59:22 INFO [http_cmd]: http status code OK 06:59:22 DEBUG --- status code --- 06:59:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:22 DEBUG --- http response --- 06:59:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:59:22 DEBUG ----- output ----- 06:59:22 DEBUG { 06:59:22 DEBUG "status": "success", 06:59:22 DEBUG "data": { 06:59:22 DEBUG "resultType": "vector", 06:59:22 DEBUG "result": [ 06:59:22 DEBUG { 06:59:22 DEBUG "metric": { 06:59:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG [...] 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 1712815162, 06:59:22 DEBUG "0" 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG } 06:59:22 INFO [http_cmd]: http status code OK 06:59:22 DEBUG --- status code --- 06:59:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:22 DEBUG --- http response --- 06:59:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:59:22 DEBUG ----- output ----- 06:59:22 DEBUG { 06:59:22 DEBUG "status": "success", 06:59:22 DEBUG "data": { 06:59:22 DEBUG "resultType": "vector", 06:59:22 DEBUG "result": [ 06:59:22 DEBUG { 06:59:22 DEBUG "metric": { 06:59:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG [...] 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 1712815162, 06:59:22 DEBUG "1366.6666666666667" 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG } 06:59:22 INFO [http_cmd]: http status code OK 06:59:22 DEBUG --- status code --- 06:59:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:59:22 DEBUG --- http response --- 06:59:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:59:22 DEBUG ----- output ----- 06:59:22 DEBUG { 06:59:22 DEBUG "status": "success", 06:59:22 DEBUG "data": { 06:59:22 DEBUG "resultType": "vector", 06:59:22 DEBUG "result": [ 06:59:22 DEBUG { 06:59:22 DEBUG "metric": { 06:59:22 DEBUG "instance": "10.142.0.107:9100" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG [...] 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG 06:59:22 DEBUG "instance": "10.142.0.77:9100" 06:59:22 DEBUG }, 06:59:22 DEBUG "value": [ 06:59:22 DEBUG 1712815162, 06:59:22 DEBUG "0" 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG ] 06:59:22 DEBUG } 06:59:22 DEBUG } 06:59:34 INFO 06:59:34 INFO [loop_until]: kubectl --namespace=xlou top pods 06:59:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:59:34 INFO [loop_until]: OK (rc = 0) 06:59:34 DEBUG --- stdout --- 06:59:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 7711m 8508Mi am-5b5bd8567b-8h2w4 7068m 8817Mi am-5b5bd8567b-qlkw6 7460m 8459Mi ds-cts-0 8375m 13097Mi ds-cts-1 7870m 13399Mi ds-cts-2 8897m 13273Mi ds-idrepo-0 9161m 5504Mi ds-idrepo-1 379m 5480Mi ds-idrepo-2 297m 5784Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 4m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 1m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4114m 1869Mi 06:59:34 DEBUG --- stderr --- 06:59:34 DEBUG 06:59:38 INFO 06:59:38 INFO [loop_until]: kubectl --namespace=xlou top node 06:59:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:59:38 INFO [loop_until]: OK (rc = 0) 06:59:38 DEBUG --- stdout --- 06:59:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 2719Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 62m 0% 2771Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7925m 26% 10111Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 8673m 29% 2981Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9694m 32% 10089Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 146m 0% 3837Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9631m 32% 10435Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 344m 2% 6472Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 9736m 61% 6207Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8885m 55% 13888Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 7579m 47% 13793Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 7821m 49% 14040Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 424m 2% 6164Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3787m 23% 4912Mi 8% 06:59:38 DEBUG --- stderr --- 06:59:38 DEBUG 07:00:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815222" 07:00:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815222" 07:00:22 INFO [http_cmd]: http status code OK 07:00:22 DEBUG --- status code --- 07:00:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:22 DEBUG --- http response --- 07:00:22 INFO [http_cmd]: http status code OK 07:00:22 DEBUG --- status code --- 07:00:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:00:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:22 DEBUG ----- output ----- 07:00:22 DEBUG --- http response --- 07:00:22 DEBUG { 07:00:22 DEBUG "status": "success", 07:00:22 DEBUG "data": { 07:00:22 DEBUG "resultType": "vector", 07:00:22 DEBUG "result": [ 07:00:22 DEBUG { 07:00:22 DEBUG "metric": { 07:00:22 DEBUG "pod": "ds-idrepo-2" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG [...] 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG "pod": "ds-idrepo-0" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 1712815222, 07:00:22 DEBUG "0" 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG } 07:00:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:00:22 DEBUG ----- output ----- 07:00:22 DEBUG { 07:00:22 DEBUG "status": "success", 07:00:22 DEBUG "data": { 07:00:22 DEBUG "resultType": "vector", 07:00:22 DEBUG "result": [ 07:00:22 DEBUG { 07:00:22 DEBUG "metric": { 07:00:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG [...] 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 1712815222, 07:00:22 DEBUG "187156.5" 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG } 07:00:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712815222" 07:00:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815222" 07:00:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815222" 07:00:22 ERROR [http_cmd]: ERROR 07:00:22 ERROR --- status code --- 07:00:22 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:00:22 ERROR --- http response --- 07:00:22 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:00:22 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:00:22 INFO [http_cmd]: http status code OK 07:00:22 DEBUG --- status code --- 07:00:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:22 DEBUG --- http response --- 07:00:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 07:00:22 DEBUG ----- output ----- 07:00:22 DEBUG { 07:00:22 DEBUG "status": "success", 07:00:22 DEBUG "data": { 07:00:22 DEBUG "resultType": "vector", 07:00:22 DEBUG "result": [ 07:00:22 DEBUG { 07:00:22 DEBUG "metric": { 07:00:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG [...] 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG "pod": "ds-idrepo-1" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 1712815222, 07:00:22 DEBUG "0" 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG ] 07:00:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815222" 07:00:22 DEBUG } 07:00:22 DEBUG } 07:00:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712815222" 07:00:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815222" 07:00:22 INFO [http_cmd]: http status code OK 07:00:22 DEBUG --- status code --- 07:00:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:22 DEBUG --- http response --- 07:00:22 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:00:22 INFO [http_cmd]: http status code OK 07:00:22 DEBUG --- status code --- 07:00:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:22 DEBUG --- http response --- 07:00:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 07:00:22 DEBUG ----- output ----- 07:00:22 DEBUG { 07:00:22 DEBUG "status": "success", 07:00:22 DEBUG "data": { 07:00:22 DEBUG "resultType": "vector", 07:00:22 DEBUG "result": [ 07:00:22 DEBUG { 07:00:22 DEBUG "metric": { 07:00:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG [...] 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG "pod": "ds-idrepo-1" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 1712815222, 07:00:22 DEBUG "0" 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG } 07:00:22 INFO [http_cmd]: http status code OK 07:00:22 DEBUG --- status code --- 07:00:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:22 DEBUG --- http response --- 07:00:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:00:22 DEBUG ----- output ----- 07:00:22 DEBUG { 07:00:22 DEBUG "status": "success", 07:00:22 DEBUG "data": { 07:00:22 DEBUG "resultType": "vector", 07:00:22 DEBUG "result": [ 07:00:22 DEBUG { 07:00:22 DEBUG "metric": { 07:00:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG [...] 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 1712815222, 07:00:22 DEBUG "1371.8333333333333" 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG } 07:00:22 INFO [http_cmd]: http status code OK 07:00:22 DEBUG --- status code --- 07:00:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:22 DEBUG --- http response --- 07:00:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:00:22 DEBUG ----- output ----- 07:00:22 DEBUG { 07:00:22 DEBUG "status": "success", 07:00:22 DEBUG "data": { 07:00:22 DEBUG "resultType": "vector", 07:00:22 DEBUG "result": [ 07:00:22 DEBUG { 07:00:22 DEBUG "metric": { 07:00:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG [...] 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 1712815222, 07:00:22 DEBUG "0.021" 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG } 07:00:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815222" 07:00:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815222" 07:00:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815222" 07:00:22 INFO [http_cmd]: http status code OK 07:00:22 DEBUG --- status code --- 07:00:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:22 DEBUG --- http response --- 07:00:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:00:22 DEBUG ----- output ----- 07:00:22 DEBUG { 07:00:22 DEBUG "status": "success", 07:00:22 DEBUG "data": { 07:00:22 DEBUG "resultType": "vector", 07:00:22 DEBUG "result": [ 07:00:22 DEBUG { 07:00:22 DEBUG "metric": { 07:00:22 DEBUG "instance": "10.142.0.107:9100" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG [...] 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG "instance": "10.142.0.77:9100" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 1712815222, 07:00:22 DEBUG "0.002066666666666824" 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG } 07:00:22 INFO [http_cmd]: http status code OK 07:00:22 DEBUG --- status code --- 07:00:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:22 DEBUG --- http response --- 07:00:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:00:22 DEBUG ----- output ----- 07:00:22 DEBUG { 07:00:22 DEBUG "status": "success", 07:00:22 DEBUG "data": { 07:00:22 DEBUG "resultType": "vector", 07:00:22 DEBUG "result": [ 07:00:22 DEBUG { 07:00:22 DEBUG "metric": { 07:00:22 DEBUG "instance": "10.142.0.107:9100" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG [...] 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG "instance": "10.142.0.77:9100" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 1712815222, 07:00:22 DEBUG "52975731712" 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG } 07:00:22 INFO [http_cmd]: http status code OK 07:00:22 DEBUG --- status code --- 07:00:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:22 DEBUG --- http response --- 07:00:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:00:22 DEBUG ----- output ----- 07:00:22 DEBUG { 07:00:22 DEBUG "status": "success", 07:00:22 DEBUG "data": { 07:00:22 DEBUG "resultType": "vector", 07:00:22 DEBUG "result": [ 07:00:22 DEBUG { 07:00:22 DEBUG "metric": { 07:00:22 DEBUG "instance": "10.142.0.107:9100" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG [...] 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG "instance": "10.142.0.77:9100" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 1712815222, 07:00:22 DEBUG "11971681.466666665" 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG } 07:00:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815222" 07:00:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712815222" 07:00:22 INFO [http_cmd]: http status code OK 07:00:22 DEBUG --- status code --- 07:00:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:22 DEBUG --- http response --- 07:00:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:00:22 DEBUG ----- output ----- 07:00:22 DEBUG { 07:00:22 DEBUG "status": "success", 07:00:22 DEBUG "data": { 07:00:22 DEBUG "resultType": "vector", 07:00:22 DEBUG "result": [ 07:00:22 DEBUG { 07:00:22 DEBUG "metric": { 07:00:22 DEBUG "instance": "10.142.0.107:9100" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG [...] 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG "instance": "10.142.0.77:9100" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 1712815222, 07:00:22 DEBUG "11972684" 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG } 07:00:22 INFO [http_cmd]: http status code OK 07:00:22 DEBUG --- status code --- 07:00:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:22 DEBUG --- http response --- 07:00:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:00:22 DEBUG ----- output ----- 07:00:22 DEBUG { 07:00:22 DEBUG "status": "success", 07:00:22 DEBUG "data": { 07:00:22 DEBUG "resultType": "vector", 07:00:22 DEBUG "result": [ 07:00:22 DEBUG { 07:00:22 DEBUG "metric": { 07:00:22 DEBUG "pod": "ds-idrepo-2" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG [...] 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG "pod": "ds-cts-2" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 1712815222, 07:00:22 DEBUG "0" 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG } 07:00:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815222" 07:00:22 INFO [http_cmd]: http status code OK 07:00:22 DEBUG --- status code --- 07:00:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:22 DEBUG --- http response --- 07:00:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:00:22 DEBUG ----- output ----- 07:00:22 DEBUG { 07:00:22 DEBUG "status": "success", 07:00:22 DEBUG "data": { 07:00:22 DEBUG "resultType": "vector", 07:00:22 DEBUG "result": [ 07:00:22 DEBUG { 07:00:22 DEBUG "metric": { 07:00:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG [...] 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 1712815222, 07:00:22 DEBUG "0" 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG } 07:00:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815222" 07:00:22 INFO [http_cmd]: http status code OK 07:00:22 DEBUG --- status code --- 07:00:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:22 DEBUG --- http response --- 07:00:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:00:22 DEBUG ----- output ----- 07:00:22 DEBUG { 07:00:22 DEBUG "status": "success", 07:00:22 DEBUG "data": { 07:00:22 DEBUG "resultType": "vector", 07:00:22 DEBUG "result": [ 07:00:22 DEBUG { 07:00:22 DEBUG "metric": { 07:00:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG [...] 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG "pod": "ds-cts-0" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 1712815222, 07:00:22 DEBUG "17841307.95427844" 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG } 07:00:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815222" 07:00:22 INFO [http_cmd]: http status code OK 07:00:22 DEBUG --- status code --- 07:00:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:22 DEBUG --- http response --- 07:00:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:00:22 DEBUG ----- output ----- 07:00:22 DEBUG { 07:00:22 DEBUG "status": "success", 07:00:22 DEBUG "data": { 07:00:22 DEBUG "resultType": "vector", 07:00:22 DEBUG "result": [ 07:00:22 DEBUG { 07:00:22 DEBUG "metric": { 07:00:22 DEBUG "pod": "ds-idrepo-2" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG [...] 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG "pod": "ds-idrepo-0" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 1712815222, 07:00:22 DEBUG "0" 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG } 07:00:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712815222" 07:00:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815222" 07:00:22 INFO [http_cmd]: http status code OK 07:00:22 DEBUG --- status code --- 07:00:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:22 DEBUG --- http response --- 07:00:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:00:22 DEBUG ----- output ----- 07:00:22 DEBUG { 07:00:22 DEBUG "status": "success", 07:00:22 DEBUG "data": { 07:00:22 DEBUG "resultType": "vector", 07:00:22 DEBUG "result": [ 07:00:22 DEBUG { 07:00:22 DEBUG "metric": { 07:00:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG [...] 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 1712815222, 07:00:22 DEBUG "0.02868872831511292" 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG } 07:00:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815222" 07:00:22 INFO [http_cmd]: http status code OK 07:00:22 DEBUG --- status code --- 07:00:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:22 DEBUG --- http response --- 07:00:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:00:22 DEBUG ----- output ----- 07:00:22 DEBUG { 07:00:22 DEBUG "status": "success", 07:00:22 DEBUG "data": { 07:00:22 DEBUG "resultType": "vector", 07:00:22 DEBUG "result": [ 07:00:22 DEBUG { 07:00:22 DEBUG "metric": { 07:00:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG [...] 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 1712815222, 07:00:22 DEBUG "0" 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG } 07:00:22 INFO [http_cmd]: http status code OK 07:00:22 DEBUG --- status code --- 07:00:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:22 DEBUG --- http response --- 07:00:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:00:22 DEBUG ----- output ----- 07:00:22 DEBUG { 07:00:22 DEBUG "status": "success", 07:00:22 DEBUG "data": { 07:00:22 DEBUG "resultType": "vector", 07:00:22 DEBUG "result": [ 07:00:22 DEBUG { 07:00:22 DEBUG "metric": { 07:00:22 DEBUG "pod": "ds-idrepo-2" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG [...] 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG "pod": "ds-cts-2" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 1712815222, 07:00:22 DEBUG "0" 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG } 07:00:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712815222" 07:00:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815222" 07:00:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815222" 07:00:22 INFO [http_cmd]: http status code OK 07:00:22 DEBUG --- status code --- 07:00:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:22 DEBUG --- http response --- 07:00:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:00:22 DEBUG ----- output ----- 07:00:22 DEBUG { 07:00:22 DEBUG "status": "success", 07:00:22 DEBUG "data": { 07:00:22 DEBUG "resultType": "vector", 07:00:22 DEBUG "result": [ 07:00:22 DEBUG { 07:00:22 DEBUG "metric": { 07:00:22 DEBUG "pod": "ds-idrepo-2" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG [...] 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG "pod": "ds-cts-2" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 1712815222, 07:00:22 DEBUG "29.2" 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG } 07:00:22 INFO [http_cmd]: http status code OK 07:00:22 DEBUG --- status code --- 07:00:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:22 DEBUG --- http response --- 07:00:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:00:22 DEBUG ----- output ----- 07:00:22 DEBUG { 07:00:22 DEBUG "status": "success", 07:00:22 DEBUG "data": { 07:00:22 DEBUG "resultType": "vector", 07:00:22 DEBUG "result": [ 07:00:22 DEBUG { 07:00:22 DEBUG "metric": { 07:00:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG [...] 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 1712815222, 07:00:22 DEBUG "13944352768" 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG } 07:00:22 INFO [http_cmd]: http status code OK 07:00:22 DEBUG --- status code --- 07:00:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:22 DEBUG --- http response --- 07:00:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:00:22 DEBUG ----- output ----- 07:00:22 DEBUG { 07:00:22 DEBUG "status": "success", 07:00:22 DEBUG "data": { 07:00:22 DEBUG "resultType": "vector", 07:00:22 DEBUG "result": [ 07:00:22 DEBUG { 07:00:22 DEBUG "metric": { 07:00:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG [...] 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG "pod": "ds-cts-0" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 1712815222, 07:00:22 DEBUG "36765954.949885696" 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG } 07:00:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815222" 07:00:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815222" 07:00:22 INFO [http_cmd]: http status code OK 07:00:22 DEBUG --- status code --- 07:00:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:22 DEBUG --- http response --- 07:00:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:00:22 DEBUG ----- output ----- 07:00:22 DEBUG { 07:00:22 DEBUG "status": "success", 07:00:22 DEBUG "data": { 07:00:22 DEBUG "resultType": "vector", 07:00:22 DEBUG "result": [ 07:00:22 DEBUG { 07:00:22 DEBUG "metric": { 07:00:22 DEBUG "pod": "ds-idrepo-2" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG [...] 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG "pod": "ds-idrepo-0" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 1712815222, 07:00:22 DEBUG "0" 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG } 07:00:22 INFO [http_cmd]: http status code OK 07:00:22 DEBUG --- status code --- 07:00:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:22 DEBUG --- http response --- 07:00:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:00:22 DEBUG ----- output ----- 07:00:22 DEBUG { 07:00:22 DEBUG "status": "success", 07:00:22 DEBUG "data": { 07:00:22 DEBUG "resultType": "vector", 07:00:22 DEBUG "result": [ 07:00:22 DEBUG { 07:00:22 DEBUG "metric": { 07:00:22 DEBUG "pod": "ds-idrepo-2" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG [...] 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG "pod": "ds-cts-2" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 1712815222, 07:00:22 DEBUG "0.0002666666666666667" 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG } 07:00:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815222" 07:00:22 INFO [http_cmd]: http status code OK 07:00:22 DEBUG --- status code --- 07:00:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:22 DEBUG --- http response --- 07:00:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:00:22 DEBUG ----- output ----- 07:00:22 DEBUG { 07:00:22 DEBUG "status": "success", 07:00:22 DEBUG "data": { 07:00:22 DEBUG "resultType": "vector", 07:00:22 DEBUG "result": [ 07:00:22 DEBUG { 07:00:22 DEBUG "metric": { 07:00:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG [...] 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 1712815222, 07:00:22 DEBUG "0" 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG } 07:00:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815222" 07:00:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815222" 07:00:22 INFO [http_cmd]: http status code OK 07:00:22 DEBUG --- status code --- 07:00:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:22 DEBUG --- http response --- 07:00:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:00:22 DEBUG ----- output ----- 07:00:22 DEBUG { 07:00:22 DEBUG "status": "success", 07:00:22 DEBUG "data": { 07:00:22 DEBUG "resultType": "vector", 07:00:22 DEBUG "result": [ 07:00:22 DEBUG { 07:00:22 DEBUG "metric": { 07:00:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG [...] 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 1712815222, 07:00:22 DEBUG "0" 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG } 07:00:22 INFO [http_cmd]: http status code OK 07:00:22 DEBUG --- status code --- 07:00:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:00:22 DEBUG --- http response --- 07:00:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:00:22 DEBUG ----- output ----- 07:00:22 DEBUG { 07:00:22 DEBUG "status": "success", 07:00:22 DEBUG "data": { 07:00:22 DEBUG "resultType": "vector", 07:00:22 DEBUG "result": [ 07:00:22 DEBUG { 07:00:22 DEBUG "metric": { 07:00:22 DEBUG "instance": "10.142.0.107:9100" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG [...] 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG 07:00:22 DEBUG "instance": "10.142.0.77:9100" 07:00:22 DEBUG }, 07:00:22 DEBUG "value": [ 07:00:22 DEBUG 1712815222, 07:00:22 DEBUG "0" 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG ] 07:00:22 DEBUG } 07:00:22 DEBUG } 07:00:34 INFO 07:00:34 INFO [loop_until]: kubectl --namespace=xlou top pods 07:00:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:00:34 INFO [loop_until]: OK (rc = 0) 07:00:34 DEBUG --- stdout --- 07:00:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 1453m 8506Mi am-5b5bd8567b-8h2w4 1495m 8761Mi am-5b5bd8567b-qlkw6 1326m 8409Mi ds-cts-0 3846m 13220Mi ds-cts-1 4105m 13195Mi ds-cts-2 4273m 13401Mi ds-idrepo-0 1377m 5503Mi ds-idrepo-1 11m 5480Mi ds-idrepo-2 11m 5784Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 4m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 2m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 1m 1153Mi 07:00:34 DEBUG --- stderr --- 07:00:34 DEBUG 07:00:38 INFO 07:00:38 INFO [loop_until]: kubectl --namespace=xlou top node 07:00:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:00:38 INFO [loop_until]: OK (rc = 0) 07:00:38 DEBUG --- stdout --- 07:00:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 2721Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 60m 0% 2780Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1412m 4% 10170Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 119m 0% 2971Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1666m 5% 10147Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 144m 0% 3853Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1588m 5% 10515Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 66m 0% 6467Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 1441m 9% 6205Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 4242m 26% 14009Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 4035m 25% 13856Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 4272m 26% 13849Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 69m 0% 6159Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 66m 0% 4914Mi 8% 07:00:38 DEBUG --- stderr --- 07:00:38 DEBUG 07:01:13 WARNING File /tmp/lodemon_data-20240411_022725/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [11/Apr/2024 07:01:13] "GET /monitoring/average?start_time=20240411_050028&stop_time=20240411_055912 HTTP/1.1" 200 - 07:01:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815282" 07:01:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815282" 07:01:22 INFO [http_cmd]: http status code OK 07:01:22 INFO [http_cmd]: http status code OK 07:01:22 DEBUG --- status code --- 07:01:22 DEBUG --- status code --- 07:01:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:22 DEBUG --- http response --- 07:01:22 DEBUG --- http response --- 07:01:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:01:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:01:22 DEBUG ----- output ----- 07:01:22 DEBUG ----- output ----- 07:01:22 DEBUG { 07:01:22 DEBUG { 07:01:22 DEBUG "status": "success", 07:01:22 DEBUG "status": "success", 07:01:22 DEBUG "data": { 07:01:22 DEBUG "data": { 07:01:22 DEBUG "resultType": "vector", 07:01:22 DEBUG "resultType": "vector", 07:01:22 DEBUG "result": [ 07:01:22 DEBUG "result": [ 07:01:22 DEBUG { 07:01:22 DEBUG { 07:01:22 DEBUG "metric": { 07:01:22 DEBUG "metric": { 07:01:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:01:22 DEBUG "pod": "ds-idrepo-2" 07:01:22 DEBUG }, 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG [...] 07:01:22 DEBUG [...] 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:01:22 DEBUG "pod": "ds-idrepo-0" 07:01:22 DEBUG }, 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 1712815282, 07:01:22 DEBUG 1712815282, 07:01:22 DEBUG "105538.5" 07:01:22 DEBUG "0" 07:01:22 DEBUG ] 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG } 07:01:22 DEBUG ] 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG } 07:01:22 DEBUG } 07:01:22 DEBUG } 07:01:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712815282" 07:01:22 INFO [http_cmd]: http status code OK 07:01:22 DEBUG --- status code --- 07:01:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:22 DEBUG --- http response --- 07:01:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:01:22 DEBUG ----- output ----- 07:01:22 DEBUG { 07:01:22 DEBUG "status": "success", 07:01:22 DEBUG "data": { 07:01:22 DEBUG "resultType": "vector", 07:01:22 DEBUG "result": [ 07:01:22 DEBUG { 07:01:22 DEBUG "metric": { 07:01:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG [...] 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 1712815282, 07:01:22 DEBUG "0.021" 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG } 07:01:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712815282" 07:01:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815282" 07:01:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815282" 07:01:22 ERROR [http_cmd]: ERROR 07:01:22 ERROR --- status code --- 07:01:22 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:01:22 ERROR --- http response --- 07:01:22 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:01:22 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:01:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815282" 07:01:22 INFO [http_cmd]: http status code OK 07:01:22 DEBUG --- status code --- 07:01:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:22 DEBUG --- http response --- 07:01:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 07:01:22 DEBUG ----- output ----- 07:01:22 DEBUG { 07:01:22 DEBUG "status": "success", 07:01:22 DEBUG "data": { 07:01:22 DEBUG "resultType": "vector", 07:01:22 DEBUG "result": [ 07:01:22 DEBUG { 07:01:22 DEBUG "metric": { 07:01:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG [...] 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 1712815282, 07:01:22 DEBUG "1830.3077696475452" 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG } 07:01:22 INFO [http_cmd]: http status code OK 07:01:22 DEBUG --- status code --- 07:01:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:22 DEBUG --- http response --- 07:01:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 07:01:22 DEBUG ----- output ----- 07:01:22 DEBUG { 07:01:22 DEBUG "status": "success", 07:01:22 DEBUG "data": { 07:01:22 DEBUG "resultType": "vector", 07:01:22 DEBUG "result": [ 07:01:22 DEBUG { 07:01:22 DEBUG "metric": { 07:01:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG [...] 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 1712815282, 07:01:22 DEBUG "0.0011647314695446816" 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG } 07:01:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815282" 07:01:22 INFO [http_cmd]: http status code OK 07:01:22 DEBUG --- status code --- 07:01:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:22 DEBUG --- http response --- 07:01:22 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:01:22 INFO [http_cmd]: http status code OK 07:01:22 DEBUG --- status code --- 07:01:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:22 DEBUG --- http response --- 07:01:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:01:22 DEBUG ----- output ----- 07:01:22 DEBUG { 07:01:22 DEBUG "status": "success", 07:01:22 DEBUG "data": { 07:01:22 DEBUG "resultType": "vector", 07:01:22 DEBUG "result": [ 07:01:22 DEBUG { 07:01:22 DEBUG "metric": { 07:01:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG [...] 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 1712815282, 07:01:22 DEBUG "1362.8333333333333" 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG } 07:01:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815282" 07:01:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815282" 07:01:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815282" 07:01:22 INFO [http_cmd]: http status code OK 07:01:22 DEBUG --- status code --- 07:01:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:22 DEBUG --- http response --- 07:01:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:01:22 DEBUG ----- output ----- 07:01:22 DEBUG { 07:01:22 DEBUG "status": "success", 07:01:22 DEBUG "data": { 07:01:22 DEBUG "resultType": "vector", 07:01:22 DEBUG "result": [ 07:01:22 DEBUG { 07:01:22 DEBUG "metric": { 07:01:22 DEBUG "instance": "10.142.0.107:9100" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG [...] 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG "instance": "10.142.0.77:9100" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 1712815282, 07:01:22 DEBUG "0.001966666666666583" 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG } 07:01:22 INFO [http_cmd]: http status code OK 07:01:22 DEBUG --- status code --- 07:01:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:22 DEBUG --- http response --- 07:01:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:01:22 DEBUG ----- output ----- 07:01:22 DEBUG { 07:01:22 DEBUG "status": "success", 07:01:22 DEBUG "data": { 07:01:22 DEBUG "resultType": "vector", 07:01:22 DEBUG "result": [ 07:01:22 DEBUG { 07:01:22 DEBUG "metric": { 07:01:22 DEBUG "instance": "10.142.0.107:9100" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG [...] 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG "instance": "10.142.0.77:9100" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 1712815282, 07:01:22 DEBUG "52977341440" 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG } 07:01:22 INFO [http_cmd]: http status code OK 07:01:22 DEBUG --- status code --- 07:01:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:22 DEBUG --- http response --- 07:01:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:01:22 DEBUG ----- output ----- 07:01:22 DEBUG { 07:01:22 DEBUG "status": "success", 07:01:22 DEBUG "data": { 07:01:22 DEBUG "resultType": "vector", 07:01:22 DEBUG "result": [ 07:01:22 DEBUG { 07:01:22 DEBUG "metric": { 07:01:22 DEBUG "instance": "10.142.0.107:9100" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG [...] 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG "instance": "10.142.0.77:9100" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 1712815282, 07:01:22 DEBUG "12020281.666666666" 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG } 07:01:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815282" 07:01:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712815282" 07:01:22 INFO [http_cmd]: http status code OK 07:01:22 DEBUG --- status code --- 07:01:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:22 DEBUG --- http response --- 07:01:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:01:22 DEBUG ----- output ----- 07:01:22 DEBUG { 07:01:22 DEBUG "status": "success", 07:01:22 DEBUG "data": { 07:01:22 DEBUG "resultType": "vector", 07:01:22 DEBUG "result": [ 07:01:22 DEBUG { 07:01:22 DEBUG "metric": { 07:01:22 DEBUG "instance": "10.142.0.107:9100" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG [...] 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG "instance": "10.142.0.77:9100" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 1712815282, 07:01:22 DEBUG "12021351.133333333" 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG } 07:01:22 INFO [http_cmd]: http status code OK 07:01:22 DEBUG --- status code --- 07:01:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:22 DEBUG --- http response --- 07:01:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:01:22 DEBUG ----- output ----- 07:01:22 DEBUG { 07:01:22 DEBUG "status": "success", 07:01:22 DEBUG "data": { 07:01:22 DEBUG "resultType": "vector", 07:01:22 DEBUG "result": [ 07:01:22 DEBUG { 07:01:22 DEBUG "metric": { 07:01:22 DEBUG "pod": "ds-idrepo-2" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG [...] 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG "pod": "ds-cts-2" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 1712815282, 07:01:22 DEBUG "0" 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG } 07:01:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815282" 07:01:22 INFO [http_cmd]: http status code OK 07:01:22 DEBUG --- status code --- 07:01:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:22 DEBUG --- http response --- 07:01:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:01:22 DEBUG ----- output ----- 07:01:22 DEBUG { 07:01:22 DEBUG "status": "success", 07:01:22 DEBUG "data": { 07:01:22 DEBUG "resultType": "vector", 07:01:22 DEBUG "result": [ 07:01:22 DEBUG { 07:01:22 DEBUG "metric": { 07:01:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG [...] 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 1712815282, 07:01:22 DEBUG "0" 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG } 07:01:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815282" 07:01:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712815282" 07:01:22 INFO [http_cmd]: http status code OK 07:01:22 DEBUG --- status code --- 07:01:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:22 DEBUG --- http response --- 07:01:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:01:22 DEBUG ----- output ----- 07:01:22 DEBUG { 07:01:22 DEBUG "status": "success", 07:01:22 DEBUG "data": { 07:01:22 DEBUG "resultType": "vector", 07:01:22 DEBUG "result": [ 07:01:22 DEBUG { 07:01:22 DEBUG "metric": { 07:01:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG [...] 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 1712815282, 07:01:22 DEBUG "0" 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG } 07:01:22 INFO [http_cmd]: http status code OK 07:01:22 DEBUG --- status code --- 07:01:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:22 DEBUG --- http response --- 07:01:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:01:22 DEBUG ----- output ----- 07:01:22 DEBUG { 07:01:22 DEBUG "status": "success", 07:01:22 DEBUG "data": { 07:01:22 DEBUG "resultType": "vector", 07:01:22 DEBUG "result": [ 07:01:22 DEBUG { 07:01:22 DEBUG "metric": { 07:01:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG [...] 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 1712815282, 07:01:22 DEBUG "0.026003925860647623" 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG } 07:01:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815282" 07:01:22 INFO [http_cmd]: http status code OK 07:01:22 DEBUG --- status code --- 07:01:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:22 DEBUG --- http response --- 07:01:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:01:22 DEBUG ----- output ----- 07:01:22 DEBUG { 07:01:22 DEBUG "status": "success", 07:01:22 DEBUG "data": { 07:01:22 DEBUG "resultType": "vector", 07:01:22 DEBUG "result": [ 07:01:22 DEBUG { 07:01:22 DEBUG "metric": { 07:01:22 DEBUG "pod": "ds-idrepo-2" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG [...] 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG "pod": "ds-idrepo-0" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 1712815282, 07:01:22 DEBUG "0" 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG } 07:01:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815282" 07:01:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815282" 07:01:22 INFO [http_cmd]: http status code OK 07:01:22 DEBUG --- status code --- 07:01:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:22 DEBUG --- http response --- 07:01:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:01:22 DEBUG ----- output ----- 07:01:22 DEBUG { 07:01:22 DEBUG "status": "success", 07:01:22 DEBUG "data": { 07:01:22 DEBUG "resultType": "vector", 07:01:22 DEBUG "result": [ 07:01:22 DEBUG { 07:01:22 DEBUG "metric": { 07:01:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG [...] 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 1712815282, 07:01:22 DEBUG "0" 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG } 07:01:22 INFO [http_cmd]: http status code OK 07:01:22 DEBUG --- status code --- 07:01:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:22 DEBUG --- http response --- 07:01:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:01:22 DEBUG ----- output ----- 07:01:22 DEBUG { 07:01:22 DEBUG "status": "success", 07:01:22 DEBUG "data": { 07:01:22 DEBUG "resultType": "vector", 07:01:22 DEBUG "result": [ 07:01:22 DEBUG { 07:01:22 DEBUG "metric": { 07:01:22 DEBUG "pod": "ds-idrepo-2" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG [...] 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG "pod": "ds-cts-2" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 1712815282, 07:01:22 DEBUG "0" 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG } 07:01:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815282" 07:01:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712815282" 07:01:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815282" 07:01:22 INFO [http_cmd]: http status code OK 07:01:22 DEBUG --- status code --- 07:01:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:22 DEBUG --- http response --- 07:01:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:01:22 DEBUG ----- output ----- 07:01:22 DEBUG { 07:01:22 DEBUG "status": "success", 07:01:22 DEBUG "data": { 07:01:22 DEBUG "resultType": "vector", 07:01:22 DEBUG "result": [ 07:01:22 DEBUG { 07:01:22 DEBUG "metric": { 07:01:22 DEBUG "pod": "ds-idrepo-2" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG [...] 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG "pod": "ds-cts-2" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 1712815282, 07:01:22 DEBUG "29.599999999999998" 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG } 07:01:22 INFO [http_cmd]: http status code OK 07:01:22 DEBUG --- status code --- 07:01:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:22 DEBUG --- http response --- 07:01:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:01:22 DEBUG ----- output ----- 07:01:22 DEBUG { 07:01:22 DEBUG "status": "success", 07:01:22 DEBUG "data": { 07:01:22 DEBUG "resultType": "vector", 07:01:22 DEBUG "result": [ 07:01:22 DEBUG { 07:01:22 DEBUG "metric": { 07:01:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG [...] 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 1712815282, 07:01:22 DEBUG "13981413376" 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG } 07:01:22 INFO [http_cmd]: http status code OK 07:01:22 DEBUG --- status code --- 07:01:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:22 DEBUG --- http response --- 07:01:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:01:22 DEBUG ----- output ----- 07:01:22 DEBUG { 07:01:22 DEBUG "status": "success", 07:01:22 DEBUG "data": { 07:01:22 DEBUG "resultType": "vector", 07:01:22 DEBUG "result": [ 07:01:22 DEBUG { 07:01:22 DEBUG "metric": { 07:01:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG [...] 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 1712815282, 07:01:22 DEBUG "0" 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG } 07:01:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815282" 07:01:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815282" 07:01:22 INFO [http_cmd]: http status code OK 07:01:22 DEBUG --- status code --- 07:01:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:22 DEBUG --- http response --- 07:01:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:01:22 DEBUG ----- output ----- 07:01:22 DEBUG { 07:01:22 DEBUG "status": "success", 07:01:22 DEBUG "data": { 07:01:22 DEBUG "resultType": "vector", 07:01:22 DEBUG "result": [ 07:01:22 DEBUG { 07:01:22 DEBUG "metric": { 07:01:22 DEBUG "pod": "ds-idrepo-2" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG [...] 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG "pod": "ds-idrepo-0" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 1712815282, 07:01:22 DEBUG "0" 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG } 07:01:22 INFO [http_cmd]: http status code OK 07:01:22 DEBUG --- status code --- 07:01:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:22 DEBUG --- http response --- 07:01:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:01:22 DEBUG ----- output ----- 07:01:22 DEBUG { 07:01:22 DEBUG "status": "success", 07:01:22 DEBUG "data": { 07:01:22 DEBUG "resultType": "vector", 07:01:22 DEBUG "result": [ 07:01:22 DEBUG { 07:01:22 DEBUG "metric": { 07:01:22 DEBUG "pod": "ds-idrepo-2" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG [...] 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG "pod": "ds-cts-2" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 1712815282, 07:01:22 DEBUG "0.000033333333333333335" 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG } 07:01:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815282" 07:01:22 INFO [http_cmd]: http status code OK 07:01:22 DEBUG --- status code --- 07:01:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:22 DEBUG --- http response --- 07:01:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:01:22 DEBUG ----- output ----- 07:01:22 DEBUG { 07:01:22 DEBUG "status": "success", 07:01:22 DEBUG "data": { 07:01:22 DEBUG "resultType": "vector", 07:01:22 DEBUG "result": [ 07:01:22 DEBUG { 07:01:22 DEBUG "metric": { 07:01:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG [...] 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 1712815282, 07:01:22 DEBUG "0" 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG } 07:01:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815282" 07:01:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815282" 07:01:22 INFO [http_cmd]: http status code OK 07:01:22 DEBUG --- status code --- 07:01:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:22 DEBUG --- http response --- 07:01:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:01:22 DEBUG ----- output ----- 07:01:22 DEBUG { 07:01:22 DEBUG "status": "success", 07:01:22 DEBUG "data": { 07:01:22 DEBUG "resultType": "vector", 07:01:22 DEBUG "result": [ 07:01:22 DEBUG { 07:01:22 DEBUG "metric": { 07:01:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG [...] 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 1712815282, 07:01:22 DEBUG "0" 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG } 07:01:22 INFO [http_cmd]: http status code OK 07:01:22 DEBUG --- status code --- 07:01:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:01:22 DEBUG --- http response --- 07:01:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:01:22 DEBUG ----- output ----- 07:01:22 DEBUG { 07:01:22 DEBUG "status": "success", 07:01:22 DEBUG "data": { 07:01:22 DEBUG "resultType": "vector", 07:01:22 DEBUG "result": [ 07:01:22 DEBUG { 07:01:22 DEBUG "metric": { 07:01:22 DEBUG "instance": "10.142.0.107:9100" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG [...] 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG 07:01:22 DEBUG "instance": "10.142.0.77:9100" 07:01:22 DEBUG }, 07:01:22 DEBUG "value": [ 07:01:22 DEBUG 1712815282, 07:01:22 DEBUG "0.0003333333333333854" 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG ] 07:01:22 DEBUG } 07:01:22 DEBUG } 07:01:34 INFO 07:01:34 INFO [loop_until]: kubectl --namespace=xlou top pods 07:01:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:01:34 INFO [loop_until]: OK (rc = 0) 07:01:34 DEBUG --- stdout --- 07:01:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 1513m 8428Mi am-5b5bd8567b-8h2w4 1495m 8746Mi am-5b5bd8567b-qlkw6 1337m 8404Mi ds-cts-0 3664m 12940Mi ds-cts-1 3940m 12626Mi ds-cts-2 3907m 12896Mi ds-idrepo-0 1319m 5504Mi ds-idrepo-1 10m 5479Mi ds-idrepo-2 66m 5784Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 4m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 4m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 3m 1153Mi 07:01:34 DEBUG --- stderr --- 07:01:34 DEBUG 07:01:38 INFO 07:01:38 INFO [loop_until]: kubectl --namespace=xlou top node 07:01:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:01:38 INFO [loop_until]: OK (rc = 0) 07:01:38 DEBUG --- stdout --- 07:01:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 2721Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 63m 0% 2778Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1381m 4% 10105Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 114m 0% 2975Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1661m 5% 10072Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 144m 0% 3838Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1615m 5% 10422Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 117m 0% 6466Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 1431m 9% 6210Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 3847m 24% 13732Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 3749m 23% 13588Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 3873m 24% 13298Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 66m 0% 6165Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 92m 0% 4910Mi 8% 07:01:38 DEBUG --- stderr --- 07:01:38 DEBUG 07:02:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815342" 07:02:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712815342" 07:02:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815342" 07:02:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712815342" 07:02:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815342" 07:02:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815342" 07:02:22 INFO [http_cmd]: http status code OK 07:02:22 INFO [http_cmd]: http status code OK 07:02:22 DEBUG --- status code --- 07:02:22 ERROR [http_cmd]: ERROR 07:02:22 INFO [http_cmd]: http status code OK 07:02:22 DEBUG --- status code --- 07:02:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:02:22 ERROR --- status code --- 07:02:22 DEBUG --- status code --- 07:02:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:02:22 DEBUG --- http response --- 07:02:22 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:02:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:02:22 DEBUG --- http response --- 07:02:22 ERROR --- http response --- 07:02:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:02:22 DEBUG --- http response --- 07:02:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:02:22 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:02:22 DEBUG ----- output ----- 07:02:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:02:22 DEBUG ----- output ----- 07:02:22 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:02:22 DEBUG { 07:02:22 DEBUG ----- output ----- 07:02:22 DEBUG { 07:02:22 DEBUG "status": "success", 07:02:22 DEBUG { 07:02:22 DEBUG "status": "success", 07:02:22 DEBUG "data": { 07:02:22 DEBUG "status": "success", 07:02:22 DEBUG "data": { 07:02:22 DEBUG "resultType": "vector", 07:02:22 DEBUG "data": { 07:02:22 DEBUG "resultType": "vector", 07:02:22 DEBUG "result": [ 07:02:22 DEBUG "resultType": "vector", 07:02:22 DEBUG "result": [ 07:02:22 DEBUG { 07:02:22 DEBUG "result": [ 07:02:22 DEBUG { 07:02:22 DEBUG "metric": { 07:02:22 DEBUG { 07:02:22 DEBUG "metric": { 07:02:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:02:22 DEBUG "metric": { 07:02:22 DEBUG "pod": "ds-idrepo-2" 07:02:22 DEBUG }, 07:02:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG [...] 07:02:22 DEBUG 07:02:22 DEBUG [...] 07:02:22 DEBUG 07:02:22 DEBUG [...] 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:02:22 DEBUG 07:02:22 DEBUG "pod": "ds-idrepo-0" 07:02:22 DEBUG }, 07:02:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 1712815342, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 1712815342, 07:02:22 DEBUG "0.022" 07:02:22 DEBUG 1712815342, 07:02:22 DEBUG "0" 07:02:22 DEBUG ] 07:02:22 DEBUG "45786" 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG } 07:02:22 DEBUG } 07:02:22 DEBUG } 07:02:22 DEBUG } 07:02:22 INFO [http_cmd]: http status code OK 07:02:22 INFO [http_cmd]: http status code OK 07:02:22 DEBUG --- status code --- 07:02:22 DEBUG --- status code --- 07:02:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:02:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:02:22 DEBUG --- http response --- 07:02:22 DEBUG --- http response --- 07:02:22 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:02:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 07:02:22 DEBUG ----- output ----- 07:02:22 DEBUG { 07:02:22 DEBUG "status": "success", 07:02:22 DEBUG "data": { 07:02:22 DEBUG "resultType": "vector", 07:02:22 DEBUG "result": [ 07:02:22 DEBUG { 07:02:22 DEBUG "metric": { 07:02:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG [...] 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 1712815342, 07:02:22 DEBUG "0" 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG } 07:02:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815342" 07:02:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815342" 07:02:22 INFO [http_cmd]: http status code OK 07:02:22 DEBUG --- status code --- 07:02:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:02:22 DEBUG --- http response --- 07:02:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:02:22 DEBUG ----- output ----- 07:02:22 DEBUG { 07:02:22 DEBUG "status": "success", 07:02:22 DEBUG "data": { 07:02:22 DEBUG "resultType": "vector", 07:02:22 DEBUG "result": [ 07:02:22 DEBUG { 07:02:22 DEBUG "metric": { 07:02:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG [...] 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 1712815342, 07:02:22 DEBUG "1364.8333333333333" 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG } 07:02:22 INFO [http_cmd]: http status code OK 07:02:22 DEBUG --- status code --- 07:02:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:02:22 DEBUG --- http response --- 07:02:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 07:02:22 DEBUG ----- output ----- 07:02:22 DEBUG { 07:02:22 DEBUG "status": "success", 07:02:22 DEBUG "data": { 07:02:22 DEBUG "resultType": "vector", 07:02:22 DEBUG "result": [ 07:02:22 DEBUG { 07:02:22 DEBUG "metric": { 07:02:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG [...] 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 1712815342, 07:02:22 DEBUG "0" 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG } 07:02:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815342" 07:02:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815342" 07:02:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815342" 07:02:22 INFO [http_cmd]: http status code OK 07:02:22 DEBUG --- status code --- 07:02:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:02:22 DEBUG --- http response --- 07:02:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:02:22 DEBUG ----- output ----- 07:02:22 DEBUG { 07:02:22 DEBUG "status": "success", 07:02:22 DEBUG "data": { 07:02:22 DEBUG "resultType": "vector", 07:02:22 DEBUG "result": [ 07:02:22 DEBUG { 07:02:22 DEBUG "metric": { 07:02:22 DEBUG "instance": "10.142.0.107:9100" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG [...] 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG "instance": "10.142.0.77:9100" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 1712815342, 07:02:22 DEBUG "0.0020000000000003123" 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG } 07:02:22 INFO [http_cmd]: http status code OK 07:02:22 DEBUG --- status code --- 07:02:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:02:22 DEBUG --- http response --- 07:02:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:02:22 DEBUG ----- output ----- 07:02:22 DEBUG { 07:02:22 DEBUG "status": "success", 07:02:22 INFO [http_cmd]: http status code OK 07:02:22 DEBUG "data": { 07:02:22 DEBUG --- status code --- 07:02:22 DEBUG "resultType": "vector", 07:02:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:02:22 DEBUG "result": [ 07:02:22 DEBUG --- http response --- 07:02:22 DEBUG { 07:02:22 DEBUG "metric": { 07:02:22 DEBUG "instance": "10.142.0.107:9100" 07:02:22 DEBUG }, 07:02:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:02:22 DEBUG "value": [ 07:02:22 DEBUG ----- output ----- 07:02:22 DEBUG 07:02:22 DEBUG { 07:02:22 DEBUG 07:02:22 DEBUG "status": "success", 07:02:22 DEBUG 07:02:22 DEBUG "data": { 07:02:22 DEBUG [...] 07:02:22 DEBUG "resultType": "vector", 07:02:22 DEBUG 07:02:22 DEBUG "result": [ 07:02:22 DEBUG 07:02:22 DEBUG { 07:02:22 DEBUG 07:02:22 DEBUG "metric": { 07:02:22 DEBUG "instance": "10.142.0.77:9100" 07:02:22 DEBUG "instance": "10.142.0.107:9100" 07:02:22 DEBUG }, 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 1712815342, 07:02:22 DEBUG 07:02:22 DEBUG "52978727936" 07:02:22 DEBUG 07:02:22 DEBUG ] 07:02:22 DEBUG 07:02:22 DEBUG } 07:02:22 DEBUG [...] 07:02:22 DEBUG ] 07:02:22 DEBUG 07:02:22 DEBUG } 07:02:22 DEBUG 07:02:22 DEBUG } 07:02:22 DEBUG 07:02:22 DEBUG "instance": "10.142.0.77:9100" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 1712815342, 07:02:22 DEBUG "80238245.76666667" 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG } 07:02:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815342" 07:02:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712815342" 07:02:22 INFO [http_cmd]: http status code OK 07:02:22 DEBUG --- status code --- 07:02:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:02:22 DEBUG --- http response --- 07:02:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:02:22 DEBUG ----- output ----- 07:02:22 DEBUG { 07:02:22 DEBUG "status": "success", 07:02:22 DEBUG "data": { 07:02:22 DEBUG "resultType": "vector", 07:02:22 DEBUG "result": [ 07:02:22 DEBUG { 07:02:22 DEBUG "metric": { 07:02:22 DEBUG "instance": "10.142.0.107:9100" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG [...] 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG "instance": "10.142.0.77:9100" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 1712815342, 07:02:22 DEBUG "80241883.06666666" 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG } 07:02:22 INFO [http_cmd]: http status code OK 07:02:22 DEBUG --- status code --- 07:02:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:02:22 DEBUG --- http response --- 07:02:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:02:22 DEBUG ----- output ----- 07:02:22 DEBUG { 07:02:22 DEBUG "status": "success", 07:02:22 DEBUG "data": { 07:02:22 DEBUG "resultType": "vector", 07:02:22 DEBUG "result": [ 07:02:22 DEBUG { 07:02:22 DEBUG "metric": { 07:02:22 DEBUG "pod": "ds-idrepo-2" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG [...] 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG "pod": "ds-cts-2" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 1712815342, 07:02:22 DEBUG "0" 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG } 07:02:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815342" 07:02:22 INFO [http_cmd]: http status code OK 07:02:22 DEBUG --- status code --- 07:02:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:02:22 DEBUG --- http response --- 07:02:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:02:22 DEBUG ----- output ----- 07:02:22 DEBUG { 07:02:22 DEBUG "status": "success", 07:02:22 DEBUG "data": { 07:02:22 DEBUG "resultType": "vector", 07:02:22 DEBUG "result": [ 07:02:22 DEBUG { 07:02:22 DEBUG "metric": { 07:02:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG [...] 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 1712815342, 07:02:22 DEBUG "1224.1666666666667" 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG } 07:02:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815342" 07:02:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815342" 07:02:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712815342" 07:02:22 INFO [http_cmd]: http status code OK 07:02:22 DEBUG --- status code --- 07:02:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:02:22 DEBUG --- http response --- 07:02:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:02:22 DEBUG ----- output ----- 07:02:22 DEBUG { 07:02:22 DEBUG "status": "success", 07:02:22 DEBUG "data": { 07:02:22 DEBUG "resultType": "vector", 07:02:22 DEBUG "result": [ 07:02:22 DEBUG { 07:02:22 DEBUG "metric": { 07:02:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG [...] 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG "pod": "idm-668d898cfd-st2t7" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 1712815342, 07:02:22 DEBUG "432.4159822145372" 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG } 07:02:22 INFO [http_cmd]: http status code OK 07:02:22 DEBUG --- status code --- 07:02:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:02:22 DEBUG --- http response --- 07:02:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:02:22 DEBUG ----- output ----- 07:02:22 DEBUG { 07:02:22 DEBUG "status": "success", 07:02:22 DEBUG "data": { 07:02:22 DEBUG "resultType": "vector", 07:02:22 DEBUG "result": [ 07:02:22 DEBUG { 07:02:22 DEBUG "metric": { 07:02:22 DEBUG "pod": "ds-idrepo-2" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG [...] 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG "pod": "ds-idrepo-0" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 1712815342, 07:02:22 DEBUG "0" 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG } 07:02:22 INFO [http_cmd]: http status code OK 07:02:22 DEBUG --- status code --- 07:02:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:02:22 DEBUG --- http response --- 07:02:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:02:22 DEBUG ----- output ----- 07:02:22 DEBUG { 07:02:22 DEBUG "status": "success", 07:02:22 DEBUG "data": { 07:02:22 DEBUG "resultType": "vector", 07:02:22 DEBUG "result": [ 07:02:22 DEBUG { 07:02:22 DEBUG "metric": { 07:02:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG [...] 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 1712815342, 07:02:22 DEBUG "3.0816062577693946" 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG } 07:02:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815342" 07:02:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815342" 07:02:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815342" 07:02:22 INFO [http_cmd]: http status code OK 07:02:22 DEBUG --- status code --- 07:02:22 INFO [http_cmd]: http status code OK 07:02:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:02:22 DEBUG --- status code --- 07:02:22 DEBUG --- http response --- 07:02:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:02:22 DEBUG --- http response --- 07:02:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:02:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:02:22 DEBUG ----- output ----- 07:02:22 DEBUG ----- output ----- 07:02:22 DEBUG { 07:02:22 DEBUG { 07:02:22 DEBUG "status": "success", 07:02:22 DEBUG "status": "success", 07:02:22 DEBUG "data": { 07:02:22 DEBUG "data": { 07:02:22 DEBUG "resultType": "vector", 07:02:22 DEBUG "resultType": "vector", 07:02:22 DEBUG "result": [ 07:02:22 DEBUG "result": [ 07:02:22 DEBUG { 07:02:22 DEBUG { 07:02:22 DEBUG "metric": { 07:02:22 DEBUG "metric": { 07:02:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:02:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:02:22 DEBUG }, 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG [...] 07:02:22 DEBUG [...] 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 INFO [http_cmd]: http status code OK 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG --- status code --- 07:02:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:02:22 DEBUG "pod": "idm-668d898cfd-st2t7" 07:02:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:02:22 DEBUG }, 07:02:22 DEBUG }, 07:02:22 DEBUG --- http response --- 07:02:22 DEBUG "value": [ 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 1712815342, 07:02:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:02:22 DEBUG 1712815342, 07:02:22 DEBUG "1224.6666666666667" 07:02:22 DEBUG ----- output ----- 07:02:22 DEBUG "0" 07:02:22 DEBUG ] 07:02:22 DEBUG { 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG "status": "success", 07:02:22 DEBUG } 07:02:22 DEBUG ] 07:02:22 DEBUG "data": { 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG "resultType": "vector", 07:02:22 DEBUG } 07:02:22 DEBUG } 07:02:22 DEBUG "result": [ 07:02:22 DEBUG } 07:02:22 DEBUG { 07:02:22 DEBUG "metric": { 07:02:22 DEBUG "pod": "ds-idrepo-2" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG [...] 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG "pod": "ds-cts-2" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 1712815342, 07:02:22 DEBUG "0" 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG } 07:02:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815342" 07:02:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712815342" 07:02:22 INFO [http_cmd]: http status code OK 07:02:22 DEBUG --- status code --- 07:02:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:02:22 DEBUG --- http response --- 07:02:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:02:22 DEBUG ----- output ----- 07:02:22 DEBUG { 07:02:22 DEBUG "status": "success", 07:02:22 DEBUG "data": { 07:02:22 DEBUG "resultType": "vector", 07:02:22 DEBUG "result": [ 07:02:22 DEBUG { 07:02:22 DEBUG "metric": { 07:02:22 DEBUG "pod": "ds-idrepo-2" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG [...] 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG "pod": "ds-cts-2" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 1712815342, 07:02:22 DEBUG "29.866666666666667" 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG } 07:02:22 INFO [http_cmd]: http status code OK 07:02:22 DEBUG --- status code --- 07:02:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:02:22 DEBUG --- http response --- 07:02:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:02:22 DEBUG ----- output ----- 07:02:22 DEBUG { 07:02:22 DEBUG "status": "success", 07:02:22 DEBUG "data": { 07:02:22 DEBUG "resultType": "vector", 07:02:22 DEBUG "result": [ 07:02:22 DEBUG { 07:02:22 DEBUG "metric": { 07:02:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG [...] 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 1712815342, 07:02:22 DEBUG "13588983808" 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG } 07:02:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815342" 07:02:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815342" 07:02:22 INFO [http_cmd]: http status code OK 07:02:22 DEBUG --- status code --- 07:02:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:02:22 DEBUG --- http response --- 07:02:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:02:22 DEBUG ----- output ----- 07:02:22 DEBUG { 07:02:22 DEBUG "status": "success", 07:02:22 DEBUG "data": { 07:02:22 DEBUG "resultType": "vector", 07:02:22 DEBUG "result": [ 07:02:22 DEBUG { 07:02:22 DEBUG "metric": { 07:02:22 DEBUG "pod": "ds-idrepo-2" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG [...] 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG "pod": "ds-idrepo-0" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 1712815342, 07:02:22 DEBUG "0" 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG } 07:02:22 INFO [http_cmd]: http status code OK 07:02:22 DEBUG --- status code --- 07:02:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:02:22 DEBUG --- http response --- 07:02:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:02:22 DEBUG ----- output ----- 07:02:22 DEBUG { 07:02:22 DEBUG "status": "success", 07:02:22 DEBUG "data": { 07:02:22 DEBUG "resultType": "vector", 07:02:22 DEBUG "result": [ 07:02:22 DEBUG { 07:02:22 DEBUG "metric": { 07:02:22 DEBUG "pod": "ds-idrepo-2" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG [...] 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG "pod": "ds-cts-2" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 1712815342, 07:02:22 DEBUG "0.000030705" 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG } 07:02:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815342" 07:02:22 INFO [http_cmd]: http status code OK 07:02:22 DEBUG --- status code --- 07:02:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:02:22 DEBUG --- http response --- 07:02:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:02:22 DEBUG ----- output ----- 07:02:22 DEBUG { 07:02:22 DEBUG "status": "success", 07:02:22 DEBUG "data": { 07:02:22 DEBUG "resultType": "vector", 07:02:22 DEBUG "result": [ 07:02:22 DEBUG { 07:02:22 DEBUG "metric": { 07:02:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG [...] 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 1712815342, 07:02:22 DEBUG "0" 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG } 07:02:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815342" 07:02:22 INFO [http_cmd]: http status code OK 07:02:22 DEBUG --- status code --- 07:02:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:02:22 DEBUG --- http response --- 07:02:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:02:22 DEBUG ----- output ----- 07:02:22 DEBUG { 07:02:22 DEBUG "status": "success", 07:02:22 DEBUG "data": { 07:02:22 DEBUG "resultType": "vector", 07:02:22 DEBUG "result": [ 07:02:22 DEBUG { 07:02:22 DEBUG "metric": { 07:02:22 DEBUG "instance": "10.142.0.107:9100" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG [...] 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG "instance": "10.142.0.77:9100" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 1712815342, 07:02:22 DEBUG "0.0010000000000002748" 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG } 07:02:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815342" 07:02:22 INFO [http_cmd]: http status code OK 07:02:22 DEBUG --- status code --- 07:02:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:02:22 DEBUG --- http response --- 07:02:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:02:22 DEBUG ----- output ----- 07:02:22 DEBUG { 07:02:22 DEBUG "status": "success", 07:02:22 DEBUG "data": { 07:02:22 DEBUG "resultType": "vector", 07:02:22 DEBUG "result": [ 07:02:22 DEBUG { 07:02:22 DEBUG "metric": { 07:02:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG [...] 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG 07:02:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:02:22 DEBUG }, 07:02:22 DEBUG "value": [ 07:02:22 DEBUG 1712815342, 07:02:22 DEBUG "1256.6666666666667" 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG ] 07:02:22 DEBUG } 07:02:22 DEBUG } 07:02:34 INFO 07:02:34 INFO [loop_until]: kubectl --namespace=xlou top pods 07:02:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:02:34 INFO [loop_until]: OK (rc = 0) 07:02:34 DEBUG --- stdout --- 07:02:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 8898m 8509Mi am-5b5bd8567b-8h2w4 7782m 8824Mi am-5b5bd8567b-qlkw6 7794m 8444Mi ds-cts-0 6823m 12139Mi ds-cts-1 7526m 12945Mi ds-cts-2 7712m 12312Mi ds-idrepo-0 11759m 5503Mi ds-idrepo-1 373m 5479Mi ds-idrepo-2 306m 5784Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 4m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 2m 68Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4183m 1870Mi 07:02:34 DEBUG --- stderr --- 07:02:34 DEBUG 07:02:38 INFO 07:02:38 INFO [loop_until]: kubectl --namespace=xlou top node 07:02:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:02:38 INFO [loop_until]: OK (rc = 0) 07:02:38 DEBUG --- stdout --- 07:02:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 2730Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 63m 0% 2786Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7450m 24% 10107Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 11863m 39% 3107Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9049m 30% 10118Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 131m 0% 3849Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9126m 30% 10455Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 425m 2% 6469Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11870m 74% 6209Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 7755m 48% 12905Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 7365m 46% 12734Mi 21% gke-xlou-cdm-ds-32e4dcb1-j2hk 7664m 48% 13585Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 427m 2% 6160Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4675m 29% 5767Mi 9% 07:02:38 DEBUG --- stderr --- 07:02:38 DEBUG 07:03:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815402" 07:03:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712815402" 07:03:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712815402" 07:03:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815402" 07:03:22 INFO [http_cmd]: http status code OK 07:03:22 INFO [http_cmd]: http status code OK 07:03:22 DEBUG --- status code --- 07:03:22 DEBUG --- status code --- 07:03:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:03:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:03:22 DEBUG --- http response --- 07:03:22 DEBUG --- http response --- 07:03:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:03:22 DEBUG ----- output ----- 07:03:22 DEBUG { 07:03:22 DEBUG "status": "success", 07:03:22 DEBUG "data": { 07:03:22 DEBUG "resultType": "vector", 07:03:22 DEBUG "result": [ 07:03:22 DEBUG { 07:03:22 DEBUG "metric": { 07:03:22 DEBUG "pod": "ds-idrepo-2" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG [...] 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG "pod": "ds-idrepo-0" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 1712815402, 07:03:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:03:22 DEBUG "0" 07:03:22 DEBUG ----- output ----- 07:03:22 DEBUG ] 07:03:22 DEBUG { 07:03:22 DEBUG } 07:03:22 DEBUG "status": "success", 07:03:22 DEBUG ] 07:03:22 DEBUG "data": { 07:03:22 DEBUG } 07:03:22 DEBUG "resultType": "vector", 07:03:22 DEBUG } 07:03:22 DEBUG "result": [ 07:03:22 DEBUG { 07:03:22 DEBUG "metric": { 07:03:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG [...] 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 1712815402, 07:03:22 DEBUG "0.022" 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG } 07:03:22 ERROR [http_cmd]: ERROR 07:03:22 ERROR --- status code --- 07:03:22 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:03:22 ERROR --- http response --- 07:03:22 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:03:22 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:03:22 INFO [http_cmd]: http status code OK 07:03:22 DEBUG --- status code --- 07:03:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:03:22 DEBUG --- http response --- 07:03:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 07:03:22 DEBUG ----- output ----- 07:03:22 DEBUG { 07:03:22 DEBUG "status": "success", 07:03:22 DEBUG "data": { 07:03:22 DEBUG "resultType": "vector", 07:03:22 DEBUG "result": [ 07:03:22 DEBUG { 07:03:22 DEBUG "metric": { 07:03:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG [...] 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG "pod": "ds-cts-0" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 1712815402, 07:03:22 DEBUG "0" 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG } 07:03:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815402" 07:03:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815402" 07:03:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815402" 07:03:22 INFO [http_cmd]: http status code OK 07:03:22 DEBUG --- status code --- 07:03:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:03:22 DEBUG --- http response --- 07:03:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:03:22 DEBUG ----- output ----- 07:03:22 DEBUG { 07:03:22 DEBUG "status": "success", 07:03:22 DEBUG "data": { 07:03:22 DEBUG "resultType": "vector", 07:03:22 DEBUG "result": [ 07:03:22 DEBUG { 07:03:22 DEBUG "metric": { 07:03:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG [...] 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 1712815402, 07:03:22 DEBUG "109889" 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG } 07:03:22 INFO [http_cmd]: http status code OK 07:03:22 DEBUG --- status code --- 07:03:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:03:22 DEBUG --- http response --- 07:03:22 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:03:22 INFO [http_cmd]: http status code OK 07:03:22 DEBUG --- status code --- 07:03:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:03:22 DEBUG --- http response --- 07:03:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 07:03:22 DEBUG ----- output ----- 07:03:22 DEBUG { 07:03:22 DEBUG "status": "success", 07:03:22 DEBUG "data": { 07:03:22 DEBUG "resultType": "vector", 07:03:22 DEBUG "result": [ 07:03:22 DEBUG { 07:03:22 DEBUG "metric": { 07:03:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG [...] 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG "pod": "ds-cts-0" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 1712815402, 07:03:22 DEBUG "0" 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG } 07:03:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815402" 07:03:22 INFO [http_cmd]: http status code OK 07:03:22 DEBUG --- status code --- 07:03:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:03:22 DEBUG --- http response --- 07:03:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:03:22 DEBUG ----- output ----- 07:03:22 DEBUG { 07:03:22 DEBUG "status": "success", 07:03:22 DEBUG "data": { 07:03:22 DEBUG "resultType": "vector", 07:03:22 DEBUG "result": [ 07:03:22 DEBUG { 07:03:22 DEBUG "metric": { 07:03:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG [...] 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 1712815402, 07:03:22 DEBUG "0" 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG } 07:03:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815402" 07:03:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815402" 07:03:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815402" 07:03:22 INFO [http_cmd]: http status code OK 07:03:22 DEBUG --- status code --- 07:03:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:03:22 DEBUG --- http response --- 07:03:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:03:22 DEBUG ----- output ----- 07:03:22 DEBUG { 07:03:22 DEBUG "status": "success", 07:03:22 DEBUG "data": { 07:03:22 DEBUG "resultType": "vector", 07:03:22 DEBUG "result": [ 07:03:22 DEBUG { 07:03:22 DEBUG "metric": { 07:03:22 DEBUG "instance": "10.142.0.107:9100" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG [...] 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG "instance": "10.142.0.77:9100" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 1712815402, 07:03:22 DEBUG "0.0023333333333335794" 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG } 07:03:22 INFO [http_cmd]: http status code OK 07:03:22 DEBUG --- status code --- 07:03:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:03:22 DEBUG --- http response --- 07:03:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:03:22 DEBUG ----- output ----- 07:03:22 DEBUG { 07:03:22 DEBUG "status": "success", 07:03:22 DEBUG "data": { 07:03:22 DEBUG "resultType": "vector", 07:03:22 DEBUG "result": [ 07:03:22 DEBUG { 07:03:22 DEBUG "metric": { 07:03:22 DEBUG "instance": "10.142.0.107:9100" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG [...] 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG "instance": "10.142.0.77:9100" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 1712815402, 07:03:22 DEBUG "79566996.1" 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG } 07:03:22 INFO [http_cmd]: http status code OK 07:03:22 DEBUG --- status code --- 07:03:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:03:22 DEBUG --- http response --- 07:03:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:03:22 DEBUG ----- output ----- 07:03:22 DEBUG { 07:03:22 DEBUG "status": "success", 07:03:22 DEBUG "data": { 07:03:22 DEBUG "resultType": "vector", 07:03:22 DEBUG "result": [ 07:03:22 DEBUG { 07:03:22 DEBUG "metric": { 07:03:22 DEBUG "instance": "10.142.0.107:9100" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG [...] 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG "instance": "10.142.0.77:9100" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 1712815402, 07:03:22 DEBUG "52980075520" 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG } 07:03:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815402" 07:03:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712815402" 07:03:22 INFO [http_cmd]: http status code OK 07:03:22 DEBUG --- status code --- 07:03:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:03:22 DEBUG --- http response --- 07:03:22 INFO [http_cmd]: http status code OK 07:03:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:03:22 DEBUG --- status code --- 07:03:22 DEBUG ----- output ----- 07:03:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:03:22 DEBUG { 07:03:22 DEBUG --- http response --- 07:03:22 DEBUG "status": "success", 07:03:22 DEBUG "data": { 07:03:22 DEBUG "resultType": "vector", 07:03:22 DEBUG "result": [ 07:03:22 DEBUG { 07:03:22 DEBUG "metric": { 07:03:22 DEBUG "pod": "ds-idrepo-2" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG [...] 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG "pod": "ds-cts-2" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 1712815402, 07:03:22 DEBUG "0" 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG } 07:03:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:03:22 DEBUG ----- output ----- 07:03:22 DEBUG { 07:03:22 DEBUG "status": "success", 07:03:22 DEBUG "data": { 07:03:22 DEBUG "resultType": "vector", 07:03:22 DEBUG "result": [ 07:03:22 DEBUG { 07:03:22 DEBUG "metric": { 07:03:22 DEBUG "instance": "10.142.0.107:9100" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG [...] 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG "instance": "10.142.0.77:9100" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 1712815402, 07:03:22 DEBUG "79569548.36666667" 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG } 07:03:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712815402" 07:03:22 INFO [http_cmd]: http status code OK 07:03:22 DEBUG --- status code --- 07:03:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:03:22 DEBUG --- http response --- 07:03:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:03:22 DEBUG ----- output ----- 07:03:22 DEBUG { 07:03:22 DEBUG "status": "success", 07:03:22 DEBUG "data": { 07:03:22 DEBUG "resultType": "vector", 07:03:22 DEBUG "result": [ 07:03:22 DEBUG { 07:03:22 DEBUG "metric": { 07:03:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG [...] 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 1712815402, 07:03:22 DEBUG "4.844380073345623" 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG } 07:03:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815402" 07:03:22 INFO [http_cmd]: http status code OK 07:03:22 DEBUG --- status code --- 07:03:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:03:22 DEBUG --- http response --- 07:03:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:03:22 DEBUG ----- output ----- 07:03:22 DEBUG { 07:03:22 DEBUG "status": "success", 07:03:22 DEBUG "data": { 07:03:22 DEBUG "resultType": "vector", 07:03:22 DEBUG "result": [ 07:03:22 DEBUG { 07:03:22 DEBUG "metric": { 07:03:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG [...] 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 1712815402, 07:03:22 DEBUG "1515.3" 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG } 07:03:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815402" 07:03:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815402" 07:03:22 INFO [http_cmd]: http status code OK 07:03:22 DEBUG --- status code --- 07:03:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:03:22 DEBUG --- http response --- 07:03:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:03:22 DEBUG ----- output ----- 07:03:22 DEBUG { 07:03:22 DEBUG "status": "success", 07:03:22 DEBUG "data": { 07:03:22 DEBUG "resultType": "vector", 07:03:22 DEBUG "result": [ 07:03:22 DEBUG { 07:03:22 DEBUG "metric": { 07:03:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG [...] 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG "pod": "am-5b5bd8567b-qlkw6" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 1712815402, 07:03:22 DEBUG "2504968.986228181" 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG } 07:03:22 INFO [http_cmd]: http status code OK 07:03:22 DEBUG --- status code --- 07:03:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:03:22 DEBUG --- http response --- 07:03:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:03:22 DEBUG ----- output ----- 07:03:22 DEBUG { 07:03:22 DEBUG "status": "success", 07:03:22 DEBUG "data": { 07:03:22 DEBUG "resultType": "vector", 07:03:22 DEBUG "result": [ 07:03:22 DEBUG { 07:03:22 DEBUG "metric": { 07:03:22 DEBUG "pod": "ds-idrepo-2" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG [...] 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG "pod": "ds-idrepo-0" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 1712815402, 07:03:22 DEBUG "0" 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG } 07:03:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815402" 07:03:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815402" 07:03:22 INFO [http_cmd]: http status code OK 07:03:22 INFO [http_cmd]: http status code OK 07:03:22 DEBUG --- status code --- 07:03:22 DEBUG --- status code --- 07:03:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:03:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:03:22 DEBUG --- http response --- 07:03:22 DEBUG --- http response --- 07:03:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:03:22 DEBUG ----- output ----- 07:03:22 DEBUG { 07:03:22 DEBUG "status": "success", 07:03:22 DEBUG "data": { 07:03:22 DEBUG "resultType": "vector", 07:03:22 DEBUG "result": [ 07:03:22 DEBUG { 07:03:22 DEBUG "metric": { 07:03:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG [...] 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG "pod": "am-5b5bd8567b-qlkw6" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 1712815402, 07:03:22 DEBUG "0" 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG } 07:03:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:03:22 DEBUG ----- output ----- 07:03:22 DEBUG { 07:03:22 DEBUG "status": "success", 07:03:22 DEBUG "data": { 07:03:22 DEBUG "resultType": "vector", 07:03:22 DEBUG "result": [ 07:03:22 DEBUG { 07:03:22 DEBUG "metric": { 07:03:22 DEBUG "pod": "ds-idrepo-2" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG [...] 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG "pod": "ds-cts-2" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 1712815402, 07:03:22 DEBUG "0.00006141" 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG } 07:03:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815402" 07:03:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815402" 07:03:22 INFO [http_cmd]: http status code OK 07:03:22 INFO [http_cmd]: http status code OK 07:03:22 DEBUG --- status code --- 07:03:22 DEBUG --- status code --- 07:03:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:03:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:03:22 DEBUG --- http response --- 07:03:22 DEBUG --- http response --- 07:03:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:03:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:03:22 DEBUG ----- output ----- 07:03:22 DEBUG ----- output ----- 07:03:22 DEBUG { 07:03:22 DEBUG { 07:03:22 DEBUG "status": "success", 07:03:22 DEBUG "status": "success", 07:03:22 DEBUG "data": { 07:03:22 DEBUG "data": { 07:03:22 DEBUG "resultType": "vector", 07:03:22 DEBUG "resultType": "vector", 07:03:22 DEBUG "result": [ 07:03:22 DEBUG "result": [ 07:03:22 DEBUG { 07:03:22 DEBUG { 07:03:22 DEBUG "metric": { 07:03:22 DEBUG "metric": { 07:03:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:03:22 DEBUG "pod": "ds-idrepo-2" 07:03:22 DEBUG }, 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG [...] 07:03:22 DEBUG [...] 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:03:22 DEBUG "pod": "ds-cts-2" 07:03:22 DEBUG }, 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 1712815402, 07:03:22 DEBUG 1712815402, 07:03:22 DEBUG "1514.9" 07:03:22 DEBUG "0" 07:03:22 DEBUG ] 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG } 07:03:22 DEBUG ] 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG } 07:03:22 DEBUG } 07:03:22 DEBUG } 07:03:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815402" 07:03:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712815402" 07:03:22 INFO [http_cmd]: http status code OK 07:03:22 DEBUG --- status code --- 07:03:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:03:22 DEBUG --- http response --- 07:03:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:03:22 DEBUG ----- output ----- 07:03:22 DEBUG { 07:03:22 DEBUG "status": "success", 07:03:22 DEBUG "data": { 07:03:22 DEBUG "resultType": "vector", 07:03:22 DEBUG "result": [ 07:03:22 DEBUG { 07:03:22 DEBUG "metric": { 07:03:22 DEBUG "pod": "ds-idrepo-2" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG [...] 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG "pod": "ds-cts-2" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 1712815402, 07:03:22 DEBUG "30" 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG } 07:03:22 INFO [http_cmd]: http status code OK 07:03:22 DEBUG --- status code --- 07:03:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:03:22 DEBUG --- http response --- 07:03:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:03:22 DEBUG ----- output ----- 07:03:22 DEBUG { 07:03:22 DEBUG "status": "success", 07:03:22 DEBUG "data": { 07:03:22 DEBUG "resultType": "vector", 07:03:22 DEBUG "result": [ 07:03:22 DEBUG { 07:03:22 DEBUG "metric": { 07:03:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG [...] 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 1712815402, 07:03:22 DEBUG "13346103296" 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG } 07:03:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815402" 07:03:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815402" 07:03:22 INFO [http_cmd]: http status code OK 07:03:22 DEBUG --- status code --- 07:03:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:03:22 DEBUG --- http response --- 07:03:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:03:22 DEBUG ----- output ----- 07:03:22 DEBUG { 07:03:22 DEBUG "status": "success", 07:03:22 DEBUG "data": { 07:03:22 DEBUG "resultType": "vector", 07:03:22 DEBUG "result": [ 07:03:22 DEBUG { 07:03:22 DEBUG "metric": { 07:03:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG [...] 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 1712815402, 07:03:22 DEBUG "0" 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG } 07:03:22 INFO [http_cmd]: http status code OK 07:03:22 DEBUG --- status code --- 07:03:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:03:22 DEBUG --- http response --- 07:03:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:03:22 DEBUG ----- output ----- 07:03:22 DEBUG { 07:03:22 DEBUG "status": "success", 07:03:22 DEBUG "data": { 07:03:22 DEBUG "resultType": "vector", 07:03:22 DEBUG "result": [ 07:03:22 DEBUG { 07:03:22 DEBUG "metric": { 07:03:22 DEBUG "pod": "ds-idrepo-2" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG [...] 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG "pod": "ds-idrepo-0" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 1712815402, 07:03:22 DEBUG "0" 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG } 07:03:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815402" 07:03:22 INFO [http_cmd]: http status code OK 07:03:22 DEBUG --- status code --- 07:03:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:03:22 DEBUG --- http response --- 07:03:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:03:22 DEBUG ----- output ----- 07:03:22 DEBUG { 07:03:22 DEBUG "status": "success", 07:03:22 DEBUG "data": { 07:03:22 DEBUG "resultType": "vector", 07:03:22 DEBUG "result": [ 07:03:22 DEBUG { 07:03:22 DEBUG "metric": { 07:03:22 DEBUG "instance": "10.142.0.107:9100" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG [...] 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG "instance": "10.142.0.77:9100" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 1712815402, 07:03:22 DEBUG "0.0003333333333330302" 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG } 07:03:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815402" 07:03:22 INFO [http_cmd]: http status code OK 07:03:22 DEBUG --- status code --- 07:03:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:03:22 DEBUG --- http response --- 07:03:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:03:22 DEBUG ----- output ----- 07:03:22 DEBUG { 07:03:22 DEBUG "status": "success", 07:03:22 DEBUG "data": { 07:03:22 DEBUG "resultType": "vector", 07:03:22 DEBUG "result": [ 07:03:22 DEBUG { 07:03:22 DEBUG "metric": { 07:03:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG [...] 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG 07:03:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:03:22 DEBUG }, 07:03:22 DEBUG "value": [ 07:03:22 DEBUG 1712815402, 07:03:22 DEBUG "1528.3" 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG ] 07:03:22 DEBUG } 07:03:22 DEBUG } 07:03:34 INFO 07:03:34 INFO [loop_until]: kubectl --namespace=xlou top pods 07:03:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:03:34 INFO [loop_until]: OK (rc = 0) 07:03:34 DEBUG --- stdout --- 07:03:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 8263m 8494Mi am-5b5bd8567b-8h2w4 7801m 8829Mi am-5b5bd8567b-qlkw6 6977m 8432Mi ds-cts-0 4377m 12775Mi ds-cts-1 4762m 12710Mi ds-cts-2 4392m 12142Mi ds-idrepo-0 11265m 5503Mi ds-idrepo-1 364m 5480Mi ds-idrepo-2 391m 5784Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 4m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 5m 68Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4834m 2023Mi 07:03:34 DEBUG --- stderr --- 07:03:34 DEBUG 07:03:38 INFO 07:03:38 INFO [loop_until]: kubectl --namespace=xlou top node 07:03:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:03:38 INFO [loop_until]: OK (rc = 0) 07:03:38 DEBUG --- stdout --- 07:03:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 2723Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 65m 0% 2795Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7209m 24% 10100Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 12370m 41% 2989Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 8829m 29% 10107Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 145m 0% 3846Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 8951m 29% 10453Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 455m 2% 6468Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11190m 70% 6208Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 4567m 28% 12702Mi 21% gke-xlou-cdm-ds-32e4dcb1-gxzj 4487m 28% 13483Mi 22% gke-xlou-cdm-ds-32e4dcb1-j2hk 4876m 30% 13400Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 462m 2% 6161Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4818m 30% 5769Mi 9% 07:03:38 DEBUG --- stderr --- 07:03:38 DEBUG 07:04:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815462" 07:04:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815462" 07:04:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815462" 07:04:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712815462" 07:04:22 INFO [http_cmd]: http status code OK 07:04:22 DEBUG --- status code --- 07:04:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:04:22 DEBUG --- http response --- 07:04:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815462" 07:04:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815462" 07:04:22 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:04:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712815462" 07:04:22 INFO [http_cmd]: http status code OK 07:04:22 DEBUG --- status code --- 07:04:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:04:22 DEBUG --- http response --- 07:04:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:04:22 DEBUG ----- output ----- 07:04:22 DEBUG { 07:04:22 DEBUG "status": "success", 07:04:22 DEBUG "data": { 07:04:22 DEBUG "resultType": "vector", 07:04:22 DEBUG "result": [ 07:04:22 DEBUG { 07:04:22 DEBUG "metric": { 07:04:22 DEBUG "pod": "ds-idrepo-2" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG [...] 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG "pod": "ds-idrepo-0" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 1712815462, 07:04:22 DEBUG "0" 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG } 07:04:22 INFO [http_cmd]: http status code OK 07:04:22 DEBUG --- status code --- 07:04:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:04:22 DEBUG --- http response --- 07:04:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:04:22 DEBUG ----- output ----- 07:04:22 DEBUG { 07:04:22 DEBUG "status": "success", 07:04:22 DEBUG "data": { 07:04:22 DEBUG "resultType": "vector", 07:04:22 DEBUG "result": [ 07:04:22 DEBUG { 07:04:22 DEBUG "metric": { 07:04:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG [...] 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 1712815462, 07:04:22 DEBUG "2.7419244632634863" 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG } 07:04:22 INFO [http_cmd]: http status code OK 07:04:22 DEBUG --- status code --- 07:04:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:04:22 DEBUG --- http response --- 07:04:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:04:22 DEBUG ----- output ----- 07:04:22 DEBUG { 07:04:22 DEBUG "status": "success", 07:04:22 DEBUG "data": { 07:04:22 DEBUG "resultType": "vector", 07:04:22 DEBUG "result": [ 07:04:22 DEBUG { 07:04:22 DEBUG "metric": { 07:04:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG [...] 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 1712815462, 07:04:22 DEBUG "0.0225" 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG } 07:04:22 ERROR [http_cmd]: ERROR 07:04:22 ERROR --- status code --- 07:04:22 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:04:22 ERROR --- http response --- 07:04:22 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:04:22 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:04:22 INFO [http_cmd]: http status code OK 07:04:22 DEBUG --- status code --- 07:04:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:04:22 DEBUG --- http response --- 07:04:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:04:22 INFO [http_cmd]: http status code OK 07:04:22 DEBUG ----- output ----- 07:04:22 DEBUG --- status code --- 07:04:22 DEBUG { 07:04:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:04:22 DEBUG "status": "success", 07:04:22 DEBUG --- http response --- 07:04:22 DEBUG "data": { 07:04:22 DEBUG "resultType": "vector", 07:04:22 DEBUG "result": [ 07:04:22 DEBUG { 07:04:22 DEBUG "metric": { 07:04:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG [...] 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 1712815462, 07:04:22 DEBUG "88263.26430830546" 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG } 07:04:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:04:22 DEBUG ----- output ----- 07:04:22 DEBUG { 07:04:22 DEBUG "status": "success", 07:04:22 DEBUG "data": { 07:04:22 DEBUG "resultType": "vector", 07:04:22 DEBUG "result": [ 07:04:22 DEBUG { 07:04:22 DEBUG "metric": { 07:04:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG [...] 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 1712815462, 07:04:22 DEBUG "202378" 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG } 07:04:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815462" 07:04:22 INFO [http_cmd]: http status code OK 07:04:22 DEBUG --- status code --- 07:04:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:04:22 DEBUG --- http response --- 07:04:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:04:22 DEBUG ----- output ----- 07:04:22 DEBUG { 07:04:22 DEBUG "status": "success", 07:04:22 DEBUG "data": { 07:04:22 DEBUG "resultType": "vector", 07:04:22 DEBUG "result": [ 07:04:22 DEBUG { 07:04:22 DEBUG "metric": { 07:04:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG [...] 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 1712815462, 07:04:22 DEBUG "0" 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG } 07:04:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815462" 07:04:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712815462" 07:04:22 INFO [http_cmd]: http status code OK 07:04:22 DEBUG --- status code --- 07:04:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:04:22 DEBUG --- http response --- 07:04:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:04:22 DEBUG ----- output ----- 07:04:22 DEBUG { 07:04:22 DEBUG "status": "success", 07:04:22 DEBUG "data": { 07:04:22 DEBUG "resultType": "vector", 07:04:22 DEBUG "result": [ 07:04:22 DEBUG { 07:04:22 DEBUG "metric": { 07:04:22 DEBUG "instance": "10.142.0.107:9100" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG [...] 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG "instance": "10.142.0.77:9100" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 1712815462, 07:04:22 DEBUG "0.002300000000000087" 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG } 07:04:22 INFO [http_cmd]: http status code OK 07:04:22 DEBUG --- status code --- 07:04:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:04:22 DEBUG --- http response --- 07:04:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:04:22 DEBUG ----- output ----- 07:04:22 DEBUG { 07:04:22 DEBUG "status": "success", 07:04:22 DEBUG "data": { 07:04:22 DEBUG "resultType": "vector", 07:04:22 DEBUG "result": [ 07:04:22 DEBUG { 07:04:22 DEBUG "metric": { 07:04:22 DEBUG "pod": "ds-idrepo-2" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG [...] 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG "pod": "ds-cts-2" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 1712815462, 07:04:22 DEBUG "0" 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG } 07:04:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815462" 07:04:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815462" 07:04:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712815462" 07:04:22 INFO [http_cmd]: http status code OK 07:04:22 DEBUG --- status code --- 07:04:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:04:22 DEBUG --- http response --- 07:04:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:04:22 DEBUG ----- output ----- 07:04:22 DEBUG { 07:04:22 DEBUG "status": "success", 07:04:22 DEBUG "data": { 07:04:22 DEBUG "resultType": "vector", 07:04:22 DEBUG "result": [ 07:04:22 DEBUG { 07:04:22 DEBUG "metric": { 07:04:22 DEBUG "instance": "10.142.0.107:9100" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG [...] 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG "instance": "10.142.0.77:9100" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 1712815462, 07:04:22 DEBUG "80073376.86666666" 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG } 07:04:22 INFO [http_cmd]: http status code OK 07:04:22 DEBUG --- status code --- 07:04:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:04:22 DEBUG --- http response --- 07:04:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:04:22 DEBUG ----- output ----- 07:04:22 DEBUG { 07:04:22 DEBUG "status": "success", 07:04:22 DEBUG "data": { 07:04:22 DEBUG "resultType": "vector", 07:04:22 DEBUG "result": [ 07:04:22 DEBUG { 07:04:22 DEBUG "metric": { 07:04:22 DEBUG "instance": "10.142.0.107:9100" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG [...] 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG "instance": "10.142.0.77:9100" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 1712815462, 07:04:22 DEBUG "52981337088" 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG } 07:04:22 INFO [http_cmd]: http status code OK 07:04:22 DEBUG --- status code --- 07:04:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:04:22 DEBUG --- http response --- 07:04:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:04:22 DEBUG ----- output ----- 07:04:22 DEBUG { 07:04:22 DEBUG "status": "success", 07:04:22 DEBUG "data": { 07:04:22 DEBUG "resultType": "vector", 07:04:22 DEBUG "result": [ 07:04:22 DEBUG { 07:04:22 DEBUG "metric": { 07:04:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG [...] 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 1712815462, 07:04:22 DEBUG "4.870843113262624" 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG } 07:04:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815462" 07:04:22 INFO [http_cmd]: http status code OK 07:04:22 DEBUG --- status code --- 07:04:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:04:22 DEBUG --- http response --- 07:04:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:04:22 DEBUG ----- output ----- 07:04:22 DEBUG { 07:04:22 DEBUG "status": "success", 07:04:22 DEBUG "data": { 07:04:22 DEBUG "resultType": "vector", 07:04:22 DEBUG "result": [ 07:04:22 DEBUG { 07:04:22 DEBUG "metric": { 07:04:22 DEBUG "instance": "10.142.0.107:9100" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG [...] 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG "instance": "10.142.0.77:9100" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 1712815462, 07:04:22 DEBUG "80075660.83333333" 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG } 07:04:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815462" 07:04:22 INFO [http_cmd]: http status code OK 07:04:22 DEBUG --- status code --- 07:04:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:04:22 DEBUG --- http response --- 07:04:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:04:22 DEBUG ----- output ----- 07:04:22 DEBUG { 07:04:22 DEBUG "status": "success", 07:04:22 DEBUG "data": { 07:04:22 DEBUG "resultType": "vector", 07:04:22 DEBUG "result": [ 07:04:22 DEBUG { 07:04:22 DEBUG "metric": { 07:04:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG [...] 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 1712815462, 07:04:22 DEBUG "1542.8" 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG } 07:04:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815462" 07:04:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815462" 07:04:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815462" 07:04:22 INFO [http_cmd]: http status code OK 07:04:22 DEBUG --- status code --- 07:04:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:04:22 DEBUG --- http response --- 07:04:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:04:22 DEBUG ----- output ----- 07:04:22 DEBUG { 07:04:22 DEBUG "status": "success", 07:04:22 DEBUG "data": { 07:04:22 DEBUG "resultType": "vector", 07:04:22 DEBUG "result": [ 07:04:22 DEBUG { 07:04:22 DEBUG "metric": { 07:04:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG [...] 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 1712815462, 07:04:22 DEBUG "0" 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG } 07:04:22 INFO [http_cmd]: http status code OK 07:04:22 DEBUG --- status code --- 07:04:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:04:22 DEBUG --- http response --- 07:04:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:04:22 DEBUG ----- output ----- 07:04:22 DEBUG { 07:04:22 DEBUG "status": "success", 07:04:22 DEBUG "data": { 07:04:22 DEBUG "resultType": "vector", 07:04:22 DEBUG "result": [ 07:04:22 DEBUG { 07:04:22 DEBUG "metric": { 07:04:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG [...] 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 1712815462, 07:04:22 DEBUG "0" 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG } 07:04:22 INFO [http_cmd]: http status code OK 07:04:22 DEBUG --- status code --- 07:04:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:04:22 DEBUG --- http response --- 07:04:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:04:22 DEBUG ----- output ----- 07:04:22 DEBUG { 07:04:22 DEBUG "status": "success", 07:04:22 DEBUG "data": { 07:04:22 DEBUG "resultType": "vector", 07:04:22 DEBUG "result": [ 07:04:22 DEBUG { 07:04:22 DEBUG "metric": { 07:04:22 DEBUG "pod": "ds-idrepo-2" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG [...] 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG "pod": "ds-idrepo-0" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 1712815462, 07:04:22 DEBUG "0" 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG } 07:04:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815462" 07:04:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815462" 07:04:22 INFO [http_cmd]: http status code OK 07:04:22 DEBUG --- status code --- 07:04:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:04:22 DEBUG --- http response --- 07:04:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:04:22 DEBUG ----- output ----- 07:04:22 DEBUG { 07:04:22 DEBUG "status": "success", 07:04:22 DEBUG "data": { 07:04:22 DEBUG "resultType": "vector", 07:04:22 DEBUG "result": [ 07:04:22 DEBUG { 07:04:22 DEBUG "metric": { 07:04:22 DEBUG "pod": "ds-idrepo-2" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG [...] 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG "pod": "ds-cts-2" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 1712815462, 07:04:22 DEBUG "29.966666666666665" 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG } 07:04:22 INFO [http_cmd]: http status code OK 07:04:22 DEBUG --- status code --- 07:04:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:04:22 DEBUG --- http response --- 07:04:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:04:22 DEBUG ----- output ----- 07:04:22 DEBUG { 07:04:22 DEBUG "status": "success", 07:04:22 DEBUG "data": { 07:04:22 DEBUG "resultType": "vector", 07:04:22 DEBUG "result": [ 07:04:22 DEBUG { 07:04:22 DEBUG "metric": { 07:04:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG [...] 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 1712815462, 07:04:22 DEBUG "1542.5333333333333" 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG } 07:04:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815462" 07:04:22 INFO [http_cmd]: http status code OK 07:04:22 DEBUG --- status code --- 07:04:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:04:22 DEBUG --- http response --- 07:04:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:04:22 DEBUG ----- output ----- 07:04:22 DEBUG { 07:04:22 DEBUG "status": "success", 07:04:22 DEBUG "data": { 07:04:22 DEBUG "resultType": "vector", 07:04:22 DEBUG "result": [ 07:04:22 DEBUG { 07:04:22 DEBUG "metric": { 07:04:22 DEBUG "pod": "ds-idrepo-2" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG [...] 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG "pod": "ds-cts-2" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 1712815462, 07:04:22 DEBUG "0.000030705" 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG } 07:04:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815462" 07:04:22 INFO [http_cmd]: http status code OK 07:04:22 DEBUG --- status code --- 07:04:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:04:22 DEBUG --- http response --- 07:04:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:04:22 DEBUG ----- output ----- 07:04:22 DEBUG { 07:04:22 DEBUG "status": "success", 07:04:22 DEBUG "data": { 07:04:22 DEBUG "resultType": "vector", 07:04:22 DEBUG "result": [ 07:04:22 DEBUG { 07:04:22 DEBUG "metric": { 07:04:22 DEBUG "pod": "ds-idrepo-2" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG [...] 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG "pod": "ds-cts-2" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 1712815462, 07:04:22 DEBUG "0" 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG } 07:04:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815462" 07:04:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712815462" 07:04:22 INFO [http_cmd]: http status code OK 07:04:22 DEBUG --- status code --- 07:04:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:04:22 DEBUG --- http response --- 07:04:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:04:22 DEBUG ----- output ----- 07:04:22 DEBUG { 07:04:22 DEBUG "status": "success", 07:04:22 DEBUG "data": { 07:04:22 DEBUG "resultType": "vector", 07:04:22 DEBUG "result": [ 07:04:22 DEBUG { 07:04:22 DEBUG "metric": { 07:04:22 DEBUG "pod": "ds-idrepo-2" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG [...] 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG "pod": "ds-idrepo-0" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 1712815462, 07:04:22 DEBUG "0" 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG } 07:04:22 INFO [http_cmd]: http status code OK 07:04:22 DEBUG --- status code --- 07:04:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:04:22 DEBUG --- http response --- 07:04:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:04:22 DEBUG ----- output ----- 07:04:22 DEBUG { 07:04:22 DEBUG "status": "success", 07:04:22 DEBUG "data": { 07:04:22 DEBUG "resultType": "vector", 07:04:22 DEBUG "result": [ 07:04:22 DEBUG { 07:04:22 DEBUG "metric": { 07:04:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG [...] 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 1712815462, 07:04:22 DEBUG "13082152960" 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG } 07:04:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815462" 07:04:22 INFO [http_cmd]: http status code OK 07:04:22 DEBUG --- status code --- 07:04:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:04:22 DEBUG --- http response --- 07:04:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:04:22 DEBUG ----- output ----- 07:04:22 DEBUG { 07:04:22 DEBUG "status": "success", 07:04:22 DEBUG "data": { 07:04:22 DEBUG "resultType": "vector", 07:04:22 DEBUG "result": [ 07:04:22 DEBUG { 07:04:22 DEBUG "metric": { 07:04:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG [...] 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 1712815462, 07:04:22 DEBUG "0" 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG } 07:04:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815462" 07:04:22 INFO [http_cmd]: http status code OK 07:04:22 DEBUG --- status code --- 07:04:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:04:22 DEBUG --- http response --- 07:04:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:04:22 DEBUG ----- output ----- 07:04:22 DEBUG { 07:04:22 DEBUG "status": "success", 07:04:22 DEBUG "data": { 07:04:22 DEBUG "resultType": "vector", 07:04:22 DEBUG "result": [ 07:04:22 DEBUG { 07:04:22 DEBUG "metric": { 07:04:22 DEBUG "instance": "10.142.0.107:9100" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG [...] 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG "instance": "10.142.0.77:9100" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 1712815462, 07:04:22 DEBUG "0.00033333333333350387" 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG } 07:04:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815462" 07:04:22 INFO [http_cmd]: http status code OK 07:04:22 DEBUG --- status code --- 07:04:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:04:22 DEBUG --- http response --- 07:04:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:04:22 DEBUG ----- output ----- 07:04:22 DEBUG { 07:04:22 DEBUG "status": "success", 07:04:22 DEBUG "data": { 07:04:22 DEBUG "resultType": "vector", 07:04:22 DEBUG "result": [ 07:04:22 DEBUG { 07:04:22 DEBUG "metric": { 07:04:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG [...] 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG 07:04:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:04:22 DEBUG }, 07:04:22 DEBUG "value": [ 07:04:22 DEBUG 1712815462, 07:04:22 DEBUG "1541.9666666666667" 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG ] 07:04:22 DEBUG } 07:04:22 DEBUG } 07:04:34 INFO 07:04:34 INFO [loop_until]: kubectl --namespace=xlou top pods 07:04:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:04:34 INFO [loop_until]: OK (rc = 0) 07:04:34 DEBUG --- stdout --- 07:04:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 8429m 8491Mi am-5b5bd8567b-8h2w4 7951m 8816Mi am-5b5bd8567b-qlkw6 6736m 8436Mi ds-cts-0 4438m 12566Mi ds-cts-1 4733m 12355Mi ds-cts-2 4579m 12752Mi ds-idrepo-0 11135m 5503Mi ds-idrepo-1 437m 5479Mi ds-idrepo-2 447m 5784Mi end-user-ui-58878d7668-gjq6d 1m 4Mi idm-668d898cfd-2vqdm 4m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 2m 68Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4850m 2025Mi 07:04:34 DEBUG --- stderr --- 07:04:34 DEBUG 07:04:38 INFO 07:04:38 INFO [loop_until]: kubectl --namespace=xlou top node 07:04:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:04:38 INFO [loop_until]: OK (rc = 0) 07:04:38 DEBUG --- stdout --- 07:04:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 2728Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 61m 0% 2783Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7172m 24% 10094Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 12445m 41% 2980Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 8810m 29% 10093Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 145m 0% 3833Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9014m 30% 10454Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 502m 3% 6467Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11569m 72% 6207Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 4752m 29% 13369Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 4516m 28% 13189Mi 22% gke-xlou-cdm-ds-32e4dcb1-j2hk 4847m 30% 13107Mi 22% gke-xlou-cdm-ds-32e4dcb1-rr7g 496m 3% 6164Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 4989m 31% 5766Mi 9% 07:04:38 DEBUG --- stderr --- 07:04:38 DEBUG 07:05:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712815522" 07:05:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815522" 07:05:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815522" 07:05:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815522" 07:05:22 INFO [http_cmd]: http status code OK 07:05:22 DEBUG --- status code --- 07:05:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:05:22 DEBUG --- http response --- 07:05:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:05:22 DEBUG ----- output ----- 07:05:22 DEBUG { 07:05:22 DEBUG "status": "success", 07:05:22 DEBUG "data": { 07:05:22 DEBUG "resultType": "vector", 07:05:22 DEBUG "result": [ 07:05:22 DEBUG { 07:05:22 DEBUG "metric": { 07:05:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG [...] 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 1712815522, 07:05:22 DEBUG "0.023" 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG } 07:05:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712815522" 07:05:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815522" 07:05:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815522" 07:05:22 INFO [http_cmd]: http status code OK 07:05:22 DEBUG --- status code --- 07:05:22 INFO [http_cmd]: http status code OK 07:05:22 INFO [http_cmd]: http status code OK 07:05:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:05:22 DEBUG --- status code --- 07:05:22 DEBUG --- status code --- 07:05:22 DEBUG --- http response --- 07:05:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:05:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:05:22 DEBUG --- http response --- 07:05:22 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:05:22 DEBUG --- http response --- 07:05:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:05:22 DEBUG ----- output ----- 07:05:22 DEBUG { 07:05:22 DEBUG "status": "success", 07:05:22 DEBUG "data": { 07:05:22 DEBUG "resultType": "vector", 07:05:22 DEBUG "result": [ 07:05:22 DEBUG { 07:05:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 07:05:22 DEBUG "metric": { 07:05:22 DEBUG ----- output ----- 07:05:22 DEBUG "pod": "ds-idrepo-2" 07:05:22 DEBUG { 07:05:22 DEBUG }, 07:05:22 DEBUG "status": "success", 07:05:22 DEBUG "value": [ 07:05:22 DEBUG "data": { 07:05:22 DEBUG 07:05:22 DEBUG "resultType": "vector", 07:05:22 DEBUG 07:05:22 DEBUG "result": [ 07:05:22 DEBUG 07:05:22 DEBUG { 07:05:22 DEBUG [...] 07:05:22 DEBUG "metric": { 07:05:22 DEBUG 07:05:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:05:22 DEBUG 07:05:22 DEBUG }, 07:05:22 DEBUG 07:05:22 DEBUG "value": [ 07:05:22 DEBUG "pod": "ds-idrepo-0" 07:05:22 DEBUG 07:05:22 DEBUG }, 07:05:22 DEBUG 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 07:05:22 DEBUG 1712815522, 07:05:22 DEBUG [...] 07:05:22 DEBUG "0" 07:05:22 DEBUG 07:05:22 DEBUG ] 07:05:22 DEBUG 07:05:22 DEBUG } 07:05:22 DEBUG 07:05:22 DEBUG ] 07:05:22 DEBUG "pod": "am-5b5bd8567b-qlkw6" 07:05:22 DEBUG } 07:05:22 DEBUG }, 07:05:22 DEBUG } 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 1712815522, 07:05:22 DEBUG "2.7678466534392285" 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG } 07:05:22 ERROR [http_cmd]: ERROR 07:05:22 INFO [http_cmd]: http status code OK 07:05:22 ERROR --- status code --- 07:05:22 INFO [http_cmd]: http status code OK 07:05:22 DEBUG --- status code --- 07:05:22 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:05:22 DEBUG --- status code --- 07:05:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:05:22 ERROR --- http response --- 07:05:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:05:22 DEBUG --- http response --- 07:05:22 DEBUG --- http response --- 07:05:22 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:05:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:05:22 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:05:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 07:05:22 DEBUG ----- output ----- 07:05:22 DEBUG ----- output ----- 07:05:22 DEBUG { 07:05:22 DEBUG { 07:05:22 DEBUG "status": "success", 07:05:22 DEBUG "status": "success", 07:05:22 DEBUG "data": { 07:05:22 DEBUG "data": { 07:05:22 DEBUG "resultType": "vector", 07:05:22 DEBUG "resultType": "vector", 07:05:22 DEBUG "result": [ 07:05:22 DEBUG "result": [ 07:05:22 DEBUG { 07:05:22 DEBUG { 07:05:22 DEBUG "metric": { 07:05:22 DEBUG "metric": { 07:05:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:05:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:05:22 DEBUG }, 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG [...] 07:05:22 DEBUG [...] 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:05:22 DEBUG "pod": "am-5b5bd8567b-qlkw6" 07:05:22 DEBUG }, 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 1712815522, 07:05:22 DEBUG 1712815522, 07:05:22 DEBUG "271914" 07:05:22 DEBUG "139180.84594007244" 07:05:22 DEBUG ] 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG } 07:05:22 DEBUG ] 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG } 07:05:22 DEBUG } 07:05:22 DEBUG } 07:05:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815522" 07:05:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815522" 07:05:22 INFO [http_cmd]: http status code OK 07:05:22 INFO [http_cmd]: http status code OK 07:05:22 DEBUG --- status code --- 07:05:22 DEBUG --- status code --- 07:05:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:05:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:05:22 DEBUG --- http response --- 07:05:22 DEBUG --- http response --- 07:05:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:05:22 DEBUG ----- output ----- 07:05:22 DEBUG { 07:05:22 DEBUG "status": "success", 07:05:22 DEBUG "data": { 07:05:22 DEBUG "resultType": "vector", 07:05:22 DEBUG "result": [ 07:05:22 DEBUG { 07:05:22 DEBUG "metric": { 07:05:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG [...] 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 1712815522, 07:05:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:05:22 DEBUG "1233.4666666666667" 07:05:22 DEBUG ----- output ----- 07:05:22 DEBUG ] 07:05:22 DEBUG { 07:05:22 DEBUG } 07:05:22 DEBUG "status": "success", 07:05:22 DEBUG ] 07:05:22 DEBUG "data": { 07:05:22 DEBUG } 07:05:22 DEBUG "resultType": "vector", 07:05:22 DEBUG } 07:05:22 DEBUG "result": [ 07:05:22 DEBUG { 07:05:22 DEBUG "metric": { 07:05:22 DEBUG "instance": "10.142.0.107:9100" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG [...] 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG "instance": "10.142.0.77:9100" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 1712815522, 07:05:22 DEBUG "0.0020333333333333314" 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG } 07:05:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712815522" 07:05:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712815522" 07:05:22 INFO [http_cmd]: http status code OK 07:05:22 DEBUG --- status code --- 07:05:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:05:22 DEBUG --- http response --- 07:05:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:05:22 DEBUG ----- output ----- 07:05:22 DEBUG { 07:05:22 DEBUG "status": "success", 07:05:22 DEBUG "data": { 07:05:22 DEBUG "resultType": "vector", 07:05:22 DEBUG "result": [ 07:05:22 DEBUG { 07:05:22 DEBUG "metric": { 07:05:22 DEBUG "pod": "ds-idrepo-2" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG [...] 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG "pod": "ds-cts-2" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 1712815522, 07:05:22 DEBUG "0" 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG } 07:05:22 INFO [http_cmd]: http status code OK 07:05:22 DEBUG --- status code --- 07:05:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:05:22 DEBUG --- http response --- 07:05:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:05:22 DEBUG ----- output ----- 07:05:22 DEBUG { 07:05:22 DEBUG "status": "success", 07:05:22 DEBUG "data": { 07:05:22 DEBUG "resultType": "vector", 07:05:22 DEBUG "result": [ 07:05:22 DEBUG { 07:05:22 DEBUG "metric": { 07:05:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG [...] 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 1712815522, 07:05:22 DEBUG "4.862220265320892" 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG } 07:05:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815522" 07:05:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815522" 07:05:22 INFO [http_cmd]: http status code OK 07:05:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815522" 07:05:22 DEBUG --- status code --- 07:05:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:05:22 DEBUG --- http response --- 07:05:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:05:22 DEBUG ----- output ----- 07:05:22 DEBUG { 07:05:22 DEBUG "status": "success", 07:05:22 DEBUG "data": { 07:05:22 DEBUG "resultType": "vector", 07:05:22 DEBUG "result": [ 07:05:22 DEBUG { 07:05:22 DEBUG "metric": { 07:05:22 DEBUG "instance": "10.142.0.107:9100" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG [...] 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG "instance": "10.142.0.77:9100" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 1712815522, 07:05:22 DEBUG "88581614.19999999" 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG } 07:05:22 INFO [http_cmd]: http status code OK 07:05:22 DEBUG --- status code --- 07:05:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:05:22 DEBUG --- http response --- 07:05:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:05:22 DEBUG ----- output ----- 07:05:22 DEBUG { 07:05:22 DEBUG "status": "success", 07:05:22 DEBUG "data": { 07:05:22 DEBUG "resultType": "vector", 07:05:22 DEBUG "result": [ 07:05:22 DEBUG { 07:05:22 DEBUG "metric": { 07:05:22 DEBUG "instance": "10.142.0.107:9100" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG [...] 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG "instance": "10.142.0.77:9100" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 1712815522, 07:05:22 DEBUG "52982668288" 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG } 07:05:22 INFO [http_cmd]: http status code OK 07:05:22 DEBUG --- status code --- 07:05:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:05:22 DEBUG --- http response --- 07:05:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:05:22 DEBUG ----- output ----- 07:05:22 DEBUG { 07:05:22 DEBUG "status": "success", 07:05:22 DEBUG "data": { 07:05:22 DEBUG "resultType": "vector", 07:05:22 DEBUG "result": [ 07:05:22 DEBUG { 07:05:22 DEBUG "metric": { 07:05:22 DEBUG "instance": "10.142.0.107:9100" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG [...] 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG "instance": "10.142.0.77:9100" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 1712815522, 07:05:22 DEBUG "88583687.8" 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG } 07:05:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815522" 07:05:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815522" 07:05:22 INFO [http_cmd]: http status code OK 07:05:22 DEBUG --- status code --- 07:05:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:05:22 DEBUG --- http response --- 07:05:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 07:05:22 DEBUG ----- output ----- 07:05:22 DEBUG { 07:05:22 DEBUG "status": "success", 07:05:22 DEBUG "data": { 07:05:22 DEBUG "resultType": "vector", 07:05:22 DEBUG "result": [ 07:05:22 DEBUG { 07:05:22 DEBUG "metric": { 07:05:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG [...] 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG "pod": "am-5b5bd8567b-qlkw6" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 1712815522, 07:05:22 DEBUG "613041.1381758313" 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG } 07:05:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815522" 07:05:22 INFO [http_cmd]: http status code OK 07:05:22 DEBUG --- status code --- 07:05:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:05:22 DEBUG --- http response --- 07:05:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 07:05:22 DEBUG ----- output ----- 07:05:22 DEBUG { 07:05:22 DEBUG "status": "success", 07:05:22 DEBUG "data": { 07:05:22 DEBUG "resultType": "vector", 07:05:22 DEBUG "result": [ 07:05:22 DEBUG { 07:05:22 DEBUG "metric": { 07:05:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG [...] 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG "pod": "am-5b5bd8567b-qlkw6" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 1712815522, 07:05:22 DEBUG "0" 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG } 07:05:22 INFO [http_cmd]: http status code OK 07:05:22 DEBUG --- status code --- 07:05:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:05:22 DEBUG --- http response --- 07:05:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:05:22 DEBUG ----- output ----- 07:05:22 DEBUG { 07:05:22 DEBUG "status": "success", 07:05:22 DEBUG "data": { 07:05:22 DEBUG "resultType": "vector", 07:05:22 DEBUG "result": [ 07:05:22 DEBUG { 07:05:22 DEBUG "metric": { 07:05:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG [...] 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 1712815522, 07:05:22 DEBUG "1499.5" 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG } 07:05:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815522" 07:05:22 INFO [http_cmd]: http status code OK 07:05:22 DEBUG --- status code --- 07:05:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:05:22 DEBUG --- http response --- 07:05:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:05:22 DEBUG ----- output ----- 07:05:22 DEBUG { 07:05:22 DEBUG "status": "success", 07:05:22 DEBUG "data": { 07:05:22 DEBUG "resultType": "vector", 07:05:22 DEBUG "result": [ 07:05:22 DEBUG { 07:05:22 DEBUG "metric": { 07:05:22 DEBUG "pod": "ds-idrepo-2" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG [...] 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG "pod": "ds-idrepo-0" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 1712815522, 07:05:22 DEBUG "0" 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG } 07:05:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815522" 07:05:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815522" 07:05:22 INFO [http_cmd]: http status code OK 07:05:22 DEBUG --- status code --- 07:05:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:05:22 DEBUG --- http response --- 07:05:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:05:22 DEBUG ----- output ----- 07:05:22 DEBUG { 07:05:22 DEBUG "status": "success", 07:05:22 DEBUG "data": { 07:05:22 DEBUG "resultType": "vector", 07:05:22 DEBUG "result": [ 07:05:22 DEBUG { 07:05:22 DEBUG "metric": { 07:05:22 DEBUG "pod": "ds-idrepo-2" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG [...] 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG "pod": "ds-cts-2" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 1712815522, 07:05:22 DEBUG "29.599999999999998" 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG } 07:05:22 INFO [http_cmd]: http status code OK 07:05:22 DEBUG --- status code --- 07:05:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:05:22 DEBUG --- http response --- 07:05:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:05:22 DEBUG ----- output ----- 07:05:22 DEBUG { 07:05:22 DEBUG "status": "success", 07:05:22 DEBUG "data": { 07:05:22 DEBUG "resultType": "vector", 07:05:22 DEBUG "result": [ 07:05:22 DEBUG { 07:05:22 DEBUG "metric": { 07:05:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG [...] 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 1712815522, 07:05:22 DEBUG "1498.8" 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG } 07:05:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815522" 07:05:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815522" 07:05:22 INFO [http_cmd]: http status code OK 07:05:22 INFO [http_cmd]: http status code OK 07:05:22 DEBUG --- status code --- 07:05:22 DEBUG --- status code --- 07:05:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:05:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:05:22 DEBUG --- http response --- 07:05:22 DEBUG --- http response --- 07:05:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:05:22 DEBUG ----- output ----- 07:05:22 DEBUG { 07:05:22 DEBUG "status": "success", 07:05:22 DEBUG "data": { 07:05:22 DEBUG "resultType": "vector", 07:05:22 DEBUG "result": [ 07:05:22 DEBUG { 07:05:22 DEBUG "metric": { 07:05:22 DEBUG "pod": "ds-idrepo-2" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG [...] 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG "pod": "ds-idrepo-0" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 1712815522, 07:05:22 DEBUG "0" 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG } 07:05:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:05:22 DEBUG ----- output ----- 07:05:22 DEBUG { 07:05:22 DEBUG "status": "success", 07:05:22 DEBUG "data": { 07:05:22 DEBUG "resultType": "vector", 07:05:22 DEBUG "result": [ 07:05:22 DEBUG { 07:05:22 DEBUG "metric": { 07:05:22 DEBUG "pod": "ds-idrepo-2" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG [...] 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG "pod": "ds-cts-2" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 1712815522, 07:05:22 DEBUG "0.000033333333333333335" 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG } 07:05:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815522" 07:05:22 INFO [http_cmd]: http status code OK 07:05:22 DEBUG --- status code --- 07:05:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:05:22 DEBUG --- http response --- 07:05:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:05:22 DEBUG ----- output ----- 07:05:22 DEBUG { 07:05:22 DEBUG "status": "success", 07:05:22 DEBUG "data": { 07:05:22 DEBUG "resultType": "vector", 07:05:22 DEBUG "result": [ 07:05:22 DEBUG { 07:05:22 DEBUG "metric": { 07:05:22 DEBUG "pod": "ds-idrepo-2" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG [...] 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG "pod": "ds-cts-2" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 1712815522, 07:05:22 DEBUG "0" 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG } 07:05:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712815522" 07:05:22 INFO [http_cmd]: http status code OK 07:05:22 DEBUG --- status code --- 07:05:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:05:22 DEBUG --- http response --- 07:05:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:05:22 DEBUG ----- output ----- 07:05:22 DEBUG { 07:05:22 DEBUG "status": "success", 07:05:22 DEBUG "data": { 07:05:22 DEBUG "resultType": "vector", 07:05:22 DEBUG "result": [ 07:05:22 DEBUG { 07:05:22 DEBUG "metric": { 07:05:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG [...] 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 1712815522, 07:05:22 DEBUG "13747826688" 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG } 07:05:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815522" 07:05:22 INFO [http_cmd]: http status code OK 07:05:22 DEBUG --- status code --- 07:05:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:05:22 DEBUG --- http response --- 07:05:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:05:22 DEBUG ----- output ----- 07:05:22 DEBUG { 07:05:22 DEBUG "status": "success", 07:05:22 DEBUG "data": { 07:05:22 DEBUG "resultType": "vector", 07:05:22 DEBUG "result": [ 07:05:22 DEBUG { 07:05:22 DEBUG "metric": { 07:05:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG [...] 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 1712815522, 07:05:22 DEBUG "0" 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG } 07:05:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815522" 07:05:22 INFO [http_cmd]: http status code OK 07:05:22 DEBUG --- status code --- 07:05:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:05:22 DEBUG --- http response --- 07:05:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:05:22 DEBUG ----- output ----- 07:05:22 DEBUG { 07:05:22 DEBUG "status": "success", 07:05:22 DEBUG "data": { 07:05:22 DEBUG "resultType": "vector", 07:05:22 DEBUG "result": [ 07:05:22 DEBUG { 07:05:22 DEBUG "metric": { 07:05:22 DEBUG "instance": "10.142.0.107:9100" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG [...] 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG "instance": "10.142.0.77:9100" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 1712815522, 07:05:22 DEBUG "0" 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG } 07:05:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815522" 07:05:22 INFO [http_cmd]: http status code OK 07:05:22 DEBUG --- status code --- 07:05:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:05:22 DEBUG --- http response --- 07:05:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:05:22 DEBUG ----- output ----- 07:05:22 DEBUG { 07:05:22 DEBUG "status": "success", 07:05:22 DEBUG "data": { 07:05:22 DEBUG "resultType": "vector", 07:05:22 DEBUG "result": [ 07:05:22 DEBUG { 07:05:22 DEBUG "metric": { 07:05:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG [...] 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG 07:05:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:05:22 DEBUG }, 07:05:22 DEBUG "value": [ 07:05:22 DEBUG 1712815522, 07:05:22 DEBUG "1521.9666666666667" 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG ] 07:05:22 DEBUG } 07:05:22 DEBUG } 07:05:34 INFO 07:05:34 INFO [loop_until]: kubectl --namespace=xlou top pods 07:05:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:05:34 INFO [loop_until]: OK (rc = 0) 07:05:34 DEBUG --- stdout --- 07:05:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 10286m 8531Mi am-5b5bd8567b-8h2w4 9798m 8886Mi am-5b5bd8567b-qlkw6 8248m 8444Mi ds-cts-0 7932m 13149Mi ds-cts-1 8237m 13053Mi ds-cts-2 7922m 13398Mi ds-idrepo-0 12269m 5503Mi ds-idrepo-1 387m 5479Mi ds-idrepo-2 380m 5784Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 4m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 1m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4928m 2302Mi 07:05:34 DEBUG --- stderr --- 07:05:34 DEBUG 07:05:38 INFO 07:05:38 INFO [loop_until]: kubectl --namespace=xlou top node 07:05:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:05:38 INFO [loop_until]: OK (rc = 0) 07:05:38 DEBUG --- stdout --- 07:05:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 2729Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 62m 0% 2791Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8755m 29% 10097Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 12732m 42% 3275Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 10249m 34% 10146Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 142m 0% 3862Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10906m 36% 10522Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 441m 2% 6469Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 12422m 78% 6208Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8439m 53% 13991Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 8422m 53% 13855Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 9182m 57% 13783Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 446m 2% 6162Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5083m 31% 6015Mi 10% 07:05:38 DEBUG --- stderr --- 07:05:38 DEBUG 07:06:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815582" 07:06:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712815582" 07:06:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815582" 07:06:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815582" 07:06:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815582" 07:06:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815582" 07:06:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815582" 07:06:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712815582" 07:06:22 INFO [http_cmd]: http status code OK 07:06:22 INFO [http_cmd]: http status code OK 07:06:22 DEBUG --- status code --- 07:06:22 DEBUG --- status code --- 07:06:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:06:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:06:22 DEBUG --- http response --- 07:06:22 DEBUG --- http response --- 07:06:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:06:22 DEBUG ----- output ----- 07:06:22 DEBUG { 07:06:22 DEBUG "status": "success", 07:06:22 DEBUG "data": { 07:06:22 DEBUG "resultType": "vector", 07:06:22 DEBUG "result": [ 07:06:22 DEBUG { 07:06:22 DEBUG "metric": { 07:06:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG [...] 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 1712815582, 07:06:22 DEBUG "0.023" 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG } 07:06:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:06:22 DEBUG ----- output ----- 07:06:22 DEBUG { 07:06:22 DEBUG "status": "success", 07:06:22 DEBUG "data": { 07:06:22 DEBUG "resultType": "vector", 07:06:22 DEBUG "result": [ 07:06:22 DEBUG { 07:06:22 DEBUG "metric": { 07:06:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG [...] 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 1712815582, 07:06:22 DEBUG "272186.5" 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG } 07:06:22 INFO [http_cmd]: http status code OK 07:06:22 ERROR [http_cmd]: ERROR 07:06:22 INFO [http_cmd]: http status code OK 07:06:22 DEBUG --- status code --- 07:06:22 INFO [http_cmd]: http status code OK 07:06:22 INFO [http_cmd]: http status code OK 07:06:22 ERROR --- status code --- 07:06:22 DEBUG --- status code --- 07:06:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:06:22 DEBUG --- status code --- 07:06:22 DEBUG --- status code --- 07:06:22 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:06:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:06:22 DEBUG --- http response --- 07:06:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:06:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:06:22 ERROR --- http response --- 07:06:22 DEBUG --- http response --- 07:06:22 DEBUG --- http response --- 07:06:22 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:06:22 DEBUG --- http response --- 07:06:22 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:06:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 07:06:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 07:06:22 INFO [http_cmd]: http status code OK 07:06:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:06:22 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:06:22 DEBUG ----- output ----- 07:06:22 DEBUG ----- output ----- 07:06:22 DEBUG --- status code --- 07:06:22 DEBUG ----- output ----- 07:06:22 DEBUG { 07:06:22 DEBUG { 07:06:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:06:22 DEBUG { 07:06:22 DEBUG "status": "success", 07:06:22 DEBUG "status": "success", 07:06:22 DEBUG --- http response --- 07:06:22 DEBUG "status": "success", 07:06:22 DEBUG "data": { 07:06:22 DEBUG "data": { 07:06:22 DEBUG "data": { 07:06:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:06:22 DEBUG "resultType": "vector", 07:06:22 DEBUG "resultType": "vector", 07:06:22 DEBUG "resultType": "vector", 07:06:22 DEBUG ----- output ----- 07:06:22 DEBUG "result": [ 07:06:22 DEBUG "result": [ 07:06:22 DEBUG "result": [ 07:06:22 DEBUG { 07:06:22 DEBUG { 07:06:22 DEBUG { 07:06:22 DEBUG { 07:06:22 DEBUG "status": "success", 07:06:22 DEBUG "metric": { 07:06:22 DEBUG "metric": { 07:06:22 DEBUG "metric": { 07:06:22 DEBUG "data": { 07:06:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:06:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:06:22 DEBUG "instance": "10.142.0.107:9100" 07:06:22 DEBUG "resultType": "vector", 07:06:22 DEBUG }, 07:06:22 DEBUG }, 07:06:22 DEBUG }, 07:06:22 DEBUG "result": [ 07:06:22 DEBUG "value": [ 07:06:22 DEBUG "value": [ 07:06:22 DEBUG "value": [ 07:06:22 DEBUG { 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG "metric": { 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG "pod": "ds-idrepo-2" 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG }, 07:06:22 DEBUG [...] 07:06:22 DEBUG [...] 07:06:22 DEBUG [...] 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG "pod": "idm-668d898cfd-st2t7" 07:06:22 DEBUG "pod": "idm-668d898cfd-st2t7" 07:06:22 DEBUG "instance": "10.142.0.77:9100" 07:06:22 DEBUG [...] 07:06:22 DEBUG }, 07:06:22 DEBUG }, 07:06:22 DEBUG }, 07:06:22 DEBUG 07:06:22 DEBUG "value": [ 07:06:22 DEBUG "value": [ 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 07:06:22 DEBUG 1712815582, 07:06:22 DEBUG 1712815582, 07:06:22 DEBUG 1712815582, 07:06:22 DEBUG 07:06:22 DEBUG "0.0031297917402042623" 07:06:22 DEBUG "18217.296152488532" 07:06:22 DEBUG "0.00220000000000032" 07:06:22 DEBUG "pod": "ds-idrepo-0" 07:06:22 DEBUG ] 07:06:22 DEBUG ] 07:06:22 DEBUG ] 07:06:22 DEBUG }, 07:06:22 DEBUG } 07:06:22 DEBUG } 07:06:22 DEBUG } 07:06:22 DEBUG "value": [ 07:06:22 DEBUG ] 07:06:22 DEBUG ] 07:06:22 DEBUG ] 07:06:22 DEBUG 1712815582, 07:06:22 DEBUG } 07:06:22 DEBUG } 07:06:22 DEBUG } 07:06:22 DEBUG "0" 07:06:22 DEBUG } 07:06:22 DEBUG } 07:06:22 DEBUG } 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG } 07:06:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815582" 07:06:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712815582" 07:06:22 INFO [http_cmd]: http status code OK 07:06:22 DEBUG --- status code --- 07:06:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:06:22 DEBUG --- http response --- 07:06:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:06:22 DEBUG ----- output ----- 07:06:22 DEBUG { 07:06:22 DEBUG "status": "success", 07:06:22 DEBUG "data": { 07:06:22 DEBUG "resultType": "vector", 07:06:22 DEBUG "result": [ 07:06:22 DEBUG { 07:06:22 DEBUG "metric": { 07:06:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG [...] 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 1712815582, 07:06:22 DEBUG "1420.8" 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG } 07:06:22 INFO [http_cmd]: http status code OK 07:06:22 DEBUG --- status code --- 07:06:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:06:22 DEBUG --- http response --- 07:06:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:06:22 DEBUG ----- output ----- 07:06:22 DEBUG { 07:06:22 DEBUG "status": "success", 07:06:22 DEBUG "data": { 07:06:22 DEBUG "resultType": "vector", 07:06:22 DEBUG "result": [ 07:06:22 DEBUG { 07:06:22 DEBUG "metric": { 07:06:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG [...] 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 1712815582, 07:06:22 DEBUG "7.270083329811758" 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG } 07:06:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712815582" 07:06:22 INFO [http_cmd]: http status code OK 07:06:22 DEBUG --- status code --- 07:06:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:06:22 DEBUG --- http response --- 07:06:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:06:22 DEBUG ----- output ----- 07:06:22 DEBUG { 07:06:22 DEBUG "status": "success", 07:06:22 DEBUG "data": { 07:06:22 DEBUG "resultType": "vector", 07:06:22 DEBUG "result": [ 07:06:22 DEBUG { 07:06:22 DEBUG "metric": { 07:06:22 DEBUG "pod": "ds-idrepo-2" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG [...] 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG "pod": "ds-cts-2" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 1712815582, 07:06:22 DEBUG "0" 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG } 07:06:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815582" 07:06:22 INFO [http_cmd]: http status code OK 07:06:22 DEBUG --- status code --- 07:06:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:06:22 DEBUG --- http response --- 07:06:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:06:22 DEBUG ----- output ----- 07:06:22 DEBUG { 07:06:22 DEBUG "status": "success", 07:06:22 DEBUG "data": { 07:06:22 DEBUG "resultType": "vector", 07:06:22 DEBUG "result": [ 07:06:22 DEBUG { 07:06:22 DEBUG "metric": { 07:06:22 DEBUG "instance": "10.142.0.107:9100" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG [...] 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG "instance": "10.142.0.77:9100" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 1712815582, 07:06:22 DEBUG "88521034.63333333" 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG } 07:06:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815582" 07:06:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815582" 07:06:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815582" 07:06:22 INFO [http_cmd]: http status code OK 07:06:22 INFO [http_cmd]: http status code OK 07:06:22 DEBUG --- status code --- 07:06:22 DEBUG --- status code --- 07:06:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:06:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:06:22 DEBUG --- http response --- 07:06:22 DEBUG --- http response --- 07:06:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:06:22 DEBUG ----- output ----- 07:06:22 DEBUG { 07:06:22 DEBUG "status": "success", 07:06:22 DEBUG "data": { 07:06:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 07:06:22 DEBUG "resultType": "vector", 07:06:22 DEBUG ----- output ----- 07:06:22 DEBUG "result": [ 07:06:22 DEBUG { 07:06:22 DEBUG { 07:06:22 DEBUG "status": "success", 07:06:22 DEBUG "metric": { 07:06:22 DEBUG "data": { 07:06:22 DEBUG "instance": "10.142.0.107:9100" 07:06:22 DEBUG "resultType": "vector", 07:06:22 DEBUG }, 07:06:22 DEBUG "result": [ 07:06:22 DEBUG "value": [ 07:06:22 DEBUG { 07:06:22 DEBUG 07:06:22 DEBUG "metric": { 07:06:22 DEBUG 07:06:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:06:22 DEBUG 07:06:22 DEBUG }, 07:06:22 DEBUG [...] 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG "instance": "10.142.0.77:9100" 07:06:22 DEBUG 07:06:22 DEBUG }, 07:06:22 DEBUG [...] 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 07:06:22 DEBUG 1712815582, 07:06:22 DEBUG 07:06:22 DEBUG "52983985152" 07:06:22 DEBUG 07:06:22 DEBUG ] 07:06:22 DEBUG "pod": "idm-668d898cfd-st2t7" 07:06:22 DEBUG } 07:06:22 DEBUG }, 07:06:22 DEBUG ] 07:06:22 DEBUG "value": [ 07:06:22 DEBUG } 07:06:22 DEBUG 1712815582, 07:06:22 DEBUG } 07:06:22 DEBUG "0" 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG } 07:06:22 INFO [http_cmd]: http status code OK 07:06:22 DEBUG --- status code --- 07:06:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:06:22 DEBUG --- http response --- 07:06:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:06:22 DEBUG ----- output ----- 07:06:22 DEBUG { 07:06:22 DEBUG "status": "success", 07:06:22 DEBUG "data": { 07:06:22 DEBUG "resultType": "vector", 07:06:22 DEBUG "result": [ 07:06:22 DEBUG { 07:06:22 DEBUG "metric": { 07:06:22 DEBUG "instance": "10.142.0.107:9100" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG [...] 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG "instance": "10.142.0.77:9100" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 1712815582, 07:06:22 DEBUG "88523388.63333333" 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG } 07:06:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815582" 07:06:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815582" 07:06:22 INFO [http_cmd]: http status code OK 07:06:22 DEBUG --- status code --- 07:06:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:06:22 DEBUG --- http response --- 07:06:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 07:06:22 DEBUG ----- output ----- 07:06:22 DEBUG { 07:06:22 DEBUG "status": "success", 07:06:22 DEBUG "data": { 07:06:22 DEBUG "resultType": "vector", 07:06:22 DEBUG "result": [ 07:06:22 DEBUG { 07:06:22 DEBUG "metric": { 07:06:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG [...] 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG "pod": "idm-668d898cfd-st2t7" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 1712815582, 07:06:22 DEBUG "578.3268619837629" 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG } 07:06:22 INFO [http_cmd]: http status code OK 07:06:22 DEBUG --- status code --- 07:06:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:06:22 DEBUG --- http response --- 07:06:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:06:22 DEBUG ----- output ----- 07:06:22 DEBUG { 07:06:22 DEBUG "status": "success", 07:06:22 DEBUG "data": { 07:06:22 DEBUG "resultType": "vector", 07:06:22 DEBUG "result": [ 07:06:22 DEBUG { 07:06:22 DEBUG "metric": { 07:06:22 DEBUG "pod": "ds-idrepo-2" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG [...] 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG "pod": "ds-idrepo-0" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 1712815582, 07:06:22 DEBUG "0" 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG } 07:06:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815582" 07:06:22 INFO [http_cmd]: http status code OK 07:06:22 DEBUG --- status code --- 07:06:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:06:22 DEBUG --- http response --- 07:06:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:06:22 DEBUG ----- output ----- 07:06:22 DEBUG { 07:06:22 DEBUG "status": "success", 07:06:22 DEBUG "data": { 07:06:22 DEBUG "resultType": "vector", 07:06:22 DEBUG "result": [ 07:06:22 DEBUG { 07:06:22 DEBUG "metric": { 07:06:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG [...] 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 1712815582, 07:06:22 DEBUG "1403.0666666666666" 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG } 07:06:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815582" 07:06:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815582" 07:06:22 INFO [http_cmd]: http status code OK 07:06:22 DEBUG --- status code --- 07:06:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:06:22 DEBUG --- http response --- 07:06:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:06:22 DEBUG ----- output ----- 07:06:22 DEBUG { 07:06:22 DEBUG "status": "success", 07:06:22 DEBUG "data": { 07:06:22 DEBUG "resultType": "vector", 07:06:22 DEBUG "result": [ 07:06:22 DEBUG { 07:06:22 DEBUG "metric": { 07:06:22 DEBUG "pod": "ds-idrepo-2" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG [...] 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG "pod": "ds-cts-2" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 1712815582, 07:06:22 DEBUG "29.466666666666665" 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG } 07:06:22 INFO [http_cmd]: http status code OK 07:06:22 DEBUG --- status code --- 07:06:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:06:22 DEBUG --- http response --- 07:06:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:06:22 DEBUG ----- output ----- 07:06:22 DEBUG { 07:06:22 DEBUG "status": "success", 07:06:22 DEBUG "data": { 07:06:22 DEBUG "resultType": "vector", 07:06:22 DEBUG "result": [ 07:06:22 DEBUG { 07:06:22 DEBUG "metric": { 07:06:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG [...] 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 1712815582, 07:06:22 DEBUG "1403.0333333333333" 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG } 07:06:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815582" 07:06:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815582" 07:06:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815582" 07:06:22 INFO [http_cmd]: http status code OK 07:06:22 DEBUG --- status code --- 07:06:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:06:22 DEBUG --- http response --- 07:06:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:06:22 DEBUG ----- output ----- 07:06:22 DEBUG { 07:06:22 DEBUG "status": "success", 07:06:22 DEBUG "data": { 07:06:22 DEBUG "resultType": "vector", 07:06:22 DEBUG "result": [ 07:06:22 DEBUG { 07:06:22 DEBUG "metric": { 07:06:22 DEBUG "pod": "ds-idrepo-2" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG [...] 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG "pod": "ds-idrepo-0" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 1712815582, 07:06:22 DEBUG "0" 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG } 07:06:22 INFO [http_cmd]: http status code OK 07:06:22 DEBUG --- status code --- 07:06:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:06:22 DEBUG --- http response --- 07:06:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:06:22 DEBUG ----- output ----- 07:06:22 DEBUG { 07:06:22 DEBUG "status": "success", 07:06:22 DEBUG "data": { 07:06:22 DEBUG "resultType": "vector", 07:06:22 DEBUG "result": [ 07:06:22 DEBUG { 07:06:22 DEBUG "metric": { 07:06:22 DEBUG "pod": "ds-idrepo-2" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG [...] 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG "pod": "ds-cts-2" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 1712815582, 07:06:22 DEBUG "0.000599165" 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG } 07:06:22 INFO [http_cmd]: http status code OK 07:06:22 DEBUG --- status code --- 07:06:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:06:22 DEBUG --- http response --- 07:06:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:06:22 DEBUG ----- output ----- 07:06:22 DEBUG { 07:06:22 DEBUG "status": "success", 07:06:22 DEBUG "data": { 07:06:22 DEBUG "resultType": "vector", 07:06:22 DEBUG "result": [ 07:06:22 DEBUG { 07:06:22 DEBUG "metric": { 07:06:22 DEBUG "pod": "ds-idrepo-2" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG [...] 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG "pod": "ds-cts-2" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 1712815582, 07:06:22 DEBUG "0" 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG } 07:06:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712815582" 07:06:22 INFO [http_cmd]: http status code OK 07:06:22 DEBUG --- status code --- 07:06:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:06:22 DEBUG --- http response --- 07:06:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:06:22 DEBUG ----- output ----- 07:06:22 DEBUG { 07:06:22 DEBUG "status": "success", 07:06:22 DEBUG "data": { 07:06:22 DEBUG "resultType": "vector", 07:06:22 DEBUG "result": [ 07:06:22 DEBUG { 07:06:22 DEBUG "metric": { 07:06:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG [...] 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 1712815582, 07:06:22 DEBUG "14145404928" 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG } 07:06:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815582" 07:06:22 INFO [http_cmd]: http status code OK 07:06:22 DEBUG --- status code --- 07:06:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:06:22 DEBUG --- http response --- 07:06:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:06:22 DEBUG ----- output ----- 07:06:22 DEBUG { 07:06:22 DEBUG "status": "success", 07:06:22 DEBUG "data": { 07:06:22 DEBUG "resultType": "vector", 07:06:22 DEBUG "result": [ 07:06:22 DEBUG { 07:06:22 DEBUG "metric": { 07:06:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG [...] 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG 07:06:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:06:22 DEBUG }, 07:06:22 DEBUG "value": [ 07:06:22 DEBUG 1712815582, 07:06:22 DEBUG "0" 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG ] 07:06:22 DEBUG } 07:06:22 DEBUG } 07:06:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815583" 07:06:23 INFO [http_cmd]: http status code OK 07:06:23 DEBUG --- status code --- 07:06:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:06:23 DEBUG --- http response --- 07:06:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:06:23 DEBUG ----- output ----- 07:06:23 DEBUG { 07:06:23 DEBUG "status": "success", 07:06:23 DEBUG "data": { 07:06:23 DEBUG "resultType": "vector", 07:06:23 DEBUG "result": [ 07:06:23 DEBUG { 07:06:23 DEBUG "metric": { 07:06:23 DEBUG "instance": "10.142.0.107:9100" 07:06:23 DEBUG }, 07:06:23 DEBUG "value": [ 07:06:23 DEBUG 07:06:23 DEBUG 07:06:23 DEBUG 07:06:23 DEBUG [...] 07:06:23 DEBUG 07:06:23 DEBUG 07:06:23 DEBUG 07:06:23 DEBUG "instance": "10.142.0.77:9100" 07:06:23 DEBUG }, 07:06:23 DEBUG "value": [ 07:06:23 DEBUG 1712815583, 07:06:23 DEBUG "0.0006666666666662973" 07:06:23 DEBUG ] 07:06:23 DEBUG } 07:06:23 DEBUG ] 07:06:23 DEBUG } 07:06:23 DEBUG } 07:06:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815583" 07:06:23 INFO [http_cmd]: http status code OK 07:06:23 DEBUG --- status code --- 07:06:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:06:23 DEBUG --- http response --- 07:06:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:06:23 DEBUG ----- output ----- 07:06:23 DEBUG { 07:06:23 DEBUG "status": "success", 07:06:23 DEBUG "data": { 07:06:23 DEBUG "resultType": "vector", 07:06:23 DEBUG "result": [ 07:06:23 DEBUG { 07:06:23 DEBUG "metric": { 07:06:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:06:23 DEBUG }, 07:06:23 DEBUG "value": [ 07:06:23 DEBUG 07:06:23 DEBUG 07:06:23 DEBUG 07:06:23 DEBUG [...] 07:06:23 DEBUG 07:06:23 DEBUG 07:06:23 DEBUG 07:06:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:06:23 DEBUG }, 07:06:23 DEBUG "value": [ 07:06:23 DEBUG 1712815583, 07:06:23 DEBUG "1382.4333333333334" 07:06:23 DEBUG ] 07:06:23 DEBUG } 07:06:23 DEBUG ] 07:06:23 DEBUG } 07:06:23 DEBUG } 07:06:34 INFO 07:06:34 INFO [loop_until]: kubectl --namespace=xlou top pods 07:06:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:06:35 INFO [loop_until]: OK (rc = 0) 07:06:35 DEBUG --- stdout --- 07:06:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 10203m 8560Mi am-5b5bd8567b-8h2w4 10327m 8866Mi am-5b5bd8567b-qlkw6 8023m 8495Mi ds-cts-0 9055m 13455Mi ds-cts-1 9267m 13404Mi ds-cts-2 9174m 13405Mi ds-idrepo-0 12178m 5503Mi ds-idrepo-1 360m 5479Mi ds-idrepo-2 364m 5784Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 3m 1010Mi idm-668d898cfd-st2t7 4m 1136Mi lodemon-ccc964465-p7gdm 4m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 7875m 4309Mi 07:06:35 DEBUG --- stderr --- 07:06:35 DEBUG 07:06:39 INFO 07:06:39 INFO [loop_until]: kubectl --namespace=xlou top node 07:06:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:06:39 INFO [loop_until]: OK (rc = 0) 07:06:39 DEBUG --- stdout --- 07:06:39 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 2722Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 63m 0% 2785Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8552m 28% 10165Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14299m 47% 3721Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 11203m 37% 10222Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 139m 0% 3865Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11389m 38% 10525Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 425m 2% 6473Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 12177m 76% 6207Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 9394m 59% 13999Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 8907m 56% 14177Mi 24% gke-xlou-cdm-ds-32e4dcb1-j2hk 9447m 59% 14048Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 417m 2% 6157Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 8189m 51% 7917Mi 13% 07:06:39 DEBUG --- stderr --- 07:06:39 DEBUG 07:07:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815642" 07:07:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815642" 07:07:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712815642" 07:07:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712815642" 07:07:22 INFO [http_cmd]: http status code OK 07:07:22 DEBUG --- status code --- 07:07:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:07:22 DEBUG --- http response --- 07:07:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:07:22 DEBUG ----- output ----- 07:07:22 DEBUG { 07:07:22 DEBUG "status": "success", 07:07:22 DEBUG "data": { 07:07:22 DEBUG "resultType": "vector", 07:07:22 DEBUG "result": [ 07:07:22 DEBUG { 07:07:22 DEBUG "metric": { 07:07:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG [...] 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 1712815642, 07:07:22 DEBUG "263534" 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG } 07:07:22 ERROR [http_cmd]: ERROR 07:07:22 ERROR --- status code --- 07:07:22 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:07:22 ERROR --- http response --- 07:07:22 INFO [http_cmd]: http status code OK 07:07:22 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:07:22 DEBUG --- status code --- 07:07:22 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:07:22 INFO [http_cmd]: http status code OK 07:07:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:07:22 DEBUG --- status code --- 07:07:22 DEBUG --- http response --- 07:07:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:07:22 DEBUG --- http response --- 07:07:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 07:07:22 DEBUG ----- output ----- 07:07:22 DEBUG { 07:07:22 DEBUG "status": "success", 07:07:22 DEBUG "data": { 07:07:22 DEBUG "resultType": "vector", 07:07:22 DEBUG "result": [ 07:07:22 DEBUG { 07:07:22 DEBUG "metric": { 07:07:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG [...] 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG "pod": "ds-cts-0" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 1712815642, 07:07:22 DEBUG "8.756025483217469" 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG } 07:07:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:07:22 DEBUG ----- output ----- 07:07:22 DEBUG { 07:07:22 DEBUG "status": "success", 07:07:22 DEBUG "data": { 07:07:22 DEBUG "resultType": "vector", 07:07:22 DEBUG "result": [ 07:07:22 DEBUG { 07:07:22 DEBUG "metric": { 07:07:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG [...] 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 1712815642, 07:07:22 DEBUG "8.033222213163956" 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG } 07:07:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712815642" 07:07:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815642" 07:07:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815642" 07:07:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815642" 07:07:22 INFO [http_cmd]: http status code OK 07:07:22 DEBUG --- status code --- 07:07:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:07:22 DEBUG --- http response --- 07:07:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:07:22 DEBUG ----- output ----- 07:07:22 DEBUG { 07:07:22 DEBUG "status": "success", 07:07:22 DEBUG "data": { 07:07:22 DEBUG "resultType": "vector", 07:07:22 DEBUG "result": [ 07:07:22 DEBUG { 07:07:22 DEBUG "metric": { 07:07:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG [...] 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 1712815642, 07:07:22 DEBUG "0.023" 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG } 07:07:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815642" 07:07:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815642" 07:07:22 INFO [http_cmd]: http status code OK 07:07:22 DEBUG --- status code --- 07:07:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:07:22 DEBUG --- http response --- 07:07:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:07:22 DEBUG ----- output ----- 07:07:22 DEBUG { 07:07:22 DEBUG "status": "success", 07:07:22 DEBUG "data": { 07:07:22 DEBUG "resultType": "vector", 07:07:22 DEBUG "result": [ 07:07:22 DEBUG { 07:07:22 DEBUG "metric": { 07:07:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG [...] 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 1712815642, 07:07:22 DEBUG "1520.3333333333333" 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG } 07:07:22 INFO [http_cmd]: http status code OK 07:07:22 DEBUG --- status code --- 07:07:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:07:22 DEBUG --- http response --- 07:07:22 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:07:22 INFO [http_cmd]: http status code OK 07:07:22 DEBUG --- status code --- 07:07:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:07:22 DEBUG --- http response --- 07:07:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 07:07:22 DEBUG ----- output ----- 07:07:22 DEBUG { 07:07:22 DEBUG "status": "success", 07:07:22 DEBUG "data": { 07:07:22 DEBUG "resultType": "vector", 07:07:22 DEBUG "result": [ 07:07:22 DEBUG { 07:07:22 DEBUG "metric": { 07:07:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG [...] 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG "pod": "ds-cts-0" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 1712815642, 07:07:22 DEBUG "447549559.5785628" 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG } 07:07:22 INFO [http_cmd]: http status code OK 07:07:22 INFO [http_cmd]: http status code OK 07:07:22 DEBUG --- status code --- 07:07:22 DEBUG --- status code --- 07:07:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:07:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:07:22 DEBUG --- http response --- 07:07:22 DEBUG --- http response --- 07:07:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:07:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:07:22 DEBUG ----- output ----- 07:07:22 DEBUG ----- output ----- 07:07:22 DEBUG { 07:07:22 DEBUG { 07:07:22 DEBUG "status": "success", 07:07:22 DEBUG "status": "success", 07:07:22 DEBUG "data": { 07:07:22 DEBUG "data": { 07:07:22 DEBUG "resultType": "vector", 07:07:22 DEBUG "resultType": "vector", 07:07:22 DEBUG "result": [ 07:07:22 DEBUG "result": [ 07:07:22 DEBUG { 07:07:22 DEBUG { 07:07:22 DEBUG "metric": { 07:07:22 DEBUG "metric": { 07:07:22 DEBUG "instance": "10.142.0.107:9100" 07:07:22 DEBUG "pod": "ds-idrepo-2" 07:07:22 DEBUG }, 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG [...] 07:07:22 DEBUG [...] 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG "instance": "10.142.0.77:9100" 07:07:22 DEBUG "pod": "ds-idrepo-0" 07:07:22 DEBUG }, 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 1712815642, 07:07:22 DEBUG 1712815642, 07:07:22 DEBUG "0.002166666666666828" 07:07:22 DEBUG "0" 07:07:22 DEBUG ] 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG } 07:07:22 DEBUG ] 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG } 07:07:22 DEBUG } 07:07:22 DEBUG } 07:07:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712815642" 07:07:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815642" 07:07:22 INFO [http_cmd]: http status code OK 07:07:22 DEBUG --- status code --- 07:07:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:07:22 DEBUG --- http response --- 07:07:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:07:22 DEBUG ----- output ----- 07:07:22 DEBUG { 07:07:22 DEBUG "status": "success", 07:07:22 DEBUG "data": { 07:07:22 DEBUG "resultType": "vector", 07:07:22 DEBUG "result": [ 07:07:22 DEBUG { 07:07:22 DEBUG "metric": { 07:07:22 DEBUG "pod": "ds-idrepo-2" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG [...] 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG "pod": "ds-cts-2" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 1712815642, 07:07:22 DEBUG "0" 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG } 07:07:22 INFO [http_cmd]: http status code OK 07:07:22 DEBUG --- status code --- 07:07:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:07:22 DEBUG --- http response --- 07:07:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:07:22 DEBUG ----- output ----- 07:07:22 DEBUG { 07:07:22 DEBUG "status": "success", 07:07:22 DEBUG "data": { 07:07:22 DEBUG "resultType": "vector", 07:07:22 DEBUG "result": [ 07:07:22 DEBUG { 07:07:22 DEBUG "metric": { 07:07:22 DEBUG "instance": "10.142.0.107:9100" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG [...] 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG "instance": "10.142.0.77:9100" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 1712815642, 07:07:22 DEBUG "84005878.33333333" 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG } 07:07:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815642" 07:07:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815642" 07:07:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815642" 07:07:22 INFO [http_cmd]: http status code OK 07:07:22 INFO [http_cmd]: http status code OK 07:07:22 DEBUG --- status code --- 07:07:22 DEBUG --- status code --- 07:07:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:07:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:07:22 DEBUG --- http response --- 07:07:22 DEBUG --- http response --- 07:07:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:07:22 DEBUG ----- output ----- 07:07:22 DEBUG { 07:07:22 DEBUG "status": "success", 07:07:22 DEBUG "data": { 07:07:22 DEBUG "resultType": "vector", 07:07:22 DEBUG "result": [ 07:07:22 DEBUG { 07:07:22 DEBUG "metric": { 07:07:22 DEBUG "instance": "10.142.0.107:9100" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG [...] 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG "instance": "10.142.0.77:9100" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 1712815642, 07:07:22 DEBUG "52985478144" 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG } 07:07:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 07:07:22 DEBUG ----- output ----- 07:07:22 DEBUG { 07:07:22 DEBUG "status": "success", 07:07:22 DEBUG "data": { 07:07:22 DEBUG "resultType": "vector", 07:07:22 DEBUG "result": [ 07:07:22 DEBUG { 07:07:22 DEBUG "metric": { 07:07:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG [...] 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG "pod": "ds-cts-0" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 1712815642, 07:07:22 DEBUG "32625574.58329256" 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG } 07:07:22 INFO [http_cmd]: http status code OK 07:07:22 DEBUG --- status code --- 07:07:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:07:22 DEBUG --- http response --- 07:07:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:07:22 DEBUG ----- output ----- 07:07:22 DEBUG { 07:07:22 DEBUG "status": "success", 07:07:22 DEBUG "data": { 07:07:22 DEBUG "resultType": "vector", 07:07:22 DEBUG "result": [ 07:07:22 DEBUG { 07:07:22 DEBUG "metric": { 07:07:22 DEBUG "instance": "10.142.0.107:9100" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG [...] 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG "instance": "10.142.0.77:9100" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 1712815642, 07:07:22 DEBUG "84010079.76666667" 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG } 07:07:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815642" 07:07:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815642" 07:07:22 INFO [http_cmd]: http status code OK 07:07:22 DEBUG --- status code --- 07:07:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:07:22 DEBUG --- http response --- 07:07:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:07:22 DEBUG ----- output ----- 07:07:22 DEBUG { 07:07:22 DEBUG "status": "success", 07:07:22 DEBUG "data": { 07:07:22 DEBUG "resultType": "vector", 07:07:22 DEBUG "result": [ 07:07:22 DEBUG { 07:07:22 DEBUG "metric": { 07:07:22 DEBUG "pod": "ds-idrepo-2" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG [...] 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG "pod": "ds-idrepo-0" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 1712815642, 07:07:22 DEBUG "0" 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG } 07:07:22 INFO [http_cmd]: http status code OK 07:07:22 DEBUG --- status code --- 07:07:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:07:22 DEBUG --- http response --- 07:07:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 07:07:22 DEBUG ----- output ----- 07:07:22 DEBUG { 07:07:22 DEBUG "status": "success", 07:07:22 DEBUG "data": { 07:07:22 DEBUG "resultType": "vector", 07:07:22 DEBUG "result": [ 07:07:22 DEBUG { 07:07:22 DEBUG "metric": { 07:07:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG [...] 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG "pod": "ds-cts-0" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 1712815642, 07:07:22 DEBUG "32453087.516717225" 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG } 07:07:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815642" 07:07:22 INFO [http_cmd]: http status code OK 07:07:22 DEBUG --- status code --- 07:07:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:07:22 DEBUG --- http response --- 07:07:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:07:22 DEBUG ----- output ----- 07:07:22 DEBUG { 07:07:22 DEBUG "status": "success", 07:07:22 DEBUG "data": { 07:07:22 DEBUG "resultType": "vector", 07:07:22 DEBUG "result": [ 07:07:22 DEBUG { 07:07:22 DEBUG "metric": { 07:07:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG [...] 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 1712815642, 07:07:22 DEBUG "1357.9333333333334" 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG } 07:07:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815642" 07:07:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815642" 07:07:22 INFO [http_cmd]: http status code OK 07:07:22 DEBUG --- status code --- 07:07:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:07:22 DEBUG --- http response --- 07:07:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:07:22 DEBUG ----- output ----- 07:07:22 DEBUG { 07:07:22 DEBUG "status": "success", 07:07:22 DEBUG "data": { 07:07:22 DEBUG "resultType": "vector", 07:07:22 DEBUG "result": [ 07:07:22 DEBUG { 07:07:22 DEBUG "metric": { 07:07:22 DEBUG "pod": "ds-idrepo-2" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG [...] 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG "pod": "ds-cts-2" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 1712815642, 07:07:22 DEBUG "29.833333333333332" 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG } 07:07:22 INFO [http_cmd]: http status code OK 07:07:22 DEBUG --- status code --- 07:07:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:07:22 DEBUG --- http response --- 07:07:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:07:22 DEBUG ----- output ----- 07:07:22 DEBUG { 07:07:22 DEBUG "status": "success", 07:07:22 DEBUG "data": { 07:07:22 DEBUG "resultType": "vector", 07:07:22 DEBUG "result": [ 07:07:22 DEBUG { 07:07:22 DEBUG "metric": { 07:07:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG [...] 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 1712815642, 07:07:22 DEBUG "1358.3333333333333" 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG } 07:07:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815642" 07:07:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815642" 07:07:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815642" 07:07:22 INFO [http_cmd]: http status code OK 07:07:22 INFO [http_cmd]: http status code OK 07:07:22 DEBUG --- status code --- 07:07:22 DEBUG --- status code --- 07:07:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:07:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:07:22 DEBUG --- http response --- 07:07:22 DEBUG --- http response --- 07:07:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:07:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:07:22 DEBUG ----- output ----- 07:07:22 DEBUG ----- output ----- 07:07:22 DEBUG { 07:07:22 DEBUG { 07:07:22 DEBUG "status": "success", 07:07:22 DEBUG "status": "success", 07:07:22 DEBUG "data": { 07:07:22 DEBUG "data": { 07:07:22 DEBUG "resultType": "vector", 07:07:22 DEBUG "resultType": "vector", 07:07:22 DEBUG "result": [ 07:07:22 DEBUG "result": [ 07:07:22 DEBUG { 07:07:22 DEBUG { 07:07:22 DEBUG "metric": { 07:07:22 INFO [http_cmd]: http status code OK 07:07:22 DEBUG "metric": { 07:07:22 DEBUG "pod": "ds-idrepo-2" 07:07:22 DEBUG --- status code --- 07:07:22 DEBUG "pod": "ds-idrepo-2" 07:07:22 DEBUG }, 07:07:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG --- http response --- 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG ----- output ----- 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG { 07:07:22 DEBUG [...] 07:07:22 DEBUG [...] 07:07:22 DEBUG "status": "success", 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG "data": { 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG "resultType": "vector", 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG "result": [ 07:07:22 DEBUG "pod": "ds-idrepo-0" 07:07:22 DEBUG "pod": "ds-cts-2" 07:07:22 DEBUG { 07:07:22 DEBUG }, 07:07:22 DEBUG }, 07:07:22 DEBUG "metric": { 07:07:22 DEBUG "value": [ 07:07:22 DEBUG "value": [ 07:07:22 DEBUG "pod": "ds-idrepo-2" 07:07:22 DEBUG 1712815642, 07:07:22 DEBUG 1712815642, 07:07:22 DEBUG }, 07:07:22 DEBUG "0" 07:07:22 DEBUG "0" 07:07:22 DEBUG "value": [ 07:07:22 DEBUG ] 07:07:22 DEBUG ] 07:07:22 DEBUG 07:07:22 DEBUG } 07:07:22 DEBUG } 07:07:22 DEBUG 07:07:22 DEBUG ] 07:07:22 DEBUG ] 07:07:22 DEBUG 07:07:22 DEBUG } 07:07:22 DEBUG } 07:07:22 DEBUG [...] 07:07:22 DEBUG } 07:07:22 DEBUG } 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG 07:07:22 DEBUG "pod": "ds-cts-2" 07:07:22 DEBUG }, 07:07:22 DEBUG "value": [ 07:07:22 DEBUG 1712815642, 07:07:22 DEBUG "0" 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG ] 07:07:22 DEBUG } 07:07:22 DEBUG } 07:07:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712815642" 07:07:23 INFO [http_cmd]: http status code OK 07:07:23 DEBUG --- status code --- 07:07:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:07:23 DEBUG --- http response --- 07:07:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:07:23 DEBUG ----- output ----- 07:07:23 DEBUG { 07:07:23 DEBUG "status": "success", 07:07:23 DEBUG "data": { 07:07:23 DEBUG "resultType": "vector", 07:07:23 DEBUG "result": [ 07:07:23 DEBUG { 07:07:23 DEBUG "metric": { 07:07:23 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:07:23 DEBUG }, 07:07:23 DEBUG "value": [ 07:07:23 DEBUG 07:07:23 DEBUG 07:07:23 DEBUG 07:07:23 DEBUG [...] 07:07:23 DEBUG 07:07:23 DEBUG 07:07:23 DEBUG 07:07:23 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 07:07:23 DEBUG }, 07:07:23 DEBUG "value": [ 07:07:23 DEBUG 1712815642, 07:07:23 DEBUG "14145531904" 07:07:23 DEBUG ] 07:07:23 DEBUG } 07:07:23 DEBUG ] 07:07:23 DEBUG } 07:07:23 DEBUG } 07:07:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815643" 07:07:23 INFO [http_cmd]: http status code OK 07:07:23 DEBUG --- status code --- 07:07:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:07:23 DEBUG --- http response --- 07:07:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:07:23 DEBUG ----- output ----- 07:07:23 DEBUG { 07:07:23 DEBUG "status": "success", 07:07:23 DEBUG "data": { 07:07:23 DEBUG "resultType": "vector", 07:07:23 DEBUG "result": [ 07:07:23 DEBUG { 07:07:23 DEBUG "metric": { 07:07:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:07:23 DEBUG }, 07:07:23 DEBUG "value": [ 07:07:23 DEBUG 07:07:23 DEBUG 07:07:23 DEBUG 07:07:23 DEBUG [...] 07:07:23 DEBUG 07:07:23 DEBUG 07:07:23 DEBUG 07:07:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:07:23 DEBUG }, 07:07:23 DEBUG "value": [ 07:07:23 DEBUG 1712815643, 07:07:23 DEBUG "0" 07:07:23 DEBUG ] 07:07:23 DEBUG } 07:07:23 DEBUG ] 07:07:23 DEBUG } 07:07:23 DEBUG } 07:07:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815643" 07:07:23 INFO [http_cmd]: http status code OK 07:07:23 DEBUG --- status code --- 07:07:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:07:23 DEBUG --- http response --- 07:07:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:07:23 DEBUG ----- output ----- 07:07:23 DEBUG { 07:07:23 DEBUG "status": "success", 07:07:23 DEBUG "data": { 07:07:23 DEBUG "resultType": "vector", 07:07:23 DEBUG "result": [ 07:07:23 DEBUG { 07:07:23 DEBUG "metric": { 07:07:23 DEBUG "instance": "10.142.0.107:9100" 07:07:23 DEBUG }, 07:07:23 DEBUG "value": [ 07:07:23 DEBUG 07:07:23 DEBUG 07:07:23 DEBUG 07:07:23 DEBUG [...] 07:07:23 DEBUG 07:07:23 DEBUG 07:07:23 DEBUG 07:07:23 DEBUG "instance": "10.142.0.77:9100" 07:07:23 DEBUG }, 07:07:23 DEBUG "value": [ 07:07:23 DEBUG 1712815643, 07:07:23 DEBUG "0.000666666666666534" 07:07:23 DEBUG ] 07:07:23 DEBUG } 07:07:23 DEBUG ] 07:07:23 DEBUG } 07:07:23 DEBUG } 07:07:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815643" 07:07:23 INFO [http_cmd]: http status code OK 07:07:23 DEBUG --- status code --- 07:07:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:07:23 DEBUG --- http response --- 07:07:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:07:23 DEBUG ----- output ----- 07:07:23 DEBUG { 07:07:23 DEBUG "status": "success", 07:07:23 DEBUG "data": { 07:07:23 DEBUG "resultType": "vector", 07:07:23 DEBUG "result": [ 07:07:23 DEBUG { 07:07:23 DEBUG "metric": { 07:07:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:07:23 DEBUG }, 07:07:23 DEBUG "value": [ 07:07:23 DEBUG 07:07:23 DEBUG 07:07:23 DEBUG 07:07:23 DEBUG [...] 07:07:23 DEBUG 07:07:23 DEBUG 07:07:23 DEBUG 07:07:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:07:23 DEBUG }, 07:07:23 DEBUG "value": [ 07:07:23 DEBUG 1712815643, 07:07:23 DEBUG "1329.5333333333333" 07:07:23 DEBUG ] 07:07:23 DEBUG } 07:07:23 DEBUG ] 07:07:23 DEBUG } 07:07:23 DEBUG } 07:07:35 INFO 07:07:35 INFO [loop_until]: kubectl --namespace=xlou top pods 07:07:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:07:35 INFO [loop_until]: OK (rc = 0) 07:07:35 DEBUG --- stdout --- 07:07:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 9518m 8835Mi am-5b5bd8567b-8h2w4 8942m 8864Mi am-5b5bd8567b-qlkw6 7797m 8555Mi ds-cts-0 8722m 13119Mi ds-cts-1 9142m 13405Mi ds-cts-2 9137m 13658Mi ds-idrepo-0 11495m 5503Mi ds-idrepo-1 328m 5479Mi ds-idrepo-2 364m 5784Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 4m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 3m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 7939m 4641Mi 07:07:35 DEBUG --- stderr --- 07:07:35 DEBUG 07:07:39 INFO 07:07:39 INFO [loop_until]: kubectl --namespace=xlou top node 07:07:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:07:39 INFO [loop_until]: OK (rc = 0) 07:07:39 DEBUG --- stdout --- 07:07:39 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 2729Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 63m 0% 2785Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8074m 27% 10238Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 13191m 44% 3645Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 10348m 34% 10463Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 136m 0% 3849Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10222m 34% 10524Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 398m 2% 6470Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 11266m 70% 6209Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 9196m 57% 14205Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 8590m 54% 13884Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 9101m 57% 14032Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 437m 2% 6162Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 8170m 51% 8209Mi 13% 07:07:39 DEBUG --- stderr --- 07:07:39 DEBUG 07:08:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815702" 07:08:22 INFO [http_cmd]: http status code OK 07:08:22 DEBUG --- status code --- 07:08:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:08:22 DEBUG --- http response --- 07:08:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:08:22 DEBUG ----- output ----- 07:08:22 DEBUG { 07:08:22 DEBUG "status": "success", 07:08:22 DEBUG "data": { 07:08:22 DEBUG "resultType": "vector", 07:08:22 DEBUG "result": [ 07:08:22 DEBUG { 07:08:22 DEBUG "metric": { 07:08:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:08:22 DEBUG }, 07:08:22 DEBUG "value": [ 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG [...] 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:08:22 DEBUG }, 07:08:22 DEBUG "value": [ 07:08:22 DEBUG 1712815702, 07:08:22 DEBUG "249517" 07:08:22 DEBUG ] 07:08:22 DEBUG } 07:08:22 DEBUG ] 07:08:22 DEBUG } 07:08:22 DEBUG } 07:08:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815702" 07:08:22 INFO [http_cmd]: http status code OK 07:08:22 DEBUG --- status code --- 07:08:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:08:22 DEBUG --- http response --- 07:08:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:08:22 DEBUG ----- output ----- 07:08:22 DEBUG { 07:08:22 DEBUG "status": "success", 07:08:22 DEBUG "data": { 07:08:22 DEBUG "resultType": "vector", 07:08:22 DEBUG "result": [ 07:08:22 DEBUG { 07:08:22 DEBUG "metric": { 07:08:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:08:22 DEBUG }, 07:08:22 DEBUG "value": [ 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG [...] 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:08:22 DEBUG }, 07:08:22 DEBUG "value": [ 07:08:22 DEBUG 1712815702, 07:08:22 DEBUG "5.5829040370985075" 07:08:22 DEBUG ] 07:08:22 DEBUG } 07:08:22 DEBUG ] 07:08:22 DEBUG } 07:08:22 DEBUG } 07:08:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815702" 07:08:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712815702" 07:08:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712815702" 07:08:22 INFO [http_cmd]: http status code OK 07:08:22 DEBUG --- status code --- 07:08:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:08:22 DEBUG --- http response --- 07:08:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:08:22 DEBUG ----- output ----- 07:08:22 DEBUG { 07:08:22 DEBUG "status": "success", 07:08:22 DEBUG "data": { 07:08:22 DEBUG "resultType": "vector", 07:08:22 DEBUG "result": [ 07:08:22 DEBUG { 07:08:22 DEBUG "metric": { 07:08:22 DEBUG "pod": "ds-idrepo-2" 07:08:22 DEBUG }, 07:08:22 DEBUG "value": [ 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG [...] 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG "pod": "ds-idrepo-0" 07:08:22 DEBUG }, 07:08:22 DEBUG "value": [ 07:08:22 DEBUG 1712815702, 07:08:22 DEBUG "0" 07:08:22 DEBUG ] 07:08:22 DEBUG } 07:08:22 DEBUG ] 07:08:22 DEBUG } 07:08:22 DEBUG } 07:08:22 ERROR [http_cmd]: ERROR 07:08:22 ERROR --- status code --- 07:08:22 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:08:22 ERROR --- http response --- 07:08:22 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:08:22 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:08:22 INFO [http_cmd]: http status code OK 07:08:22 DEBUG --- status code --- 07:08:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:08:22 DEBUG --- http response --- 07:08:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:08:22 DEBUG ----- output ----- 07:08:22 DEBUG { 07:08:22 DEBUG "status": "success", 07:08:22 DEBUG "data": { 07:08:22 DEBUG "resultType": "vector", 07:08:22 DEBUG "result": [ 07:08:22 DEBUG { 07:08:22 DEBUG "metric": { 07:08:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:08:22 DEBUG }, 07:08:22 DEBUG "value": [ 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG [...] 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:08:22 DEBUG }, 07:08:22 DEBUG "value": [ 07:08:22 DEBUG 1712815702, 07:08:22 DEBUG "8.035867189680861" 07:08:22 DEBUG ] 07:08:22 DEBUG } 07:08:22 DEBUG ] 07:08:22 DEBUG } 07:08:22 DEBUG } 07:08:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712815702" 07:08:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815702" 07:08:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815702" 07:08:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815702" 07:08:22 INFO [http_cmd]: http status code OK 07:08:22 INFO [http_cmd]: http status code OK 07:08:22 DEBUG --- status code --- 07:08:22 DEBUG --- status code --- 07:08:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:08:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815702" 07:08:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:08:22 DEBUG --- http response --- 07:08:22 DEBUG --- http response --- 07:08:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:08:22 DEBUG ----- output ----- 07:08:22 DEBUG { 07:08:22 DEBUG "status": "success", 07:08:22 DEBUG "data": { 07:08:22 DEBUG "resultType": "vector", 07:08:22 DEBUG "result": [ 07:08:22 DEBUG { 07:08:22 DEBUG "metric": { 07:08:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:08:22 DEBUG }, 07:08:22 DEBUG "value": [ 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG [...] 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:08:22 DEBUG }, 07:08:22 DEBUG "value": [ 07:08:22 DEBUG 1712815702, 07:08:22 DEBUG "0.023" 07:08:22 DEBUG ] 07:08:22 DEBUG } 07:08:22 DEBUG ] 07:08:22 DEBUG } 07:08:22 DEBUG } 07:08:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:08:22 INFO [http_cmd]: http status code OK 07:08:22 DEBUG ----- output ----- 07:08:22 DEBUG --- status code --- 07:08:22 DEBUG { 07:08:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:08:22 DEBUG "status": "success", 07:08:22 DEBUG --- http response --- 07:08:22 DEBUG "data": { 07:08:22 DEBUG "resultType": "vector", 07:08:22 DEBUG "result": [ 07:08:22 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:08:22 DEBUG { 07:08:22 DEBUG "metric": { 07:08:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:08:22 DEBUG }, 07:08:22 DEBUG "value": [ 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG [...] 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:08:22 DEBUG }, 07:08:22 DEBUG "value": [ 07:08:22 DEBUG 1712815702, 07:08:22 DEBUG "1487.8333333333333" 07:08:22 DEBUG ] 07:08:22 DEBUG } 07:08:22 DEBUG ] 07:08:22 DEBUG } 07:08:22 DEBUG } 07:08:22 INFO [http_cmd]: http status code OK 07:08:22 DEBUG --- status code --- 07:08:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:08:22 DEBUG --- http response --- 07:08:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:08:22 DEBUG ----- output ----- 07:08:22 DEBUG { 07:08:22 DEBUG "status": "success", 07:08:22 DEBUG "data": { 07:08:22 DEBUG "resultType": "vector", 07:08:22 DEBUG "result": [ 07:08:22 DEBUG { 07:08:22 DEBUG "metric": { 07:08:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:08:22 DEBUG }, 07:08:22 DEBUG "value": [ 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG [...] 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:08:22 DEBUG }, 07:08:22 DEBUG "value": [ 07:08:22 DEBUG 1712815702, 07:08:22 DEBUG "2618913.477785073" 07:08:22 DEBUG ] 07:08:22 DEBUG } 07:08:22 DEBUG ] 07:08:22 DEBUG } 07:08:22 DEBUG } 07:08:22 INFO [http_cmd]: http status code OK 07:08:22 DEBUG --- status code --- 07:08:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:08:22 DEBUG --- http response --- 07:08:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:08:22 DEBUG ----- output ----- 07:08:22 DEBUG { 07:08:22 DEBUG "status": "success", 07:08:22 DEBUG "data": { 07:08:22 DEBUG "resultType": "vector", 07:08:22 DEBUG "result": [ 07:08:22 DEBUG { 07:08:22 DEBUG "metric": { 07:08:22 DEBUG "instance": "10.142.0.107:9100" 07:08:22 DEBUG }, 07:08:22 DEBUG "value": [ 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG [...] 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG "instance": "10.142.0.77:9100" 07:08:22 DEBUG }, 07:08:22 DEBUG "value": [ 07:08:22 DEBUG 1712815702, 07:08:22 DEBUG "0.002000000000000076" 07:08:22 DEBUG ] 07:08:22 DEBUG } 07:08:22 DEBUG ] 07:08:22 DEBUG } 07:08:22 DEBUG } 07:08:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815702" 07:08:22 INFO [http_cmd]: http status code OK 07:08:22 DEBUG --- status code --- 07:08:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:08:22 DEBUG --- http response --- 07:08:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:08:22 DEBUG ----- output ----- 07:08:22 DEBUG { 07:08:22 DEBUG "status": "success", 07:08:22 DEBUG "data": { 07:08:22 DEBUG "resultType": "vector", 07:08:22 DEBUG "result": [ 07:08:22 DEBUG { 07:08:22 DEBUG "metric": { 07:08:22 DEBUG "instance": "10.142.0.107:9100" 07:08:22 DEBUG }, 07:08:22 DEBUG "value": [ 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG [...] 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG "instance": "10.142.0.77:9100" 07:08:22 DEBUG }, 07:08:22 DEBUG "value": [ 07:08:22 DEBUG 1712815702, 07:08:22 DEBUG "78112296.26666667" 07:08:22 DEBUG ] 07:08:22 DEBUG } 07:08:22 DEBUG ] 07:08:22 DEBUG } 07:08:22 DEBUG } 07:08:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712815702" 07:08:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815702" 07:08:22 INFO [http_cmd]: http status code OK 07:08:22 DEBUG --- status code --- 07:08:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:08:22 DEBUG --- http response --- 07:08:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:08:22 DEBUG ----- output ----- 07:08:22 DEBUG { 07:08:22 DEBUG "status": "success", 07:08:22 DEBUG "data": { 07:08:22 DEBUG "resultType": "vector", 07:08:22 DEBUG "result": [ 07:08:22 DEBUG { 07:08:22 DEBUG "metric": { 07:08:22 DEBUG "pod": "ds-idrepo-2" 07:08:22 DEBUG }, 07:08:22 DEBUG "value": [ 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG [...] 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG "pod": "ds-cts-2" 07:08:22 DEBUG }, 07:08:22 DEBUG "value": [ 07:08:22 DEBUG 1712815702, 07:08:22 DEBUG "0" 07:08:22 DEBUG ] 07:08:22 DEBUG } 07:08:22 DEBUG ] 07:08:22 DEBUG } 07:08:22 DEBUG } 07:08:22 INFO [http_cmd]: http status code OK 07:08:22 DEBUG --- status code --- 07:08:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:08:22 DEBUG --- http response --- 07:08:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:08:22 DEBUG ----- output ----- 07:08:22 DEBUG { 07:08:22 DEBUG "status": "success", 07:08:22 DEBUG "data": { 07:08:22 DEBUG "resultType": "vector", 07:08:22 DEBUG "result": [ 07:08:22 DEBUG { 07:08:22 DEBUG "metric": { 07:08:22 DEBUG "pod": "ds-idrepo-2" 07:08:22 DEBUG }, 07:08:22 DEBUG "value": [ 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG [...] 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG "pod": "ds-idrepo-0" 07:08:22 DEBUG }, 07:08:22 DEBUG "value": [ 07:08:22 DEBUG 1712815702, 07:08:22 DEBUG "0" 07:08:22 DEBUG ] 07:08:22 DEBUG } 07:08:22 DEBUG ] 07:08:22 DEBUG } 07:08:22 DEBUG } 07:08:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815702" 07:08:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815702" 07:08:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815702" 07:08:22 INFO [http_cmd]: http status code OK 07:08:22 DEBUG --- status code --- 07:08:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:08:22 DEBUG --- http response --- 07:08:22 INFO [http_cmd]: http status code OK 07:08:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:08:22 DEBUG --- status code --- 07:08:22 INFO [http_cmd]: http status code OK 07:08:22 DEBUG ----- output ----- 07:08:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:08:22 DEBUG --- status code --- 07:08:22 DEBUG { 07:08:22 DEBUG --- http response --- 07:08:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:08:22 DEBUG "status": "success", 07:08:22 DEBUG --- http response --- 07:08:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:08:22 DEBUG "data": { 07:08:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:08:22 DEBUG ----- output ----- 07:08:22 DEBUG "resultType": "vector", 07:08:22 DEBUG ----- output ----- 07:08:22 DEBUG { 07:08:22 DEBUG "result": [ 07:08:22 DEBUG { 07:08:22 DEBUG "status": "success", 07:08:22 DEBUG { 07:08:22 DEBUG "status": "success", 07:08:22 DEBUG "data": { 07:08:22 DEBUG "metric": { 07:08:22 DEBUG "data": { 07:08:22 DEBUG "resultType": "vector", 07:08:22 DEBUG "instance": "10.142.0.107:9100" 07:08:22 DEBUG "resultType": "vector", 07:08:22 DEBUG "result": [ 07:08:22 DEBUG }, 07:08:22 DEBUG "result": [ 07:08:22 DEBUG { 07:08:22 DEBUG "value": [ 07:08:22 DEBUG { 07:08:22 DEBUG "metric": { 07:08:22 DEBUG 07:08:22 DEBUG "metric": { 07:08:22 DEBUG "instance": "10.142.0.107:9100" 07:08:22 DEBUG 07:08:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:08:22 DEBUG }, 07:08:22 DEBUG 07:08:22 DEBUG }, 07:08:22 DEBUG "value": [ 07:08:22 DEBUG [...] 07:08:22 DEBUG "value": [ 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG [...] 07:08:22 DEBUG "instance": "10.142.0.77:9100" 07:08:22 DEBUG [...] 07:08:22 DEBUG 07:08:22 DEBUG }, 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG "value": [ 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 1712815702, 07:08:22 DEBUG 07:08:22 DEBUG "instance": "10.142.0.77:9100" 07:08:22 DEBUG "78114102" 07:08:22 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:08:22 DEBUG }, 07:08:22 DEBUG ] 07:08:22 DEBUG }, 07:08:22 DEBUG "value": [ 07:08:22 DEBUG } 07:08:22 DEBUG "value": [ 07:08:22 DEBUG 1712815702, 07:08:22 DEBUG ] 07:08:22 DEBUG 1712815702, 07:08:22 DEBUG "52986774528" 07:08:22 DEBUG } 07:08:22 DEBUG "0" 07:08:22 DEBUG ] 07:08:22 DEBUG } 07:08:22 DEBUG ] 07:08:22 DEBUG } 07:08:22 DEBUG } 07:08:22 DEBUG ] 07:08:22 DEBUG ] 07:08:22 DEBUG } 07:08:22 DEBUG } 07:08:22 DEBUG } 07:08:22 DEBUG } 07:08:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815702" 07:08:22 INFO [http_cmd]: http status code OK 07:08:22 DEBUG --- status code --- 07:08:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:08:22 DEBUG --- http response --- 07:08:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:08:22 DEBUG ----- output ----- 07:08:22 DEBUG { 07:08:22 DEBUG "status": "success", 07:08:22 DEBUG "data": { 07:08:22 DEBUG "resultType": "vector", 07:08:22 DEBUG "result": [ 07:08:22 DEBUG { 07:08:22 DEBUG "metric": { 07:08:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:08:22 DEBUG }, 07:08:22 DEBUG "value": [ 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG [...] 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:08:22 DEBUG }, 07:08:22 DEBUG "value": [ 07:08:22 DEBUG 1712815702, 07:08:22 DEBUG "0" 07:08:22 DEBUG ] 07:08:22 DEBUG } 07:08:22 DEBUG ] 07:08:22 DEBUG } 07:08:22 DEBUG } 07:08:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815702" 07:08:22 INFO [http_cmd]: http status code OK 07:08:22 DEBUG --- status code --- 07:08:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:08:22 DEBUG --- http response --- 07:08:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:08:22 DEBUG ----- output ----- 07:08:22 DEBUG { 07:08:22 DEBUG "status": "success", 07:08:22 DEBUG "data": { 07:08:22 DEBUG "resultType": "vector", 07:08:22 DEBUG "result": [ 07:08:22 DEBUG { 07:08:22 DEBUG "metric": { 07:08:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:08:22 DEBUG }, 07:08:22 DEBUG "value": [ 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG [...] 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG 07:08:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:08:22 DEBUG }, 07:08:22 DEBUG "value": [ 07:08:22 DEBUG 1712815702, 07:08:22 DEBUG "1276.2333333333333" 07:08:22 DEBUG ] 07:08:22 DEBUG } 07:08:22 DEBUG ] 07:08:22 DEBUG } 07:08:22 DEBUG } 07:08:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815703" 07:08:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815703" 07:08:23 INFO [http_cmd]: http status code OK 07:08:23 DEBUG --- status code --- 07:08:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:08:23 DEBUG --- http response --- 07:08:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:08:23 DEBUG ----- output ----- 07:08:23 DEBUG { 07:08:23 DEBUG "status": "success", 07:08:23 DEBUG "data": { 07:08:23 DEBUG "resultType": "vector", 07:08:23 DEBUG "result": [ 07:08:23 DEBUG { 07:08:23 DEBUG "metric": { 07:08:23 DEBUG "pod": "ds-idrepo-2" 07:08:23 DEBUG }, 07:08:23 DEBUG "value": [ 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG [...] 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG "pod": "ds-cts-2" 07:08:23 DEBUG }, 07:08:23 DEBUG "value": [ 07:08:23 DEBUG 1712815703, 07:08:23 DEBUG "29.733333333333334" 07:08:23 DEBUG ] 07:08:23 DEBUG } 07:08:23 DEBUG ] 07:08:23 DEBUG } 07:08:23 DEBUG } 07:08:23 INFO [http_cmd]: http status code OK 07:08:23 DEBUG --- status code --- 07:08:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:08:23 DEBUG --- http response --- 07:08:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:08:23 DEBUG ----- output ----- 07:08:23 DEBUG { 07:08:23 DEBUG "status": "success", 07:08:23 DEBUG "data": { 07:08:23 DEBUG "resultType": "vector", 07:08:23 DEBUG "result": [ 07:08:23 DEBUG { 07:08:23 DEBUG "metric": { 07:08:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:08:23 DEBUG }, 07:08:23 DEBUG "value": [ 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG [...] 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:08:23 DEBUG }, 07:08:23 DEBUG "value": [ 07:08:23 DEBUG 1712815703, 07:08:23 DEBUG "1275.8333333333333" 07:08:23 DEBUG ] 07:08:23 DEBUG } 07:08:23 DEBUG ] 07:08:23 DEBUG } 07:08:23 DEBUG } 07:08:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815703" 07:08:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815703" 07:08:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815703" 07:08:23 INFO [http_cmd]: http status code OK 07:08:23 INFO [http_cmd]: http status code OK 07:08:23 DEBUG --- status code --- 07:08:23 DEBUG --- status code --- 07:08:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:08:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:08:23 DEBUG --- http response --- 07:08:23 DEBUG --- http response --- 07:08:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:08:23 DEBUG ----- output ----- 07:08:23 DEBUG { 07:08:23 DEBUG "status": "success", 07:08:23 DEBUG "data": { 07:08:23 DEBUG "resultType": "vector", 07:08:23 DEBUG "result": [ 07:08:23 DEBUG { 07:08:23 DEBUG "metric": { 07:08:23 DEBUG "pod": "ds-idrepo-2" 07:08:23 DEBUG }, 07:08:23 DEBUG "value": [ 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG [...] 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG "pod": "ds-cts-2" 07:08:23 DEBUG }, 07:08:23 DEBUG "value": [ 07:08:23 DEBUG 1712815703, 07:08:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:08:23 DEBUG "0" 07:08:23 DEBUG ----- output ----- 07:08:23 DEBUG ] 07:08:23 DEBUG { 07:08:23 DEBUG } 07:08:23 DEBUG "status": "success", 07:08:23 DEBUG ] 07:08:23 DEBUG "data": { 07:08:23 DEBUG } 07:08:23 DEBUG "resultType": "vector", 07:08:23 DEBUG } 07:08:23 DEBUG "result": [ 07:08:23 DEBUG { 07:08:23 DEBUG "metric": { 07:08:23 DEBUG "pod": "ds-idrepo-2" 07:08:23 DEBUG }, 07:08:23 DEBUG "value": [ 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG [...] 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG "pod": "ds-idrepo-0" 07:08:23 DEBUG }, 07:08:23 DEBUG "value": [ 07:08:23 DEBUG 1712815703, 07:08:23 DEBUG "0" 07:08:23 DEBUG ] 07:08:23 DEBUG } 07:08:23 DEBUG ] 07:08:23 DEBUG } 07:08:23 DEBUG } 07:08:23 INFO [http_cmd]: http status code OK 07:08:23 DEBUG --- status code --- 07:08:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:08:23 DEBUG --- http response --- 07:08:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:08:23 DEBUG ----- output ----- 07:08:23 DEBUG { 07:08:23 DEBUG "status": "success", 07:08:23 DEBUG "data": { 07:08:23 DEBUG "resultType": "vector", 07:08:23 DEBUG "result": [ 07:08:23 DEBUG { 07:08:23 DEBUG "metric": { 07:08:23 DEBUG "pod": "ds-idrepo-2" 07:08:23 DEBUG }, 07:08:23 DEBUG "value": [ 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG [...] 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG "pod": "ds-cts-2" 07:08:23 DEBUG }, 07:08:23 DEBUG "value": [ 07:08:23 DEBUG 1712815703, 07:08:23 DEBUG "0.000033333333333333335" 07:08:23 DEBUG ] 07:08:23 DEBUG } 07:08:23 DEBUG ] 07:08:23 DEBUG } 07:08:23 DEBUG } 07:08:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712815703" 07:08:23 INFO [http_cmd]: http status code OK 07:08:23 DEBUG --- status code --- 07:08:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:08:23 DEBUG --- http response --- 07:08:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:08:23 DEBUG ----- output ----- 07:08:23 DEBUG { 07:08:23 DEBUG "status": "success", 07:08:23 DEBUG "data": { 07:08:23 DEBUG "resultType": "vector", 07:08:23 DEBUG "result": [ 07:08:23 DEBUG { 07:08:23 DEBUG "metric": { 07:08:23 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:08:23 DEBUG }, 07:08:23 DEBUG "value": [ 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG [...] 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 07:08:23 DEBUG }, 07:08:23 DEBUG "value": [ 07:08:23 DEBUG 1712815703, 07:08:23 DEBUG "14146686976" 07:08:23 DEBUG ] 07:08:23 DEBUG } 07:08:23 DEBUG ] 07:08:23 DEBUG } 07:08:23 DEBUG } 07:08:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815703" 07:08:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815703" 07:08:23 INFO [http_cmd]: http status code OK 07:08:23 DEBUG --- status code --- 07:08:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:08:23 DEBUG --- http response --- 07:08:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:08:23 DEBUG ----- output ----- 07:08:23 DEBUG { 07:08:23 DEBUG "status": "success", 07:08:23 DEBUG "data": { 07:08:23 DEBUG "resultType": "vector", 07:08:23 DEBUG "result": [ 07:08:23 DEBUG { 07:08:23 DEBUG "metric": { 07:08:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:08:23 DEBUG }, 07:08:23 DEBUG "value": [ 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG [...] 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:08:23 DEBUG }, 07:08:23 DEBUG "value": [ 07:08:23 DEBUG 1712815703, 07:08:23 DEBUG "0" 07:08:23 DEBUG ] 07:08:23 DEBUG } 07:08:23 DEBUG ] 07:08:23 DEBUG } 07:08:23 DEBUG } 07:08:23 INFO [http_cmd]: http status code OK 07:08:23 DEBUG --- status code --- 07:08:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:08:23 DEBUG --- http response --- 07:08:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:08:23 DEBUG ----- output ----- 07:08:23 DEBUG { 07:08:23 DEBUG "status": "success", 07:08:23 DEBUG "data": { 07:08:23 DEBUG "resultType": "vector", 07:08:23 DEBUG "result": [ 07:08:23 DEBUG { 07:08:23 DEBUG "metric": { 07:08:23 DEBUG "instance": "10.142.0.107:9100" 07:08:23 DEBUG }, 07:08:23 DEBUG "value": [ 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG [...] 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG "instance": "10.142.0.77:9100" 07:08:23 DEBUG }, 07:08:23 DEBUG "value": [ 07:08:23 DEBUG 1712815703, 07:08:23 DEBUG "0" 07:08:23 DEBUG ] 07:08:23 DEBUG } 07:08:23 DEBUG ] 07:08:23 DEBUG } 07:08:23 DEBUG } 07:08:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815703" 07:08:23 INFO [http_cmd]: http status code OK 07:08:23 DEBUG --- status code --- 07:08:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:08:23 DEBUG --- http response --- 07:08:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:08:23 DEBUG ----- output ----- 07:08:23 DEBUG { 07:08:23 DEBUG "status": "success", 07:08:23 DEBUG "data": { 07:08:23 DEBUG "resultType": "vector", 07:08:23 DEBUG "result": [ 07:08:23 DEBUG { 07:08:23 DEBUG "metric": { 07:08:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:08:23 DEBUG }, 07:08:23 DEBUG "value": [ 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG [...] 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG 07:08:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:08:23 DEBUG }, 07:08:23 DEBUG "value": [ 07:08:23 DEBUG 1712815703, 07:08:23 DEBUG "1275.6666666666667" 07:08:23 DEBUG ] 07:08:23 DEBUG } 07:08:23 DEBUG ] 07:08:23 DEBUG } 07:08:23 DEBUG } 07:08:35 INFO 07:08:35 INFO [loop_until]: kubectl --namespace=xlou top pods 07:08:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:08:35 INFO [loop_until]: OK (rc = 0) 07:08:35 DEBUG --- stdout --- 07:08:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 8839m 8833Mi am-5b5bd8567b-8h2w4 8808m 8858Mi am-5b5bd8567b-qlkw6 7563m 8521Mi ds-cts-0 8033m 13403Mi ds-cts-1 8596m 13406Mi ds-cts-2 8306m 13410Mi ds-idrepo-0 10687m 5503Mi ds-idrepo-1 332m 5480Mi ds-idrepo-2 323m 5784Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 4m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 3m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 7983m 4921Mi 07:08:35 DEBUG --- stderr --- 07:08:35 DEBUG 07:08:39 INFO 07:08:39 INFO [loop_until]: kubectl --namespace=xlou top node 07:08:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:08:39 INFO [loop_until]: OK (rc = 0) 07:08:39 DEBUG --- stdout --- 07:08:39 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 2732Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 64m 0% 2785Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7932m 26% 10206Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 12521m 41% 3663Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 9356m 31% 10456Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 126m 0% 3865Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9446m 31% 10501Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 398m 2% 6472Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 10772m 67% 6207Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8385m 52% 14032Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 7814m 49% 14024Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 8373m 52% 14028Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 389m 2% 6163Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 8171m 51% 8507Mi 14% 07:08:39 DEBUG --- stderr --- 07:08:39 DEBUG 07:09:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815762" 07:09:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815762" 07:09:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712815762" 07:09:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815762" 07:09:22 INFO [http_cmd]: http status code OK 07:09:22 INFO [http_cmd]: http status code OK 07:09:22 INFO [http_cmd]: http status code OK 07:09:22 DEBUG --- status code --- 07:09:22 DEBUG --- status code --- 07:09:22 DEBUG --- status code --- 07:09:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:09:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:09:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:09:22 DEBUG --- http response --- 07:09:22 DEBUG --- http response --- 07:09:22 DEBUG --- http response --- 07:09:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:09:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:09:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:09:22 DEBUG ----- output ----- 07:09:22 DEBUG ----- output ----- 07:09:22 DEBUG ----- output ----- 07:09:22 DEBUG { 07:09:22 DEBUG { 07:09:22 DEBUG { 07:09:22 DEBUG "status": "success", 07:09:22 DEBUG "status": "success", 07:09:22 DEBUG "status": "success", 07:09:22 DEBUG "data": { 07:09:22 DEBUG "data": { 07:09:22 DEBUG "data": { 07:09:22 DEBUG "resultType": "vector", 07:09:22 DEBUG "resultType": "vector", 07:09:22 DEBUG "resultType": "vector", 07:09:22 DEBUG "result": [ 07:09:22 DEBUG "result": [ 07:09:22 DEBUG "result": [ 07:09:22 DEBUG { 07:09:22 DEBUG { 07:09:22 DEBUG { 07:09:22 DEBUG "metric": { 07:09:22 DEBUG "metric": { 07:09:22 DEBUG "metric": { 07:09:22 DEBUG "pod": "ds-idrepo-2" 07:09:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:09:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:09:22 DEBUG }, 07:09:22 DEBUG }, 07:09:22 DEBUG }, 07:09:22 DEBUG "value": [ 07:09:22 DEBUG "value": [ 07:09:22 DEBUG "value": [ 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG [...] 07:09:22 DEBUG [...] 07:09:22 DEBUG [...] 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG "pod": "ds-idrepo-0" 07:09:22 DEBUG "pod": "ds-idrepo-1" 07:09:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:09:22 DEBUG }, 07:09:22 DEBUG }, 07:09:22 DEBUG }, 07:09:22 DEBUG "value": [ 07:09:22 DEBUG "value": [ 07:09:22 DEBUG "value": [ 07:09:22 DEBUG 1712815762, 07:09:22 INFO [http_cmd]: http status code OK 07:09:22 DEBUG 1712815762, 07:09:22 DEBUG 1712815762, 07:09:22 DEBUG "0" 07:09:22 DEBUG --- status code --- 07:09:22 DEBUG "0.20846816615871142" 07:09:22 DEBUG "236636" 07:09:22 DEBUG ] 07:09:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:09:22 DEBUG ] 07:09:22 DEBUG ] 07:09:22 DEBUG } 07:09:22 DEBUG --- http response --- 07:09:22 DEBUG } 07:09:22 DEBUG } 07:09:22 DEBUG ] 07:09:22 DEBUG ] 07:09:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:09:22 DEBUG ] 07:09:22 DEBUG } 07:09:22 DEBUG } 07:09:22 DEBUG ----- output ----- 07:09:22 DEBUG } 07:09:22 DEBUG } 07:09:22 DEBUG } 07:09:22 DEBUG { 07:09:22 DEBUG } 07:09:22 DEBUG "status": "success", 07:09:22 DEBUG "data": { 07:09:22 DEBUG "resultType": "vector", 07:09:22 DEBUG "result": [ 07:09:22 DEBUG { 07:09:22 DEBUG "metric": { 07:09:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:09:22 DEBUG }, 07:09:22 DEBUG "value": [ 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG [...] 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:09:22 DEBUG }, 07:09:22 DEBUG "value": [ 07:09:22 DEBUG 1712815762, 07:09:22 DEBUG "8.026101501395129" 07:09:22 DEBUG ] 07:09:22 DEBUG } 07:09:22 DEBUG ] 07:09:22 DEBUG } 07:09:22 DEBUG } 07:09:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712815762" 07:09:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815762" 07:09:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712815762" 07:09:22 INFO [http_cmd]: http status code OK 07:09:22 DEBUG --- status code --- 07:09:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:09:22 DEBUG --- http response --- 07:09:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:09:22 DEBUG ----- output ----- 07:09:22 DEBUG { 07:09:22 DEBUG "status": "success", 07:09:22 DEBUG "data": { 07:09:22 DEBUG "resultType": "vector", 07:09:22 DEBUG "result": [ 07:09:22 DEBUG { 07:09:22 DEBUG "metric": { 07:09:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:09:22 DEBUG }, 07:09:22 DEBUG "value": [ 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG [...] 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG "pod": "ds-idrepo-1" 07:09:22 DEBUG }, 07:09:22 DEBUG "value": [ 07:09:22 DEBUG 1712815762, 07:09:22 DEBUG "3879.6021440223726" 07:09:22 DEBUG ] 07:09:22 DEBUG } 07:09:22 DEBUG ] 07:09:22 DEBUG } 07:09:22 DEBUG } 07:09:22 INFO [http_cmd]: http status code OK 07:09:22 DEBUG --- status code --- 07:09:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:09:22 DEBUG --- http response --- 07:09:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:09:22 DEBUG ----- output ----- 07:09:22 DEBUG { 07:09:22 DEBUG "status": "success", 07:09:22 DEBUG "data": { 07:09:22 DEBUG "resultType": "vector", 07:09:22 DEBUG "result": [ 07:09:22 DEBUG { 07:09:22 DEBUG "metric": { 07:09:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:09:22 DEBUG }, 07:09:22 DEBUG "value": [ 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG [...] 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:09:22 DEBUG }, 07:09:22 DEBUG "value": [ 07:09:22 DEBUG 1712815762, 07:09:22 DEBUG "0.023" 07:09:22 DEBUG ] 07:09:22 DEBUG } 07:09:22 DEBUG ] 07:09:22 DEBUG } 07:09:22 DEBUG } 07:09:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815762" 07:09:22 ERROR [http_cmd]: ERROR 07:09:22 ERROR --- status code --- 07:09:22 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:09:22 ERROR --- http response --- 07:09:22 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:09:22 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:09:22 INFO [http_cmd]: http status code OK 07:09:22 DEBUG --- status code --- 07:09:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:09:22 DEBUG --- http response --- 07:09:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:09:22 DEBUG ----- output ----- 07:09:22 DEBUG { 07:09:22 DEBUG "status": "success", 07:09:22 DEBUG "data": { 07:09:22 DEBUG "resultType": "vector", 07:09:22 DEBUG "result": [ 07:09:22 DEBUG { 07:09:22 DEBUG "metric": { 07:09:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:09:22 DEBUG }, 07:09:22 DEBUG "value": [ 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG [...] 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:09:22 DEBUG }, 07:09:22 DEBUG "value": [ 07:09:22 DEBUG 1712815762, 07:09:22 DEBUG "1499.6" 07:09:22 DEBUG ] 07:09:22 DEBUG } 07:09:22 DEBUG ] 07:09:22 DEBUG } 07:09:22 DEBUG } 07:09:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815762" 07:09:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815762" 07:09:22 INFO [http_cmd]: http status code OK 07:09:22 DEBUG --- status code --- 07:09:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:09:22 DEBUG --- http response --- 07:09:22 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:09:22 INFO [http_cmd]: http status code OK 07:09:22 DEBUG --- status code --- 07:09:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:09:22 DEBUG --- http response --- 07:09:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:09:22 DEBUG ----- output ----- 07:09:22 DEBUG { 07:09:22 DEBUG "status": "success", 07:09:22 DEBUG "data": { 07:09:22 DEBUG "resultType": "vector", 07:09:22 DEBUG "result": [ 07:09:22 DEBUG { 07:09:22 DEBUG "metric": { 07:09:22 DEBUG "instance": "10.142.0.107:9100" 07:09:22 DEBUG }, 07:09:22 DEBUG "value": [ 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG [...] 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG "instance": "10.142.0.77:9100" 07:09:22 DEBUG }, 07:09:22 DEBUG "value": [ 07:09:22 DEBUG 1712815762, 07:09:22 DEBUG "0.0019666666666663464" 07:09:22 DEBUG ] 07:09:22 DEBUG } 07:09:22 DEBUG ] 07:09:22 DEBUG } 07:09:22 DEBUG } 07:09:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815762" 07:09:22 INFO [http_cmd]: http status code OK 07:09:22 DEBUG --- status code --- 07:09:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:09:22 DEBUG --- http response --- 07:09:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:09:22 DEBUG ----- output ----- 07:09:22 DEBUG { 07:09:22 DEBUG "status": "success", 07:09:22 DEBUG "data": { 07:09:22 DEBUG "resultType": "vector", 07:09:22 DEBUG "result": [ 07:09:22 DEBUG { 07:09:22 DEBUG "metric": { 07:09:22 DEBUG "instance": "10.142.0.107:9100" 07:09:22 DEBUG }, 07:09:22 DEBUG "value": [ 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG [...] 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG "instance": "10.142.0.77:9100" 07:09:22 DEBUG }, 07:09:22 DEBUG "value": [ 07:09:22 DEBUG 1712815762, 07:09:22 DEBUG "78141611.06666666" 07:09:22 DEBUG ] 07:09:22 DEBUG } 07:09:22 DEBUG ] 07:09:22 DEBUG } 07:09:22 DEBUG } 07:09:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712815762" 07:09:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815762" 07:09:22 INFO [http_cmd]: http status code OK 07:09:22 DEBUG --- status code --- 07:09:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:09:22 DEBUG --- http response --- 07:09:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:09:22 DEBUG ----- output ----- 07:09:22 DEBUG { 07:09:22 DEBUG "status": "success", 07:09:22 DEBUG "data": { 07:09:22 DEBUG "resultType": "vector", 07:09:22 DEBUG "result": [ 07:09:22 DEBUG { 07:09:22 DEBUG "metric": { 07:09:22 DEBUG "pod": "ds-idrepo-2" 07:09:22 DEBUG }, 07:09:22 DEBUG "value": [ 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG [...] 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG "pod": "ds-cts-2" 07:09:22 DEBUG }, 07:09:22 DEBUG "value": [ 07:09:22 DEBUG 1712815762, 07:09:22 DEBUG "0" 07:09:22 DEBUG ] 07:09:22 DEBUG } 07:09:22 DEBUG ] 07:09:22 DEBUG } 07:09:22 DEBUG } 07:09:22 INFO [http_cmd]: http status code OK 07:09:22 DEBUG --- status code --- 07:09:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:09:22 DEBUG --- http response --- 07:09:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:09:22 DEBUG ----- output ----- 07:09:22 DEBUG { 07:09:22 DEBUG "status": "success", 07:09:22 DEBUG "data": { 07:09:22 DEBUG "resultType": "vector", 07:09:22 DEBUG "result": [ 07:09:22 DEBUG { 07:09:22 DEBUG "metric": { 07:09:22 DEBUG "pod": "ds-idrepo-2" 07:09:22 DEBUG }, 07:09:22 DEBUG "value": [ 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG [...] 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG 07:09:22 DEBUG "pod": "ds-idrepo-0" 07:09:22 DEBUG }, 07:09:22 DEBUG "value": [ 07:09:22 DEBUG 1712815762, 07:09:22 DEBUG "0" 07:09:22 DEBUG ] 07:09:22 DEBUG } 07:09:22 DEBUG ] 07:09:22 DEBUG } 07:09:22 DEBUG } 07:09:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815762" 07:09:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815762" 07:09:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815762" 07:09:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815762" 07:09:22 INFO [http_cmd]: http status code OK 07:09:22 INFO [http_cmd]: http status code OK 07:09:22 DEBUG --- status code --- 07:09:22 DEBUG --- status code --- 07:09:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:09:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:09:22 DEBUG --- http response --- 07:09:22 DEBUG --- http response --- 07:09:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:09:23 DEBUG ----- output ----- 07:09:23 DEBUG { 07:09:23 DEBUG "status": "success", 07:09:23 DEBUG "data": { 07:09:23 DEBUG "resultType": "vector", 07:09:23 DEBUG "result": [ 07:09:23 DEBUG { 07:09:23 DEBUG "metric": { 07:09:23 DEBUG "instance": "10.142.0.107:9100" 07:09:23 DEBUG }, 07:09:23 DEBUG "value": [ 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG [...] 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG "instance": "10.142.0.77:9100" 07:09:23 DEBUG }, 07:09:23 DEBUG "value": [ 07:09:23 DEBUG 1712815762, 07:09:23 DEBUG "78143785.66666666" 07:09:23 DEBUG ] 07:09:23 DEBUG } 07:09:23 DEBUG ] 07:09:23 DEBUG } 07:09:23 DEBUG } 07:09:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:09:23 DEBUG ----- output ----- 07:09:23 DEBUG { 07:09:23 DEBUG "status": "success", 07:09:23 DEBUG "data": { 07:09:23 DEBUG "resultType": "vector", 07:09:23 DEBUG "result": [ 07:09:23 DEBUG { 07:09:23 DEBUG "metric": { 07:09:23 DEBUG "instance": "10.142.0.107:9100" 07:09:23 DEBUG }, 07:09:23 DEBUG "value": [ 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG [...] 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG "instance": "10.142.0.77:9100" 07:09:23 DEBUG }, 07:09:23 DEBUG "value": [ 07:09:23 DEBUG 1712815762, 07:09:23 DEBUG "52988052480" 07:09:23 DEBUG ] 07:09:23 DEBUG } 07:09:23 DEBUG ] 07:09:23 DEBUG } 07:09:23 DEBUG } 07:09:23 INFO [http_cmd]: http status code OK 07:09:23 INFO [http_cmd]: http status code OK 07:09:23 DEBUG --- status code --- 07:09:23 DEBUG --- status code --- 07:09:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:09:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:09:23 DEBUG --- http response --- 07:09:23 DEBUG --- http response --- 07:09:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:09:23 DEBUG ----- output ----- 07:09:23 DEBUG { 07:09:23 DEBUG "status": "success", 07:09:23 DEBUG "data": { 07:09:23 DEBUG "resultType": "vector", 07:09:23 DEBUG "result": [ 07:09:23 DEBUG { 07:09:23 DEBUG "metric": { 07:09:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:09:23 DEBUG }, 07:09:23 DEBUG "value": [ 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG [...] 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG "pod": "ds-idrepo-1" 07:09:23 DEBUG }, 07:09:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:09:23 DEBUG "value": [ 07:09:23 DEBUG ----- output ----- 07:09:23 DEBUG 1712815762, 07:09:23 DEBUG { 07:09:23 DEBUG "0" 07:09:23 DEBUG "status": "success", 07:09:23 DEBUG ] 07:09:23 DEBUG "data": { 07:09:23 DEBUG } 07:09:23 DEBUG "resultType": "vector", 07:09:23 DEBUG ] 07:09:23 DEBUG "result": [ 07:09:23 DEBUG } 07:09:23 DEBUG { 07:09:23 DEBUG } 07:09:23 DEBUG "metric": { 07:09:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:09:23 DEBUG }, 07:09:23 DEBUG "value": [ 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG [...] 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG "pod": "ds-idrepo-1" 07:09:23 DEBUG }, 07:09:23 DEBUG "value": [ 07:09:23 DEBUG 1712815762, 07:09:23 DEBUG "2586.4014293482487" 07:09:23 DEBUG ] 07:09:23 DEBUG } 07:09:23 DEBUG ] 07:09:23 DEBUG } 07:09:23 DEBUG } 07:09:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815763" 07:09:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815763" 07:09:23 INFO [http_cmd]: http status code OK 07:09:23 DEBUG --- status code --- 07:09:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:09:23 DEBUG --- http response --- 07:09:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:09:23 DEBUG ----- output ----- 07:09:23 DEBUG { 07:09:23 DEBUG "status": "success", 07:09:23 DEBUG "data": { 07:09:23 DEBUG "resultType": "vector", 07:09:23 DEBUG "result": [ 07:09:23 DEBUG { 07:09:23 DEBUG "metric": { 07:09:23 DEBUG "pod": "ds-idrepo-2" 07:09:23 DEBUG }, 07:09:23 DEBUG "value": [ 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG [...] 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG "pod": "ds-cts-2" 07:09:23 DEBUG }, 07:09:23 DEBUG "value": [ 07:09:23 DEBUG 1712815763, 07:09:23 DEBUG "29.599999999999998" 07:09:23 DEBUG ] 07:09:23 DEBUG } 07:09:23 DEBUG ] 07:09:23 DEBUG } 07:09:23 DEBUG } 07:09:23 INFO [http_cmd]: http status code OK 07:09:23 DEBUG --- status code --- 07:09:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:09:23 DEBUG --- http response --- 07:09:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:09:23 DEBUG ----- output ----- 07:09:23 DEBUG { 07:09:23 DEBUG "status": "success", 07:09:23 DEBUG "data": { 07:09:23 DEBUG "resultType": "vector", 07:09:23 DEBUG "result": [ 07:09:23 DEBUG { 07:09:23 DEBUG "metric": { 07:09:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:09:23 DEBUG }, 07:09:23 DEBUG "value": [ 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG [...] 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:09:23 DEBUG }, 07:09:23 DEBUG "value": [ 07:09:23 DEBUG 1712815763, 07:09:23 DEBUG "1239.3666666666666" 07:09:23 DEBUG ] 07:09:23 DEBUG } 07:09:23 DEBUG ] 07:09:23 DEBUG } 07:09:23 DEBUG } 07:09:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815763" 07:09:23 INFO [http_cmd]: http status code OK 07:09:23 DEBUG --- status code --- 07:09:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:09:23 DEBUG --- http response --- 07:09:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:09:23 DEBUG ----- output ----- 07:09:23 DEBUG { 07:09:23 DEBUG "status": "success", 07:09:23 DEBUG "data": { 07:09:23 DEBUG "resultType": "vector", 07:09:23 DEBUG "result": [ 07:09:23 DEBUG { 07:09:23 DEBUG "metric": { 07:09:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:09:23 DEBUG }, 07:09:23 DEBUG "value": [ 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG [...] 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:09:23 DEBUG }, 07:09:23 DEBUG "value": [ 07:09:23 DEBUG 1712815763, 07:09:23 DEBUG "1238.8333333333333" 07:09:23 DEBUG ] 07:09:23 DEBUG } 07:09:23 DEBUG ] 07:09:23 DEBUG } 07:09:23 DEBUG } 07:09:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815763" 07:09:23 INFO [http_cmd]: http status code OK 07:09:23 DEBUG --- status code --- 07:09:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:09:23 DEBUG --- http response --- 07:09:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:09:23 DEBUG ----- output ----- 07:09:23 DEBUG { 07:09:23 DEBUG "status": "success", 07:09:23 DEBUG "data": { 07:09:23 DEBUG "resultType": "vector", 07:09:23 DEBUG "result": [ 07:09:23 DEBUG { 07:09:23 DEBUG "metric": { 07:09:23 DEBUG "pod": "ds-idrepo-2" 07:09:23 DEBUG }, 07:09:23 DEBUG "value": [ 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG [...] 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG "pod": "ds-cts-2" 07:09:23 DEBUG }, 07:09:23 DEBUG "value": [ 07:09:23 DEBUG 1712815763, 07:09:23 DEBUG "0.000595938888888889" 07:09:23 DEBUG ] 07:09:23 DEBUG } 07:09:23 DEBUG ] 07:09:23 DEBUG } 07:09:23 DEBUG } 07:09:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815763" 07:09:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815763" 07:09:23 INFO [http_cmd]: http status code OK 07:09:23 INFO [http_cmd]: http status code OK 07:09:23 DEBUG --- status code --- 07:09:23 DEBUG --- status code --- 07:09:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:09:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:09:23 DEBUG --- http response --- 07:09:23 DEBUG --- http response --- 07:09:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:09:23 DEBUG ----- output ----- 07:09:23 DEBUG { 07:09:23 DEBUG "status": "success", 07:09:23 DEBUG "data": { 07:09:23 DEBUG "resultType": "vector", 07:09:23 DEBUG "result": [ 07:09:23 DEBUG { 07:09:23 DEBUG "metric": { 07:09:23 DEBUG "pod": "ds-idrepo-2" 07:09:23 DEBUG }, 07:09:23 DEBUG "value": [ 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG [...] 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG "pod": "ds-idrepo-0" 07:09:23 DEBUG }, 07:09:23 DEBUG "value": [ 07:09:23 DEBUG 1712815763, 07:09:23 DEBUG "0" 07:09:23 DEBUG ] 07:09:23 DEBUG } 07:09:23 DEBUG ] 07:09:23 DEBUG } 07:09:23 DEBUG } 07:09:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:09:23 DEBUG ----- output ----- 07:09:23 DEBUG { 07:09:23 DEBUG "status": "success", 07:09:23 DEBUG "data": { 07:09:23 DEBUG "resultType": "vector", 07:09:23 DEBUG "result": [ 07:09:23 DEBUG { 07:09:23 DEBUG "metric": { 07:09:23 DEBUG "pod": "ds-idrepo-2" 07:09:23 DEBUG }, 07:09:23 DEBUG "value": [ 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG [...] 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG "pod": "ds-cts-2" 07:09:23 DEBUG }, 07:09:23 DEBUG "value": [ 07:09:23 DEBUG 1712815763, 07:09:23 DEBUG "0" 07:09:23 DEBUG ] 07:09:23 DEBUG } 07:09:23 DEBUG ] 07:09:23 DEBUG } 07:09:23 DEBUG } 07:09:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712815763" 07:09:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815763" 07:09:23 INFO [http_cmd]: http status code OK 07:09:23 DEBUG --- status code --- 07:09:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:09:23 DEBUG --- http response --- 07:09:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:09:23 DEBUG ----- output ----- 07:09:23 DEBUG { 07:09:23 DEBUG "status": "success", 07:09:23 DEBUG "data": { 07:09:23 DEBUG "resultType": "vector", 07:09:23 DEBUG "result": [ 07:09:23 DEBUG { 07:09:23 DEBUG "metric": { 07:09:23 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:09:23 DEBUG }, 07:09:23 DEBUG "value": [ 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG [...] 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 07:09:23 DEBUG }, 07:09:23 DEBUG "value": [ 07:09:23 DEBUG 1712815763, 07:09:23 DEBUG "14144512000" 07:09:23 DEBUG ] 07:09:23 DEBUG } 07:09:23 DEBUG ] 07:09:23 DEBUG } 07:09:23 DEBUG } 07:09:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815763" 07:09:23 INFO [http_cmd]: http status code OK 07:09:23 DEBUG --- status code --- 07:09:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:09:23 DEBUG --- http response --- 07:09:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:09:23 DEBUG ----- output ----- 07:09:23 DEBUG { 07:09:23 DEBUG "status": "success", 07:09:23 DEBUG "data": { 07:09:23 DEBUG "resultType": "vector", 07:09:23 DEBUG "result": [ 07:09:23 DEBUG { 07:09:23 DEBUG "metric": { 07:09:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:09:23 DEBUG }, 07:09:23 DEBUG "value": [ 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG [...] 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:09:23 DEBUG }, 07:09:23 DEBUG "value": [ 07:09:23 DEBUG 1712815763, 07:09:23 DEBUG "0" 07:09:23 DEBUG ] 07:09:23 DEBUG } 07:09:23 DEBUG ] 07:09:23 DEBUG } 07:09:23 DEBUG } 07:09:23 INFO [http_cmd]: http status code OK 07:09:23 DEBUG --- status code --- 07:09:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:09:23 DEBUG --- http response --- 07:09:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:09:23 DEBUG ----- output ----- 07:09:23 DEBUG { 07:09:23 DEBUG "status": "success", 07:09:23 DEBUG "data": { 07:09:23 DEBUG "resultType": "vector", 07:09:23 DEBUG "result": [ 07:09:23 DEBUG { 07:09:23 DEBUG "metric": { 07:09:23 DEBUG "instance": "10.142.0.107:9100" 07:09:23 DEBUG }, 07:09:23 DEBUG "value": [ 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG [...] 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG "instance": "10.142.0.77:9100" 07:09:23 DEBUG }, 07:09:23 DEBUG "value": [ 07:09:23 DEBUG 1712815763, 07:09:23 DEBUG "0.0003333333333330302" 07:09:23 DEBUG ] 07:09:23 DEBUG } 07:09:23 DEBUG ] 07:09:23 DEBUG } 07:09:23 DEBUG } 07:09:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815763" 07:09:23 INFO [http_cmd]: http status code OK 07:09:23 DEBUG --- status code --- 07:09:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:09:23 DEBUG --- http response --- 07:09:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:09:23 DEBUG ----- output ----- 07:09:23 DEBUG { 07:09:23 DEBUG "status": "success", 07:09:23 DEBUG "data": { 07:09:23 DEBUG "resultType": "vector", 07:09:23 DEBUG "result": [ 07:09:23 DEBUG { 07:09:23 DEBUG "metric": { 07:09:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:09:23 DEBUG }, 07:09:23 DEBUG "value": [ 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG [...] 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG 07:09:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:09:23 DEBUG }, 07:09:23 DEBUG "value": [ 07:09:23 DEBUG 1712815763, 07:09:23 DEBUG "1234.7333333333333" 07:09:23 DEBUG ] 07:09:23 DEBUG } 07:09:23 DEBUG ] 07:09:23 DEBUG } 07:09:23 DEBUG } 07:09:35 INFO 07:09:35 INFO [loop_until]: kubectl --namespace=xlou top pods 07:09:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:09:35 INFO [loop_until]: OK (rc = 0) 07:09:35 DEBUG --- stdout --- 07:09:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 8815m 8849Mi am-5b5bd8567b-8h2w4 8515m 8859Mi am-5b5bd8567b-qlkw6 7403m 8535Mi ds-cts-0 7968m 12582Mi ds-cts-1 8478m 13404Mi ds-cts-2 8250m 13407Mi ds-idrepo-0 10416m 5503Mi ds-idrepo-1 357m 5480Mi ds-idrepo-2 365m 5784Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 3m 1011Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 6m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 8041m 5774Mi 07:09:35 DEBUG --- stderr --- 07:09:35 DEBUG 07:09:39 INFO 07:09:39 INFO [loop_until]: kubectl --namespace=xlou top node 07:09:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:09:39 INFO [loop_until]: OK (rc = 0) 07:09:39 DEBUG --- stdout --- 07:09:39 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 2717Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 62m 0% 2781Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7485m 25% 10207Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 11967m 40% 3654Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 9263m 31% 10467Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 143m 0% 3849Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9266m 31% 10504Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 447m 2% 6472Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 10907m 68% 6206Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8501m 53% 14033Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 8178m 51% 13272Mi 22% gke-xlou-cdm-ds-32e4dcb1-j2hk 8803m 55% 14052Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 427m 2% 6160Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 8205m 51% 9283Mi 15% 07:09:39 DEBUG --- stderr --- 07:09:39 DEBUG 07:10:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815822" 07:10:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815822" 07:10:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815822" 07:10:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712815822" 07:10:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712815822" 07:10:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712815822" 07:10:22 INFO [http_cmd]: http status code OK 07:10:22 INFO [http_cmd]: http status code OK 07:10:22 INFO [http_cmd]: http status code OK 07:10:22 INFO [http_cmd]: http status code OK 07:10:22 DEBUG --- status code --- 07:10:22 INFO [http_cmd]: http status code OK 07:10:22 DEBUG --- status code --- 07:10:22 DEBUG --- status code --- 07:10:22 DEBUG --- status code --- 07:10:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:10:22 DEBUG --- status code --- 07:10:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:10:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:10:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:10:22 DEBUG --- http response --- 07:10:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:10:22 DEBUG --- http response --- 07:10:22 DEBUG --- http response --- 07:10:22 DEBUG --- http response --- 07:10:22 DEBUG --- http response --- 07:10:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:10:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:10:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:10:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:10:22 DEBUG ----- output ----- 07:10:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:10:22 DEBUG ----- output ----- 07:10:22 DEBUG ----- output ----- 07:10:22 DEBUG ----- output ----- 07:10:22 DEBUG { 07:10:22 DEBUG ----- output ----- 07:10:22 DEBUG { 07:10:22 DEBUG { 07:10:22 DEBUG { 07:10:22 DEBUG "status": "success", 07:10:22 DEBUG { 07:10:22 DEBUG "status": "success", 07:10:22 DEBUG "status": "success", 07:10:22 DEBUG "status": "success", 07:10:22 DEBUG "data": { 07:10:22 DEBUG "status": "success", 07:10:22 DEBUG "data": { 07:10:22 DEBUG "data": { 07:10:22 DEBUG "data": { 07:10:22 DEBUG "resultType": "vector", 07:10:22 DEBUG "data": { 07:10:22 DEBUG "resultType": "vector", 07:10:22 DEBUG "resultType": "vector", 07:10:22 DEBUG "resultType": "vector", 07:10:22 DEBUG "result": [ 07:10:22 DEBUG "resultType": "vector", 07:10:22 DEBUG "result": [ 07:10:22 DEBUG "result": [ 07:10:22 DEBUG "result": [ 07:10:22 DEBUG { 07:10:22 DEBUG "result": [ 07:10:22 DEBUG { 07:10:22 DEBUG { 07:10:22 DEBUG { 07:10:22 DEBUG "metric": { 07:10:22 DEBUG { 07:10:22 DEBUG "metric": { 07:10:22 DEBUG "metric": { 07:10:22 DEBUG "metric": { 07:10:22 DEBUG "pod": "ds-idrepo-2" 07:10:22 DEBUG "metric": { 07:10:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:10:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:10:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:10:22 DEBUG }, 07:10:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:10:22 DEBUG }, 07:10:22 DEBUG }, 07:10:22 DEBUG }, 07:10:22 DEBUG "value": [ 07:10:22 DEBUG }, 07:10:22 DEBUG "value": [ 07:10:22 DEBUG "value": [ 07:10:22 DEBUG "value": [ 07:10:22 DEBUG 07:10:22 DEBUG "value": [ 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG [...] 07:10:22 DEBUG 07:10:22 DEBUG [...] 07:10:22 DEBUG [...] 07:10:22 DEBUG [...] 07:10:22 DEBUG 07:10:22 DEBUG [...] 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG "pod": "ds-idrepo-0" 07:10:22 DEBUG 07:10:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:10:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:10:22 DEBUG "pod": "ds-idrepo-1" 07:10:22 DEBUG }, 07:10:22 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:10:22 DEBUG }, 07:10:22 DEBUG }, 07:10:22 DEBUG }, 07:10:22 DEBUG "value": [ 07:10:22 DEBUG }, 07:10:22 DEBUG "value": [ 07:10:22 DEBUG 1712815822, 07:10:22 DEBUG "value": [ 07:10:22 DEBUG 1712815822, 07:10:22 DEBUG "value": [ 07:10:22 DEBUG "value": [ 07:10:22 DEBUG "227593" 07:10:22 DEBUG 1712815822, 07:10:22 DEBUG "0" 07:10:22 DEBUG 1712815822, 07:10:22 DEBUG 1712815822, 07:10:22 DEBUG ] 07:10:22 DEBUG "0.023" 07:10:22 DEBUG ] 07:10:22 DEBUG "8.039676903335964" 07:10:22 DEBUG "0" 07:10:22 DEBUG } 07:10:22 DEBUG ] 07:10:22 DEBUG } 07:10:22 DEBUG ] 07:10:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815822" 07:10:22 DEBUG ] 07:10:22 DEBUG ] 07:10:22 DEBUG } 07:10:22 DEBUG ] 07:10:22 DEBUG } 07:10:22 DEBUG } 07:10:22 DEBUG } 07:10:22 DEBUG ] 07:10:22 DEBUG } 07:10:22 DEBUG ] 07:10:22 DEBUG ] 07:10:22 DEBUG } 07:10:22 DEBUG } 07:10:22 DEBUG } 07:10:22 DEBUG } 07:10:22 DEBUG } 07:10:22 DEBUG } 07:10:22 DEBUG } 07:10:22 DEBUG } 07:10:22 ERROR [http_cmd]: ERROR 07:10:22 ERROR --- status code --- 07:10:22 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:10:22 ERROR --- http response --- 07:10:22 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:10:22 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:10:22 INFO [http_cmd]: http status code OK 07:10:22 DEBUG --- status code --- 07:10:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:10:22 DEBUG --- http response --- 07:10:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:10:22 DEBUG ----- output ----- 07:10:22 DEBUG { 07:10:22 DEBUG "status": "success", 07:10:22 DEBUG "data": { 07:10:22 DEBUG "resultType": "vector", 07:10:22 DEBUG "result": [ 07:10:22 DEBUG { 07:10:22 DEBUG "metric": { 07:10:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:10:22 DEBUG }, 07:10:22 DEBUG "value": [ 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG [...] 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG "pod": "ds-idrepo-1" 07:10:22 DEBUG }, 07:10:22 DEBUG "value": [ 07:10:22 DEBUG 1712815822, 07:10:22 DEBUG "0" 07:10:22 DEBUG ] 07:10:22 DEBUG } 07:10:22 DEBUG ] 07:10:22 DEBUG } 07:10:22 DEBUG } 07:10:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815822" 07:10:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815822" 07:10:22 INFO [http_cmd]: http status code OK 07:10:22 DEBUG --- status code --- 07:10:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:10:22 DEBUG --- http response --- 07:10:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:10:22 DEBUG ----- output ----- 07:10:22 DEBUG { 07:10:22 DEBUG "status": "success", 07:10:22 DEBUG "data": { 07:10:22 DEBUG "resultType": "vector", 07:10:22 DEBUG "result": [ 07:10:22 DEBUG { 07:10:22 DEBUG "metric": { 07:10:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:10:22 DEBUG }, 07:10:22 DEBUG "value": [ 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG [...] 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:10:22 DEBUG }, 07:10:22 DEBUG "value": [ 07:10:22 DEBUG 1712815822, 07:10:22 DEBUG "1385.4333333333334" 07:10:22 DEBUG ] 07:10:22 DEBUG } 07:10:22 DEBUG ] 07:10:22 DEBUG } 07:10:22 DEBUG } 07:10:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815822" 07:10:22 INFO [http_cmd]: http status code OK 07:10:22 DEBUG --- status code --- 07:10:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:10:22 DEBUG --- http response --- 07:10:22 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:10:22 INFO [http_cmd]: http status code OK 07:10:22 DEBUG --- status code --- 07:10:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:10:22 DEBUG --- http response --- 07:10:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:10:22 DEBUG ----- output ----- 07:10:22 DEBUG { 07:10:22 DEBUG "status": "success", 07:10:22 DEBUG "data": { 07:10:22 DEBUG "resultType": "vector", 07:10:22 DEBUG "result": [ 07:10:22 DEBUG { 07:10:22 DEBUG "metric": { 07:10:22 DEBUG "instance": "10.142.0.107:9100" 07:10:22 DEBUG }, 07:10:22 DEBUG "value": [ 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG [...] 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG 07:10:22 DEBUG "instance": "10.142.0.77:9100" 07:10:22 DEBUG }, 07:10:22 DEBUG "value": [ 07:10:22 DEBUG 1712815822, 07:10:22 DEBUG "0.002133333333333572" 07:10:22 DEBUG ] 07:10:22 DEBUG } 07:10:22 DEBUG ] 07:10:22 DEBUG } 07:10:22 DEBUG } 07:10:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815822" 07:10:23 INFO [http_cmd]: http status code OK 07:10:23 DEBUG --- status code --- 07:10:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:10:23 DEBUG --- http response --- 07:10:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:10:23 DEBUG ----- output ----- 07:10:23 DEBUG { 07:10:23 DEBUG "status": "success", 07:10:23 DEBUG "data": { 07:10:23 DEBUG "resultType": "vector", 07:10:23 DEBUG "result": [ 07:10:23 DEBUG { 07:10:23 DEBUG "metric": { 07:10:23 DEBUG "instance": "10.142.0.107:9100" 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG [...] 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG "instance": "10.142.0.77:9100" 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 1712815822, 07:10:23 DEBUG "75915867.9" 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG } 07:10:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815823" 07:10:23 INFO [http_cmd]: http status code OK 07:10:23 DEBUG --- status code --- 07:10:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:10:23 DEBUG --- http response --- 07:10:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 07:10:23 DEBUG ----- output ----- 07:10:23 DEBUG { 07:10:23 DEBUG "status": "success", 07:10:23 DEBUG "data": { 07:10:23 DEBUG "resultType": "vector", 07:10:23 DEBUG "result": [ 07:10:23 DEBUG { 07:10:23 DEBUG "metric": { 07:10:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG [...] 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 1712815823, 07:10:23 DEBUG "1452439.1280346161" 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG } 07:10:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712815823" 07:10:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815823" 07:10:23 INFO [http_cmd]: http status code OK 07:10:23 DEBUG --- status code --- 07:10:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:10:23 DEBUG --- http response --- 07:10:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:10:23 DEBUG ----- output ----- 07:10:23 DEBUG { 07:10:23 DEBUG "status": "success", 07:10:23 DEBUG "data": { 07:10:23 DEBUG "resultType": "vector", 07:10:23 DEBUG "result": [ 07:10:23 DEBUG { 07:10:23 DEBUG "metric": { 07:10:23 DEBUG "pod": "ds-idrepo-2" 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG [...] 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG "pod": "ds-cts-2" 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 1712815823, 07:10:23 DEBUG "0" 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG } 07:10:23 INFO [http_cmd]: http status code OK 07:10:23 DEBUG --- status code --- 07:10:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:10:23 DEBUG --- http response --- 07:10:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:10:23 DEBUG ----- output ----- 07:10:23 DEBUG { 07:10:23 DEBUG "status": "success", 07:10:23 DEBUG "data": { 07:10:23 DEBUG "resultType": "vector", 07:10:23 DEBUG "result": [ 07:10:23 DEBUG { 07:10:23 DEBUG "metric": { 07:10:23 DEBUG "pod": "ds-idrepo-2" 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG [...] 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG "pod": "ds-idrepo-0" 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 1712815823, 07:10:23 DEBUG "0" 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG } 07:10:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815823" 07:10:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815823" 07:10:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815823" 07:10:23 INFO [http_cmd]: http status code OK 07:10:23 INFO [http_cmd]: http status code OK 07:10:23 DEBUG --- status code --- 07:10:23 DEBUG --- status code --- 07:10:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:10:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:10:23 DEBUG --- http response --- 07:10:23 DEBUG --- http response --- 07:10:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:10:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:10:23 DEBUG ----- output ----- 07:10:23 DEBUG ----- output ----- 07:10:23 DEBUG { 07:10:23 DEBUG { 07:10:23 DEBUG "status": "success", 07:10:23 DEBUG "status": "success", 07:10:23 DEBUG "data": { 07:10:23 DEBUG "data": { 07:10:23 DEBUG "resultType": "vector", 07:10:23 DEBUG "resultType": "vector", 07:10:23 DEBUG "result": [ 07:10:23 DEBUG "result": [ 07:10:23 DEBUG { 07:10:23 DEBUG { 07:10:23 DEBUG "metric": { 07:10:23 DEBUG "metric": { 07:10:23 DEBUG "instance": "10.142.0.107:9100" 07:10:23 DEBUG "instance": "10.142.0.107:9100" 07:10:23 DEBUG }, 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG [...] 07:10:23 DEBUG [...] 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG "instance": "10.142.0.77:9100" 07:10:23 DEBUG "instance": "10.142.0.77:9100" 07:10:23 DEBUG }, 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 1712815823, 07:10:23 DEBUG 1712815823, 07:10:23 DEBUG "52989326336" 07:10:23 DEBUG "75917792.96666667" 07:10:23 DEBUG ] 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG } 07:10:23 DEBUG ] 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG } 07:10:23 DEBUG } 07:10:23 DEBUG } 07:10:23 INFO [http_cmd]: http status code OK 07:10:23 DEBUG --- status code --- 07:10:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:10:23 DEBUG --- http response --- 07:10:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 07:10:23 DEBUG ----- output ----- 07:10:23 DEBUG { 07:10:23 DEBUG "status": "success", 07:10:23 DEBUG "data": { 07:10:23 DEBUG "resultType": "vector", 07:10:23 DEBUG "result": [ 07:10:23 DEBUG { 07:10:23 DEBUG "metric": { 07:10:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG [...] 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 1712815823, 07:10:23 DEBUG "0" 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG } 07:10:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815823" 07:10:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815823" 07:10:23 INFO [http_cmd]: http status code OK 07:10:23 DEBUG --- status code --- 07:10:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:10:23 DEBUG --- http response --- 07:10:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:10:23 DEBUG ----- output ----- 07:10:23 DEBUG { 07:10:23 DEBUG "status": "success", 07:10:23 DEBUG "data": { 07:10:23 DEBUG "resultType": "vector", 07:10:23 DEBUG "result": [ 07:10:23 DEBUG { 07:10:23 DEBUG "metric": { 07:10:23 DEBUG "pod": "ds-idrepo-2" 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG [...] 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG "pod": "ds-cts-2" 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 1712815823, 07:10:23 DEBUG "28.666666666666668" 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG } 07:10:23 INFO [http_cmd]: http status code OK 07:10:23 DEBUG --- status code --- 07:10:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:10:23 DEBUG --- http response --- 07:10:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:10:23 DEBUG ----- output ----- 07:10:23 DEBUG { 07:10:23 DEBUG "status": "success", 07:10:23 DEBUG "data": { 07:10:23 DEBUG "resultType": "vector", 07:10:23 DEBUG "result": [ 07:10:23 DEBUG { 07:10:23 DEBUG "metric": { 07:10:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG [...] 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 1712815823, 07:10:23 DEBUG "1245.3666666666666" 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG } 07:10:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815823" 07:10:23 INFO [http_cmd]: http status code OK 07:10:23 DEBUG --- status code --- 07:10:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:10:23 DEBUG --- http response --- 07:10:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:10:23 DEBUG ----- output ----- 07:10:23 DEBUG { 07:10:23 DEBUG "status": "success", 07:10:23 DEBUG "data": { 07:10:23 DEBUG "resultType": "vector", 07:10:23 DEBUG "result": [ 07:10:23 DEBUG { 07:10:23 DEBUG "metric": { 07:10:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG [...] 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 1712815823, 07:10:23 DEBUG "1246.4" 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG } 07:10:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815823" 07:10:23 INFO [http_cmd]: http status code OK 07:10:23 DEBUG --- status code --- 07:10:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:10:23 DEBUG --- http response --- 07:10:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:10:23 DEBUG ----- output ----- 07:10:23 DEBUG { 07:10:23 DEBUG "status": "success", 07:10:23 DEBUG "data": { 07:10:23 DEBUG "resultType": "vector", 07:10:23 DEBUG "result": [ 07:10:23 DEBUG { 07:10:23 DEBUG "metric": { 07:10:23 DEBUG "pod": "ds-idrepo-2" 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG [...] 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG "pod": "ds-cts-2" 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 1712815823, 07:10:23 DEBUG "0" 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG } 07:10:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815823" 07:10:23 INFO [http_cmd]: http status code OK 07:10:23 DEBUG --- status code --- 07:10:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:10:23 DEBUG --- http response --- 07:10:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:10:23 DEBUG ----- output ----- 07:10:23 DEBUG { 07:10:23 DEBUG "status": "success", 07:10:23 DEBUG "data": { 07:10:23 DEBUG "resultType": "vector", 07:10:23 DEBUG "result": [ 07:10:23 DEBUG { 07:10:23 DEBUG "metric": { 07:10:23 DEBUG "pod": "ds-idrepo-2" 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG [...] 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG "pod": "ds-cts-2" 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 1712815823, 07:10:23 DEBUG "0.0004666666666666666" 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG } 07:10:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815823" 07:10:23 INFO [http_cmd]: http status code OK 07:10:23 DEBUG --- status code --- 07:10:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:10:23 DEBUG --- http response --- 07:10:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:10:23 DEBUG ----- output ----- 07:10:23 DEBUG { 07:10:23 DEBUG "status": "success", 07:10:23 DEBUG "data": { 07:10:23 DEBUG "resultType": "vector", 07:10:23 DEBUG "result": [ 07:10:23 DEBUG { 07:10:23 DEBUG "metric": { 07:10:23 DEBUG "pod": "ds-idrepo-2" 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG [...] 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG "pod": "ds-idrepo-0" 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 1712815823, 07:10:23 DEBUG "0" 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG } 07:10:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712815823" 07:10:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815823" 07:10:23 INFO [http_cmd]: http status code OK 07:10:23 DEBUG --- status code --- 07:10:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:10:23 DEBUG --- http response --- 07:10:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:10:23 DEBUG ----- output ----- 07:10:23 DEBUG { 07:10:23 DEBUG "status": "success", 07:10:23 DEBUG "data": { 07:10:23 DEBUG "resultType": "vector", 07:10:23 DEBUG "result": [ 07:10:23 DEBUG { 07:10:23 DEBUG "metric": { 07:10:23 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG [...] 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 1712815823, 07:10:23 DEBUG "13665255424" 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG } 07:10:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815823" 07:10:23 INFO [http_cmd]: http status code OK 07:10:23 DEBUG --- status code --- 07:10:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:10:23 DEBUG --- http response --- 07:10:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:10:23 DEBUG ----- output ----- 07:10:23 DEBUG { 07:10:23 DEBUG "status": "success", 07:10:23 DEBUG "data": { 07:10:23 DEBUG "resultType": "vector", 07:10:23 DEBUG "result": [ 07:10:23 DEBUG { 07:10:23 DEBUG "metric": { 07:10:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG [...] 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 1712815823, 07:10:23 DEBUG "0" 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG } 07:10:23 INFO [http_cmd]: http status code OK 07:10:23 DEBUG --- status code --- 07:10:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:10:23 DEBUG --- http response --- 07:10:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:10:23 DEBUG ----- output ----- 07:10:23 DEBUG { 07:10:23 DEBUG "status": "success", 07:10:23 DEBUG "data": { 07:10:23 DEBUG "resultType": "vector", 07:10:23 DEBUG "result": [ 07:10:23 DEBUG { 07:10:23 DEBUG "metric": { 07:10:23 DEBUG "instance": "10.142.0.107:9100" 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG [...] 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG "instance": "10.142.0.77:9100" 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 1712815823, 07:10:23 DEBUG "0.00033333333333350387" 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG } 07:10:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815823" 07:10:23 INFO [http_cmd]: http status code OK 07:10:23 DEBUG --- status code --- 07:10:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:10:23 DEBUG --- http response --- 07:10:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:10:23 DEBUG ----- output ----- 07:10:23 DEBUG { 07:10:23 DEBUG "status": "success", 07:10:23 DEBUG "data": { 07:10:23 DEBUG "resultType": "vector", 07:10:23 DEBUG "result": [ 07:10:23 DEBUG { 07:10:23 DEBUG "metric": { 07:10:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG [...] 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG 07:10:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:10:23 DEBUG }, 07:10:23 DEBUG "value": [ 07:10:23 DEBUG 1712815823, 07:10:23 DEBUG "1249.2666666666667" 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG ] 07:10:23 DEBUG } 07:10:23 DEBUG } 07:10:35 INFO 07:10:35 INFO [loop_until]: kubectl --namespace=xlou top pods 07:10:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:10:35 INFO [loop_until]: OK (rc = 0) 07:10:35 DEBUG --- stdout --- 07:10:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 8563m 8827Mi am-5b5bd8567b-8h2w4 8016m 8857Mi am-5b5bd8567b-qlkw6 7254m 8537Mi ds-cts-0 7619m 13204Mi ds-cts-1 8256m 12859Mi ds-cts-2 7836m 12970Mi ds-idrepo-0 10521m 5503Mi ds-idrepo-1 310m 5480Mi ds-idrepo-2 328m 5784Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 3m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 2m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 7802m 6270Mi 07:10:35 DEBUG --- stderr --- 07:10:35 DEBUG 07:10:39 INFO 07:10:39 INFO [loop_until]: kubectl --namespace=xlou top node 07:10:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:10:39 INFO [loop_until]: OK (rc = 0) 07:10:39 DEBUG --- stdout --- 07:10:39 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 2741Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 66m 0% 2788Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7356m 24% 10207Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 12374m 41% 3622Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 8948m 29% 10454Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 145m 0% 3841Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 8712m 29% 10513Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 384m 2% 6472Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 10181m 64% 6208Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 8176m 51% 13587Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 7650m 48% 13830Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 8085m 50% 13506Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 359m 2% 6164Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 8180m 51% 10486Mi 17% 07:10:39 DEBUG --- stderr --- 07:10:39 DEBUG 07:11:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815882" 07:11:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815882" 07:11:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712815882" 07:11:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712815882" 07:11:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712815882" 07:11:22 ERROR [http_cmd]: ERROR 07:11:22 INFO [http_cmd]: http status code OK 07:11:22 INFO [http_cmd]: http status code OK 07:11:22 ERROR --- status code --- 07:11:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815882" 07:11:22 INFO [http_cmd]: http status code OK 07:11:22 DEBUG --- status code --- 07:11:22 DEBUG --- status code --- 07:11:22 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:11:22 DEBUG --- status code --- 07:11:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:11:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:11:22 ERROR --- http response --- 07:11:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:11:22 DEBUG --- http response --- 07:11:22 DEBUG --- http response --- 07:11:22 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:11:22 DEBUG --- http response --- 07:11:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:11:22 INFO [http_cmd]: http status code OK 07:11:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:11:22 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:11:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:11:22 DEBUG ----- output ----- 07:11:22 DEBUG --- status code --- 07:11:22 DEBUG ----- output ----- 07:11:22 DEBUG ----- output ----- 07:11:22 DEBUG { 07:11:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:11:22 DEBUG { 07:11:22 DEBUG { 07:11:22 DEBUG "status": "success", 07:11:22 DEBUG --- http response --- 07:11:22 DEBUG "status": "success", 07:11:22 DEBUG "status": "success", 07:11:22 DEBUG "data": { 07:11:22 DEBUG "data": { 07:11:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:11:22 DEBUG "data": { 07:11:22 DEBUG "resultType": "vector", 07:11:22 DEBUG "resultType": "vector", 07:11:22 DEBUG ----- output ----- 07:11:22 DEBUG "resultType": "vector", 07:11:22 DEBUG "result": [ 07:11:22 DEBUG "result": [ 07:11:22 DEBUG { 07:11:22 DEBUG "result": [ 07:11:22 DEBUG { 07:11:22 DEBUG { 07:11:22 DEBUG "status": "success", 07:11:22 DEBUG { 07:11:22 DEBUG "metric": { 07:11:22 DEBUG "metric": { 07:11:22 DEBUG "data": { 07:11:22 DEBUG "metric": { 07:11:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:11:22 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:11:22 DEBUG "resultType": "vector", 07:11:22 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:11:22 DEBUG }, 07:11:22 DEBUG }, 07:11:22 DEBUG "result": [ 07:11:22 DEBUG }, 07:11:22 DEBUG "value": [ 07:11:22 DEBUG "value": [ 07:11:22 DEBUG { 07:11:22 DEBUG "value": [ 07:11:22 DEBUG 07:11:22 DEBUG 07:11:22 DEBUG "metric": { 07:11:22 DEBUG 07:11:22 DEBUG 07:11:22 DEBUG 07:11:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:11:22 DEBUG 07:11:22 DEBUG 07:11:22 DEBUG 07:11:22 DEBUG }, 07:11:22 DEBUG 07:11:22 DEBUG [...] 07:11:22 DEBUG [...] 07:11:22 DEBUG "value": [ 07:11:22 DEBUG [...] 07:11:22 DEBUG 07:11:22 DEBUG 07:11:22 DEBUG 07:11:22 DEBUG 07:11:22 DEBUG 07:11:22 DEBUG 07:11:22 DEBUG 07:11:22 DEBUG 07:11:22 DEBUG 07:11:22 DEBUG 07:11:22 DEBUG 07:11:22 DEBUG 07:11:22 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:11:22 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:11:22 DEBUG [...] 07:11:22 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:11:22 DEBUG }, 07:11:22 DEBUG }, 07:11:22 DEBUG 07:11:22 DEBUG }, 07:11:22 DEBUG "value": [ 07:11:22 DEBUG "value": [ 07:11:22 DEBUG 07:11:22 DEBUG "value": [ 07:11:22 DEBUG 1712815882, 07:11:22 DEBUG 1712815882, 07:11:22 DEBUG 07:11:22 DEBUG 1712815882, 07:11:22 DEBUG "4520396.52758331" 07:11:22 DEBUG "8.002590164729893" 07:11:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:11:22 DEBUG "8.03111831392051" 07:11:22 DEBUG ] 07:11:22 DEBUG ] 07:11:22 DEBUG }, 07:11:22 DEBUG ] 07:11:22 DEBUG } 07:11:22 DEBUG } 07:11:22 DEBUG "value": [ 07:11:22 DEBUG } 07:11:22 DEBUG ] 07:11:22 DEBUG ] 07:11:22 DEBUG 1712815882, 07:11:22 DEBUG ] 07:11:22 DEBUG } 07:11:22 DEBUG } 07:11:22 DEBUG "0.023" 07:11:22 DEBUG } 07:11:22 DEBUG } 07:11:22 DEBUG } 07:11:22 DEBUG ] 07:11:22 DEBUG } 07:11:22 DEBUG } 07:11:22 DEBUG ] 07:11:22 DEBUG } 07:11:22 DEBUG } 07:11:22 INFO [http_cmd]: http status code OK 07:11:22 DEBUG --- status code --- 07:11:22 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:11:22 DEBUG --- http response --- 07:11:22 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:11:22 DEBUG ----- output ----- 07:11:22 DEBUG { 07:11:22 DEBUG "status": "success", 07:11:22 DEBUG "data": { 07:11:22 DEBUG "resultType": "vector", 07:11:22 DEBUG "result": [ 07:11:22 DEBUG { 07:11:22 DEBUG "metric": { 07:11:22 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:11:22 DEBUG }, 07:11:22 DEBUG "value": [ 07:11:22 DEBUG 07:11:22 DEBUG 07:11:22 DEBUG 07:11:22 DEBUG [...] 07:11:22 DEBUG 07:11:22 DEBUG 07:11:22 DEBUG 07:11:22 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:11:22 DEBUG }, 07:11:22 DEBUG "value": [ 07:11:22 DEBUG 1712815882, 07:11:22 DEBUG "222111" 07:11:22 DEBUG ] 07:11:22 DEBUG } 07:11:22 DEBUG ] 07:11:22 DEBUG } 07:11:22 DEBUG } 07:11:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815882" 07:11:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815882" 07:11:22 INFO [http_cmd]: http status code OK 07:11:23 DEBUG --- status code --- 07:11:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:11:23 DEBUG --- http response --- 07:11:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:11:23 DEBUG ----- output ----- 07:11:23 DEBUG { 07:11:23 DEBUG "status": "success", 07:11:23 DEBUG "data": { 07:11:23 DEBUG "resultType": "vector", 07:11:23 DEBUG "result": [ 07:11:23 DEBUG { 07:11:23 DEBUG "metric": { 07:11:23 DEBUG "pod": "ds-idrepo-2" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG [...] 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG "pod": "ds-idrepo-0" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 1712815882, 07:11:23 DEBUG "0" 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG } 07:11:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815883" 07:11:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815883" 07:11:23 INFO [http_cmd]: http status code OK 07:11:23 DEBUG --- status code --- 07:11:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:11:23 DEBUG --- http response --- 07:11:23 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:11:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815883" 07:11:23 INFO [http_cmd]: http status code OK 07:11:23 DEBUG --- status code --- 07:11:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:11:23 DEBUG --- http response --- 07:11:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:11:23 DEBUG ----- output ----- 07:11:23 DEBUG { 07:11:23 DEBUG "status": "success", 07:11:23 DEBUG "data": { 07:11:23 DEBUG "resultType": "vector", 07:11:23 DEBUG "result": [ 07:11:23 DEBUG { 07:11:23 DEBUG "metric": { 07:11:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG [...] 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 1712815883, 07:11:23 DEBUG "1275.4666666666667" 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG } 07:11:23 INFO [http_cmd]: http status code OK 07:11:23 DEBUG --- status code --- 07:11:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:11:23 DEBUG --- http response --- 07:11:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:11:23 DEBUG ----- output ----- 07:11:23 DEBUG { 07:11:23 DEBUG "status": "success", 07:11:23 DEBUG "data": { 07:11:23 DEBUG "resultType": "vector", 07:11:23 DEBUG "result": [ 07:11:23 DEBUG { 07:11:23 DEBUG "metric": { 07:11:23 DEBUG "instance": "10.142.0.107:9100" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG [...] 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG "instance": "10.142.0.77:9100" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 1712815883, 07:11:23 DEBUG "73392968.73333333" 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG } 07:11:23 INFO [http_cmd]: http status code OK 07:11:23 DEBUG --- status code --- 07:11:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:11:23 DEBUG --- http response --- 07:11:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:11:23 DEBUG ----- output ----- 07:11:23 DEBUG { 07:11:23 DEBUG "status": "success", 07:11:23 DEBUG "data": { 07:11:23 DEBUG "resultType": "vector", 07:11:23 DEBUG "result": [ 07:11:23 DEBUG { 07:11:23 DEBUG "metric": { 07:11:23 DEBUG "instance": "10.142.0.107:9100" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG [...] 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG "instance": "10.142.0.77:9100" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 1712815883, 07:11:23 DEBUG "0.002133333333333335" 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG } 07:11:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815883" 07:11:23 INFO [http_cmd]: http status code OK 07:11:23 DEBUG --- status code --- 07:11:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:11:23 DEBUG --- http response --- 07:11:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 07:11:23 DEBUG ----- output ----- 07:11:23 DEBUG { 07:11:23 DEBUG "status": "success", 07:11:23 DEBUG "data": { 07:11:23 DEBUG "resultType": "vector", 07:11:23 DEBUG "result": [ 07:11:23 DEBUG { 07:11:23 DEBUG "metric": { 07:11:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG [...] 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 1712815883, 07:11:23 DEBUG "0" 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG } 07:11:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815883" 07:11:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712815883" 07:11:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815883" 07:11:23 INFO [http_cmd]: http status code OK 07:11:23 DEBUG --- status code --- 07:11:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:11:23 DEBUG --- http response --- 07:11:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 07:11:23 DEBUG ----- output ----- 07:11:23 DEBUG { 07:11:23 DEBUG "status": "success", 07:11:23 DEBUG "data": { 07:11:23 DEBUG "resultType": "vector", 07:11:23 DEBUG "result": [ 07:11:23 DEBUG { 07:11:23 DEBUG "metric": { 07:11:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG [...] 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 1712815883, 07:11:23 DEBUG "0" 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG } 07:11:23 INFO [http_cmd]: http status code OK 07:11:23 DEBUG --- status code --- 07:11:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:11:23 DEBUG --- http response --- 07:11:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:11:23 DEBUG ----- output ----- 07:11:23 DEBUG { 07:11:23 DEBUG "status": "success", 07:11:23 DEBUG "data": { 07:11:23 DEBUG "resultType": "vector", 07:11:23 DEBUG "result": [ 07:11:23 DEBUG { 07:11:23 DEBUG "metric": { 07:11:23 DEBUG "pod": "ds-idrepo-2" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG [...] 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG "pod": "ds-cts-2" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 1712815883, 07:11:23 DEBUG "0" 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG } 07:11:23 INFO [http_cmd]: http status code OK 07:11:23 DEBUG --- status code --- 07:11:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:11:23 DEBUG --- http response --- 07:11:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:11:23 DEBUG ----- output ----- 07:11:23 DEBUG { 07:11:23 DEBUG "status": "success", 07:11:23 DEBUG "data": { 07:11:23 DEBUG "resultType": "vector", 07:11:23 DEBUG "result": [ 07:11:23 DEBUG { 07:11:23 DEBUG "metric": { 07:11:23 DEBUG "pod": "ds-idrepo-2" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG [...] 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG "pod": "ds-idrepo-0" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 1712815883, 07:11:23 DEBUG "0" 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG } 07:11:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815883" 07:11:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815883" 07:11:23 INFO [http_cmd]: http status code OK 07:11:23 INFO [http_cmd]: http status code OK 07:11:23 DEBUG --- status code --- 07:11:23 DEBUG --- status code --- 07:11:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:11:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:11:23 DEBUG --- http response --- 07:11:23 DEBUG --- http response --- 07:11:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:11:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:11:23 DEBUG ----- output ----- 07:11:23 DEBUG ----- output ----- 07:11:23 DEBUG { 07:11:23 DEBUG { 07:11:23 DEBUG "status": "success", 07:11:23 DEBUG "status": "success", 07:11:23 DEBUG "data": { 07:11:23 DEBUG "data": { 07:11:23 DEBUG "resultType": "vector", 07:11:23 DEBUG "resultType": "vector", 07:11:23 DEBUG "result": [ 07:11:23 DEBUG "result": [ 07:11:23 DEBUG { 07:11:23 DEBUG { 07:11:23 DEBUG "metric": { 07:11:23 DEBUG "metric": { 07:11:23 DEBUG "instance": "10.142.0.107:9100" 07:11:23 DEBUG "instance": "10.142.0.107:9100" 07:11:23 DEBUG }, 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG [...] 07:11:23 DEBUG [...] 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG "instance": "10.142.0.77:9100" 07:11:23 DEBUG "instance": "10.142.0.77:9100" 07:11:23 DEBUG }, 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 1712815883, 07:11:23 DEBUG 1712815883, 07:11:23 DEBUG "52990594048" 07:11:23 DEBUG "73394839.36666667" 07:11:23 DEBUG ] 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG } 07:11:23 DEBUG ] 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG } 07:11:23 DEBUG } 07:11:23 DEBUG } 07:11:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815883" 07:11:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815883" 07:11:23 INFO [http_cmd]: http status code OK 07:11:23 DEBUG --- status code --- 07:11:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:11:23 DEBUG --- http response --- 07:11:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:11:23 DEBUG ----- output ----- 07:11:23 DEBUG { 07:11:23 DEBUG "status": "success", 07:11:23 DEBUG "data": { 07:11:23 DEBUG "resultType": "vector", 07:11:23 DEBUG "result": [ 07:11:23 DEBUG { 07:11:23 DEBUG "metric": { 07:11:23 DEBUG "pod": "ds-idrepo-2" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG [...] 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG "pod": "ds-cts-2" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 1712815883, 07:11:23 DEBUG "28.966666666666665" 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG } 07:11:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815883" 07:11:23 INFO [http_cmd]: http status code OK 07:11:23 DEBUG --- status code --- 07:11:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:11:23 DEBUG --- http response --- 07:11:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:11:23 DEBUG ----- output ----- 07:11:23 DEBUG { 07:11:23 DEBUG "status": "success", 07:11:23 DEBUG "data": { 07:11:23 DEBUG "resultType": "vector", 07:11:23 DEBUG "result": [ 07:11:23 DEBUG { 07:11:23 DEBUG "metric": { 07:11:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG [...] 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 1712815883, 07:11:23 DEBUG "1228.1" 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG } 07:11:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815883" 07:11:23 INFO [http_cmd]: http status code OK 07:11:23 DEBUG --- status code --- 07:11:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:11:23 DEBUG --- http response --- 07:11:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:11:23 DEBUG ----- output ----- 07:11:23 DEBUG { 07:11:23 DEBUG "status": "success", 07:11:23 DEBUG "data": { 07:11:23 DEBUG "resultType": "vector", 07:11:23 DEBUG "result": [ 07:11:23 DEBUG { 07:11:23 DEBUG "metric": { 07:11:23 DEBUG "pod": "ds-idrepo-2" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG [...] 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG "pod": "ds-cts-2" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 1712815883, 07:11:23 DEBUG "0" 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG } 07:11:23 INFO [http_cmd]: http status code OK 07:11:23 DEBUG --- status code --- 07:11:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:11:23 DEBUG --- http response --- 07:11:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:11:23 DEBUG ----- output ----- 07:11:23 DEBUG { 07:11:23 DEBUG "status": "success", 07:11:23 DEBUG "data": { 07:11:23 DEBUG "resultType": "vector", 07:11:23 DEBUG "result": [ 07:11:23 DEBUG { 07:11:23 DEBUG "metric": { 07:11:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG [...] 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 1712815883, 07:11:23 DEBUG "1228.6666666666667" 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG } 07:11:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815883" 07:11:23 INFO [http_cmd]: http status code OK 07:11:23 DEBUG --- status code --- 07:11:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:11:23 DEBUG --- http response --- 07:11:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:11:23 DEBUG ----- output ----- 07:11:23 DEBUG { 07:11:23 DEBUG "status": "success", 07:11:23 DEBUG "data": { 07:11:23 DEBUG "resultType": "vector", 07:11:23 DEBUG "result": [ 07:11:23 DEBUG { 07:11:23 DEBUG "metric": { 07:11:23 DEBUG "pod": "ds-idrepo-2" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG [...] 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG "pod": "ds-idrepo-0" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 1712815883, 07:11:23 DEBUG "0" 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG } 07:11:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815883" 07:11:23 INFO [http_cmd]: http status code OK 07:11:23 DEBUG --- status code --- 07:11:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:11:23 DEBUG --- http response --- 07:11:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:11:23 DEBUG ----- output ----- 07:11:23 DEBUG { 07:11:23 DEBUG "status": "success", 07:11:23 DEBUG "data": { 07:11:23 DEBUG "resultType": "vector", 07:11:23 DEBUG "result": [ 07:11:23 DEBUG { 07:11:23 DEBUG "metric": { 07:11:23 DEBUG "pod": "ds-idrepo-2" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG [...] 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG "pod": "ds-cts-2" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 1712815883, 07:11:23 DEBUG "0.0000666666666666667" 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG } 07:11:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712815883" 07:11:23 INFO [http_cmd]: http status code OK 07:11:23 DEBUG --- status code --- 07:11:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:11:23 DEBUG --- http response --- 07:11:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:11:23 DEBUG ----- output ----- 07:11:23 DEBUG { 07:11:23 DEBUG "status": "success", 07:11:23 DEBUG "data": { 07:11:23 DEBUG "resultType": "vector", 07:11:23 DEBUG "result": [ 07:11:23 DEBUG { 07:11:23 DEBUG "metric": { 07:11:23 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG [...] 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 1712815883, 07:11:23 DEBUG "14143291392" 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG } 07:11:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815883" 07:11:23 INFO [http_cmd]: http status code OK 07:11:23 DEBUG --- status code --- 07:11:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:11:23 DEBUG --- http response --- 07:11:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:11:23 DEBUG ----- output ----- 07:11:23 DEBUG { 07:11:23 DEBUG "status": "success", 07:11:23 DEBUG "data": { 07:11:23 DEBUG "resultType": "vector", 07:11:23 DEBUG "result": [ 07:11:23 DEBUG { 07:11:23 DEBUG "metric": { 07:11:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG [...] 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 1712815883, 07:11:23 DEBUG "1214.1333333333334" 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG } 07:11:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815883" 07:11:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815883" 07:11:23 INFO [http_cmd]: http status code OK 07:11:23 DEBUG --- status code --- 07:11:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:11:23 DEBUG --- http response --- 07:11:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:11:23 DEBUG ----- output ----- 07:11:23 DEBUG { 07:11:23 DEBUG "status": "success", 07:11:23 DEBUG "data": { 07:11:23 DEBUG "resultType": "vector", 07:11:23 DEBUG "result": [ 07:11:23 DEBUG { 07:11:23 DEBUG "metric": { 07:11:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG [...] 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 1712815883, 07:11:23 DEBUG "0" 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG } 07:11:23 INFO [http_cmd]: http status code OK 07:11:23 DEBUG --- status code --- 07:11:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:11:23 DEBUG --- http response --- 07:11:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:11:23 DEBUG ----- output ----- 07:11:23 DEBUG { 07:11:23 DEBUG "status": "success", 07:11:23 DEBUG "data": { 07:11:23 DEBUG "resultType": "vector", 07:11:23 DEBUG "result": [ 07:11:23 DEBUG { 07:11:23 DEBUG "metric": { 07:11:23 DEBUG "instance": "10.142.0.107:9100" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG [...] 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG 07:11:23 DEBUG "instance": "10.142.0.77:9100" 07:11:23 DEBUG }, 07:11:23 DEBUG "value": [ 07:11:23 DEBUG 1712815883, 07:11:23 DEBUG "0.0010000000000001561" 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG ] 07:11:23 DEBUG } 07:11:23 DEBUG } 07:11:35 INFO 07:11:35 INFO [loop_until]: kubectl --namespace=xlou top pods 07:11:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:11:35 INFO [loop_until]: OK (rc = 0) 07:11:35 DEBUG --- stdout --- 07:11:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 8331m 8835Mi am-5b5bd8567b-8h2w4 7787m 8868Mi am-5b5bd8567b-qlkw6 6867m 8531Mi ds-cts-0 7203m 13401Mi ds-cts-1 7751m 13405Mi ds-cts-2 7749m 12954Mi ds-idrepo-0 10199m 5503Mi ds-idrepo-1 303m 5480Mi ds-idrepo-2 348m 5784Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 5m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 4m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 8055m 7293Mi 07:11:35 DEBUG --- stderr --- 07:11:35 DEBUG 07:11:39 INFO 07:11:39 INFO [loop_until]: kubectl --namespace=xlou top node 07:11:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:11:39 INFO [loop_until]: OK (rc = 0) 07:11:39 DEBUG --- stdout --- 07:11:39 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 2732Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 64m 0% 2786Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7260m 24% 10208Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 12029m 40% 3656Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 8841m 29% 10459Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 127m 0% 3871Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 8811m 29% 10513Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 392m 2% 6472Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 10417m 65% 6202Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 7765m 48% 14030Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 7266m 45% 14043Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 7921m 49% 14055Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 362m 2% 6166Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 7915m 49% 10740Mi 18% 07:11:39 DEBUG --- stderr --- 07:11:39 DEBUG 07:12:22 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712815942" 07:12:23 ERROR [http_cmd]: ERROR 07:12:23 ERROR --- status code --- 07:12:23 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:12:23 ERROR --- http response --- 07:12:23 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:12:23 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:12:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815943" 07:12:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815943" 07:12:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712815943" 07:12:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815943" 07:12:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712815943" 07:12:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815943" 07:12:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815943" 07:12:23 INFO [http_cmd]: http status code OK 07:12:23 DEBUG --- status code --- 07:12:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:12:23 DEBUG --- http response --- 07:12:23 INFO [http_cmd]: http status code OK 07:12:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:12:23 DEBUG --- status code --- 07:12:23 INFO [http_cmd]: http status code OK 07:12:23 DEBUG ----- output ----- 07:12:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:12:23 DEBUG --- status code --- 07:12:23 DEBUG { 07:12:23 DEBUG --- http response --- 07:12:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:12:23 DEBUG "status": "success", 07:12:23 DEBUG --- http response --- 07:12:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 07:12:23 DEBUG "data": { 07:12:23 DEBUG ----- output ----- 07:12:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:12:23 DEBUG "resultType": "vector", 07:12:23 DEBUG { 07:12:23 DEBUG ----- output ----- 07:12:23 DEBUG "result": [ 07:12:23 DEBUG "status": "success", 07:12:23 DEBUG { 07:12:23 DEBUG { 07:12:23 DEBUG "data": { 07:12:23 DEBUG "status": "success", 07:12:23 DEBUG "metric": { 07:12:23 DEBUG "resultType": "vector", 07:12:23 DEBUG "data": { 07:12:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:12:23 DEBUG "result": [ 07:12:23 DEBUG "resultType": "vector", 07:12:23 DEBUG }, 07:12:23 DEBUG { 07:12:23 DEBUG "result": [ 07:12:23 DEBUG "value": [ 07:12:23 DEBUG "metric": { 07:12:23 DEBUG { 07:12:23 DEBUG 07:12:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:12:23 DEBUG "metric": { 07:12:23 DEBUG 07:12:23 DEBUG }, 07:12:23 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:12:23 DEBUG 07:12:23 DEBUG "value": [ 07:12:23 DEBUG }, 07:12:23 DEBUG [...] 07:12:23 DEBUG 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG [...] 07:12:23 DEBUG 07:12:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:12:23 DEBUG 07:12:23 DEBUG [...] 07:12:23 DEBUG }, 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 INFO [http_cmd]: http status code OK 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 07:12:23 INFO [http_cmd]: http status code OK 07:12:23 DEBUG 07:12:23 DEBUG --- status code --- 07:12:23 DEBUG 1712815943, 07:12:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:12:23 DEBUG --- status code --- 07:12:23 DEBUG 07:12:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:12:23 DEBUG "1237.9333333333334" 07:12:23 DEBUG }, 07:12:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:12:23 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:12:23 DEBUG --- http response --- 07:12:23 INFO [http_cmd]: http status code OK 07:12:23 INFO [http_cmd]: http status code OK 07:12:23 DEBUG ] 07:12:23 DEBUG "value": [ 07:12:23 DEBUG --- http response --- 07:12:23 DEBUG }, 07:12:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:12:23 DEBUG --- status code --- 07:12:23 DEBUG --- status code --- 07:12:23 DEBUG } 07:12:23 DEBUG 1712815943, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 07:12:23 DEBUG ----- output ----- 07:12:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:12:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:12:23 DEBUG ] 07:12:23 DEBUG "7.996586658992751" 07:12:23 DEBUG 1712815943, 07:12:23 DEBUG ----- output ----- 07:12:23 DEBUG { 07:12:23 DEBUG --- http response --- 07:12:23 DEBUG --- http response --- 07:12:23 DEBUG } 07:12:23 DEBUG ] 07:12:23 DEBUG "8.03788669900897" 07:12:23 DEBUG { 07:12:23 DEBUG "status": "success", 07:12:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:12:23 DEBUG } 07:12:23 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:12:23 DEBUG } 07:12:23 DEBUG ] 07:12:23 DEBUG "status": "success", 07:12:23 DEBUG "data": { 07:12:23 DEBUG ----- output ----- 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG "data": { 07:12:23 DEBUG "resultType": "vector", 07:12:23 DEBUG { 07:12:23 DEBUG } 07:12:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815943" 07:12:23 DEBUG ] 07:12:23 DEBUG "resultType": "vector", 07:12:23 DEBUG "result": [ 07:12:23 DEBUG "status": "success", 07:12:23 DEBUG } 07:12:23 DEBUG } 07:12:23 DEBUG "result": [ 07:12:23 DEBUG { 07:12:23 DEBUG "data": { 07:12:23 DEBUG } 07:12:23 DEBUG { 07:12:23 DEBUG "metric": { 07:12:23 DEBUG "resultType": "vector", 07:12:23 DEBUG "metric": { 07:12:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:12:23 DEBUG "result": [ 07:12:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:12:23 DEBUG }, 07:12:23 DEBUG { 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG "metric": { 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 07:12:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG }, 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 07:12:23 DEBUG [...] 07:12:23 DEBUG 07:12:23 DEBUG [...] 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG [...] 07:12:23 DEBUG 07:12:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:12:23 DEBUG 07:12:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:12:23 DEBUG }, 07:12:23 DEBUG 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 1712815943, 07:12:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:12:23 DEBUG 1712815943, 07:12:23 DEBUG "0.023" 07:12:23 DEBUG }, 07:12:23 DEBUG "2432123.2795849256" 07:12:23 DEBUG ] 07:12:23 DEBUG "value": [ 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG 1712815943, 07:12:23 DEBUG } 07:12:23 DEBUG ] 07:12:23 DEBUG "217799" 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG } 07:12:23 DEBUG } 07:12:23 DEBUG } 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG } 07:12:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815943" 07:12:23 INFO [http_cmd]: http status code OK 07:12:23 DEBUG --- status code --- 07:12:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:12:23 DEBUG --- http response --- 07:12:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:12:23 DEBUG ----- output ----- 07:12:23 DEBUG { 07:12:23 DEBUG "status": "success", 07:12:23 DEBUG "data": { 07:12:23 DEBUG "resultType": "vector", 07:12:23 DEBUG "result": [ 07:12:23 DEBUG { 07:12:23 DEBUG "metric": { 07:12:23 DEBUG "pod": "ds-idrepo-2" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG [...] 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG "pod": "ds-idrepo-0" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 1712815943, 07:12:23 DEBUG "0" 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG } 07:12:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815943" 07:12:23 INFO [http_cmd]: http status code OK 07:12:23 DEBUG --- status code --- 07:12:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:12:23 DEBUG --- http response --- 07:12:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:12:23 DEBUG ----- output ----- 07:12:23 DEBUG { 07:12:23 DEBUG "status": "success", 07:12:23 DEBUG "data": { 07:12:23 DEBUG "resultType": "vector", 07:12:23 DEBUG "result": [ 07:12:23 DEBUG { 07:12:23 DEBUG "metric": { 07:12:23 DEBUG "instance": "10.142.0.107:9100" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG [...] 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG "instance": "10.142.0.77:9100" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 1712815943, 07:12:23 DEBUG "0.0025999999999998615" 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG } 07:12:23 INFO [http_cmd]: http status code OK 07:12:23 DEBUG --- status code --- 07:12:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:12:23 DEBUG --- http response --- 07:12:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:12:23 DEBUG ----- output ----- 07:12:23 DEBUG { 07:12:23 DEBUG "status": "success", 07:12:23 DEBUG "data": { 07:12:23 DEBUG "resultType": "vector", 07:12:23 DEBUG "result": [ 07:12:23 DEBUG { 07:12:23 DEBUG "metric": { 07:12:23 DEBUG "instance": "10.142.0.107:9100" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG [...] 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG "instance": "10.142.0.77:9100" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 1712815943, 07:12:23 DEBUG "74813154.63333334" 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG } 07:12:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815943" 07:12:23 INFO [http_cmd]: http status code OK 07:12:23 DEBUG --- status code --- 07:12:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:12:23 DEBUG --- http response --- 07:12:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 07:12:23 DEBUG ----- output ----- 07:12:23 DEBUG { 07:12:23 DEBUG "status": "success", 07:12:23 DEBUG "data": { 07:12:23 DEBUG "resultType": "vector", 07:12:23 DEBUG "result": [ 07:12:23 DEBUG { 07:12:23 DEBUG "metric": { 07:12:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG [...] 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 1712815943, 07:12:23 DEBUG "0" 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG } 07:12:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815943" 07:12:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712815943" 07:12:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815943" 07:12:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815943" 07:12:23 INFO [http_cmd]: http status code OK 07:12:23 DEBUG --- status code --- 07:12:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:12:23 DEBUG --- http response --- 07:12:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 07:12:23 DEBUG ----- output ----- 07:12:23 DEBUG { 07:12:23 DEBUG "status": "success", 07:12:23 DEBUG "data": { 07:12:23 DEBUG "resultType": "vector", 07:12:23 DEBUG "result": [ 07:12:23 DEBUG { 07:12:23 DEBUG "metric": { 07:12:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG [...] 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 1712815943, 07:12:23 DEBUG "0" 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG } 07:12:23 INFO [http_cmd]: http status code OK 07:12:23 DEBUG --- status code --- 07:12:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:12:23 DEBUG --- http response --- 07:12:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:12:23 DEBUG ----- output ----- 07:12:23 DEBUG { 07:12:23 DEBUG "status": "success", 07:12:23 DEBUG "data": { 07:12:23 DEBUG "resultType": "vector", 07:12:23 DEBUG "result": [ 07:12:23 DEBUG { 07:12:23 DEBUG "metric": { 07:12:23 DEBUG "pod": "ds-idrepo-2" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG [...] 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG "pod": "ds-cts-2" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 1712815943, 07:12:23 DEBUG "0" 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG } 07:12:23 INFO [http_cmd]: http status code OK 07:12:23 DEBUG --- status code --- 07:12:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:12:23 DEBUG --- http response --- 07:12:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:12:23 DEBUG ----- output ----- 07:12:23 DEBUG { 07:12:23 DEBUG "status": "success", 07:12:23 DEBUG "data": { 07:12:23 DEBUG "resultType": "vector", 07:12:23 DEBUG "result": [ 07:12:23 DEBUG { 07:12:23 DEBUG "metric": { 07:12:23 DEBUG "pod": "ds-idrepo-2" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG [...] 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG "pod": "ds-idrepo-0" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 1712815943, 07:12:23 DEBUG "0" 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG } 07:12:23 INFO [http_cmd]: http status code OK 07:12:23 DEBUG --- status code --- 07:12:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:12:23 DEBUG --- http response --- 07:12:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:12:23 DEBUG ----- output ----- 07:12:23 DEBUG { 07:12:23 DEBUG "status": "success", 07:12:23 DEBUG "data": { 07:12:23 DEBUG "resultType": "vector", 07:12:23 DEBUG "result": [ 07:12:23 DEBUG { 07:12:23 DEBUG "metric": { 07:12:23 DEBUG "pod": "ds-idrepo-2" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG [...] 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG "pod": "ds-cts-2" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 1712815943, 07:12:23 DEBUG "29.03333333333333" 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG } 07:12:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815943" 07:12:23 INFO [http_cmd]: http status code OK 07:12:23 DEBUG --- status code --- 07:12:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:12:23 DEBUG --- http response --- 07:12:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:12:23 DEBUG ----- output ----- 07:12:23 DEBUG { 07:12:23 DEBUG "status": "success", 07:12:23 DEBUG "data": { 07:12:23 DEBUG "resultType": "vector", 07:12:23 DEBUG "result": [ 07:12:23 DEBUG { 07:12:23 DEBUG "metric": { 07:12:23 DEBUG "instance": "10.142.0.107:9100" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG [...] 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG "instance": "10.142.0.77:9100" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 1712815943, 07:12:23 DEBUG "52992035840" 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG } 07:12:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815943" 07:12:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815943" 07:12:23 INFO [http_cmd]: http status code OK 07:12:23 DEBUG --- status code --- 07:12:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:12:23 DEBUG --- http response --- 07:12:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:12:23 DEBUG ----- output ----- 07:12:23 DEBUG { 07:12:23 DEBUG "status": "success", 07:12:23 DEBUG "data": { 07:12:23 DEBUG "resultType": "vector", 07:12:23 DEBUG "result": [ 07:12:23 DEBUG { 07:12:23 DEBUG "metric": { 07:12:23 DEBUG "pod": "ds-idrepo-2" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG [...] 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG "pod": "ds-cts-2" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 1712815943, 07:12:23 DEBUG "0" 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG } 07:12:23 INFO [http_cmd]: http status code OK 07:12:23 DEBUG --- status code --- 07:12:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:12:23 DEBUG --- http response --- 07:12:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:12:23 DEBUG ----- output ----- 07:12:23 DEBUG { 07:12:23 DEBUG "status": "success", 07:12:23 DEBUG "data": { 07:12:23 DEBUG "resultType": "vector", 07:12:23 DEBUG "result": [ 07:12:23 DEBUG { 07:12:23 DEBUG "metric": { 07:12:23 DEBUG "instance": "10.142.0.107:9100" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG [...] 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG "instance": "10.142.0.77:9100" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 1712815943, 07:12:23 DEBUG "74816887.66666667" 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG } 07:12:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815943" 07:12:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815943" 07:12:23 INFO [http_cmd]: http status code OK 07:12:23 DEBUG --- status code --- 07:12:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:12:23 DEBUG --- http response --- 07:12:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:12:23 DEBUG ----- output ----- 07:12:23 DEBUG { 07:12:23 DEBUG "status": "success", 07:12:23 DEBUG "data": { 07:12:23 DEBUG "resultType": "vector", 07:12:23 DEBUG "result": [ 07:12:23 DEBUG { 07:12:23 DEBUG "metric": { 07:12:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG [...] 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 1712815943, 07:12:23 DEBUG "1205.3666666666666" 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG } 07:12:23 INFO [http_cmd]: http status code OK 07:12:23 DEBUG --- status code --- 07:12:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:12:23 DEBUG --- http response --- 07:12:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:12:23 DEBUG ----- output ----- 07:12:23 DEBUG { 07:12:23 DEBUG "status": "success", 07:12:23 DEBUG "data": { 07:12:23 DEBUG "resultType": "vector", 07:12:23 DEBUG "result": [ 07:12:23 DEBUG { 07:12:23 DEBUG "metric": { 07:12:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG [...] 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 1712815943, 07:12:23 DEBUG "1205.0333333333333" 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG } 07:12:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815943" 07:12:23 INFO [http_cmd]: http status code OK 07:12:23 DEBUG --- status code --- 07:12:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:12:23 DEBUG --- http response --- 07:12:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:12:23 DEBUG ----- output ----- 07:12:23 DEBUG { 07:12:23 DEBUG "status": "success", 07:12:23 DEBUG "data": { 07:12:23 DEBUG "resultType": "vector", 07:12:23 DEBUG "result": [ 07:12:23 DEBUG { 07:12:23 DEBUG "metric": { 07:12:23 DEBUG "pod": "ds-idrepo-2" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG [...] 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG "pod": "ds-cts-2" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 1712815943, 07:12:23 DEBUG "0.00046890833333333335" 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG } 07:12:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815943" 07:12:23 INFO [http_cmd]: http status code OK 07:12:23 DEBUG --- status code --- 07:12:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:12:23 DEBUG --- http response --- 07:12:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:12:23 DEBUG ----- output ----- 07:12:23 DEBUG { 07:12:23 DEBUG "status": "success", 07:12:23 DEBUG "data": { 07:12:23 DEBUG "resultType": "vector", 07:12:23 DEBUG "result": [ 07:12:23 DEBUG { 07:12:23 DEBUG "metric": { 07:12:23 DEBUG "pod": "ds-idrepo-2" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG [...] 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG "pod": "ds-idrepo-0" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 1712815943, 07:12:23 DEBUG "0" 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG } 07:12:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712815943" 07:12:23 INFO [http_cmd]: http status code OK 07:12:23 DEBUG --- status code --- 07:12:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:12:23 DEBUG --- http response --- 07:12:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:12:23 DEBUG ----- output ----- 07:12:23 DEBUG { 07:12:23 DEBUG "status": "success", 07:12:23 DEBUG "data": { 07:12:23 DEBUG "resultType": "vector", 07:12:23 DEBUG "result": [ 07:12:23 DEBUG { 07:12:23 DEBUG "metric": { 07:12:23 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG [...] 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 1712815943, 07:12:23 DEBUG "14101864448" 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG } 07:12:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815943" 07:12:23 INFO [http_cmd]: http status code OK 07:12:23 DEBUG --- status code --- 07:12:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:12:23 DEBUG --- http response --- 07:12:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:12:23 DEBUG ----- output ----- 07:12:23 DEBUG { 07:12:23 DEBUG "status": "success", 07:12:23 DEBUG "data": { 07:12:23 DEBUG "resultType": "vector", 07:12:23 DEBUG "result": [ 07:12:23 DEBUG { 07:12:23 DEBUG "metric": { 07:12:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG [...] 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 1712815943, 07:12:23 DEBUG "1214.7333333333333" 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG } 07:12:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712815943" 07:12:23 INFO [http_cmd]: http status code OK 07:12:23 DEBUG --- status code --- 07:12:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:12:23 DEBUG --- http response --- 07:12:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:12:23 DEBUG ----- output ----- 07:12:23 DEBUG { 07:12:23 DEBUG "status": "success", 07:12:23 DEBUG "data": { 07:12:23 DEBUG "resultType": "vector", 07:12:23 DEBUG "result": [ 07:12:23 DEBUG { 07:12:23 DEBUG "metric": { 07:12:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG [...] 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 1712815943, 07:12:23 DEBUG "0" 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG } 07:12:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712815943" 07:12:23 INFO [http_cmd]: http status code OK 07:12:23 DEBUG --- status code --- 07:12:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:12:23 DEBUG --- http response --- 07:12:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:12:23 DEBUG ----- output ----- 07:12:23 DEBUG { 07:12:23 DEBUG "status": "success", 07:12:23 DEBUG "data": { 07:12:23 DEBUG "resultType": "vector", 07:12:23 DEBUG "result": [ 07:12:23 DEBUG { 07:12:23 DEBUG "metric": { 07:12:23 DEBUG "instance": "10.142.0.107:9100" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG [...] 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG 07:12:23 DEBUG "instance": "10.142.0.77:9100" 07:12:23 DEBUG }, 07:12:23 DEBUG "value": [ 07:12:23 DEBUG 1712815943, 07:12:23 DEBUG "0.0013333333333341336" 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG ] 07:12:23 DEBUG } 07:12:23 DEBUG } 07:12:35 INFO 07:12:35 INFO [loop_until]: kubectl --namespace=xlou top pods 07:12:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:12:35 INFO [loop_until]: OK (rc = 0) 07:12:35 DEBUG --- stdout --- 07:12:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 8331m 8673Mi am-5b5bd8567b-8h2w4 7969m 8861Mi am-5b5bd8567b-qlkw6 7095m 8531Mi ds-cts-0 7200m 12496Mi ds-cts-1 7614m 13403Mi ds-cts-2 7514m 12620Mi ds-idrepo-0 9963m 5503Mi ds-idrepo-1 316m 5480Mi ds-idrepo-2 364m 5784Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 3m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 4m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 7776m 7590Mi 07:12:35 DEBUG --- stderr --- 07:12:35 DEBUG 07:12:39 INFO 07:12:39 INFO [loop_until]: kubectl --namespace=xlou top node 07:12:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:12:39 INFO [loop_until]: OK (rc = 0) 07:12:39 DEBUG --- stdout --- 07:12:39 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 2731Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 65m 0% 2778Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7154m 23% 10208Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 12130m 40% 3640Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 8868m 29% 10292Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 144m 0% 3843Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 8827m 29% 10518Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 429m 2% 6475Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 10111m 63% 6202Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 7593m 47% 13205Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 7313m 46% 13201Mi 22% gke-xlou-cdm-ds-32e4dcb1-j2hk 7688m 48% 14063Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 373m 2% 6165Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 8205m 51% 10923Mi 18% 07:12:39 DEBUG --- stderr --- 07:12:39 DEBUG 07:13:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712816003" 07:13:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816003" 07:13:23 ERROR [http_cmd]: ERROR 07:13:23 ERROR --- status code --- 07:13:23 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:13:23 ERROR --- http response --- 07:13:23 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:13:23 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:13:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816003" 07:13:23 INFO [http_cmd]: http status code OK 07:13:23 DEBUG --- status code --- 07:13:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:13:23 DEBUG --- http response --- 07:13:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712816003" 07:13:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:13:23 DEBUG ----- output ----- 07:13:23 DEBUG { 07:13:23 DEBUG "status": "success", 07:13:23 DEBUG "data": { 07:13:23 DEBUG "resultType": "vector", 07:13:23 DEBUG "result": [ 07:13:23 DEBUG { 07:13:23 DEBUG "metric": { 07:13:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG [...] 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 1712816003, 07:13:23 DEBUG "1246.0333333333333" 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG } 07:13:23 INFO [http_cmd]: http status code OK 07:13:23 DEBUG --- status code --- 07:13:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:13:23 DEBUG --- http response --- 07:13:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:13:23 DEBUG ----- output ----- 07:13:23 DEBUG { 07:13:23 DEBUG "status": "success", 07:13:23 DEBUG "data": { 07:13:23 DEBUG "resultType": "vector", 07:13:23 DEBUG "result": [ 07:13:23 DEBUG { 07:13:23 DEBUG "metric": { 07:13:23 DEBUG "pod": "ds-idrepo-2" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG [...] 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG "pod": "ds-idrepo-0" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 1712816003, 07:13:23 DEBUG "0" 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG } 07:13:23 INFO [http_cmd]: http status code OK 07:13:23 DEBUG --- status code --- 07:13:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:13:23 DEBUG --- http response --- 07:13:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:13:23 DEBUG ----- output ----- 07:13:23 DEBUG { 07:13:23 DEBUG "status": "success", 07:13:23 DEBUG "data": { 07:13:23 DEBUG "resultType": "vector", 07:13:23 DEBUG "result": [ 07:13:23 DEBUG { 07:13:23 DEBUG "metric": { 07:13:23 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG [...] 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 1712816003, 07:13:23 DEBUG "8.03675638277215" 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG } 07:13:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816003" 07:13:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816003" 07:13:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816003" 07:13:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816003" 07:13:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712816003" 07:13:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816003" 07:13:23 INFO [http_cmd]: http status code OK 07:13:23 INFO [http_cmd]: http status code OK 07:13:23 INFO [http_cmd]: http status code OK 07:13:23 DEBUG --- status code --- 07:13:23 DEBUG --- status code --- 07:13:23 DEBUG --- status code --- 07:13:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:13:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:13:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:13:23 DEBUG --- http response --- 07:13:23 DEBUG --- http response --- 07:13:23 DEBUG --- http response --- 07:13:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:13:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:13:23 INFO [http_cmd]: http status code OK 07:13:23 DEBUG ----- output ----- 07:13:23 INFO [http_cmd]: http status code OK 07:13:23 INFO [http_cmd]: http status code OK 07:13:23 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:13:23 DEBUG ----- output ----- 07:13:23 DEBUG --- status code --- 07:13:23 DEBUG { 07:13:23 DEBUG --- status code --- 07:13:23 DEBUG --- status code --- 07:13:23 DEBUG { 07:13:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:13:23 DEBUG "status": "success", 07:13:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:13:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:13:23 DEBUG "status": "success", 07:13:23 DEBUG --- http response --- 07:13:23 DEBUG "data": { 07:13:23 DEBUG --- http response --- 07:13:23 DEBUG --- http response --- 07:13:23 DEBUG "data": { 07:13:23 DEBUG "resultType": "vector", 07:13:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:13:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:13:23 DEBUG "resultType": "vector", 07:13:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:13:23 DEBUG "result": [ 07:13:23 DEBUG ----- output ----- 07:13:23 DEBUG ----- output ----- 07:13:23 DEBUG "result": [ 07:13:23 DEBUG ----- output ----- 07:13:23 DEBUG { 07:13:23 DEBUG { 07:13:23 DEBUG { 07:13:23 DEBUG { 07:13:23 DEBUG { 07:13:23 DEBUG "metric": { 07:13:23 DEBUG "status": "success", 07:13:23 DEBUG "status": "success", 07:13:23 DEBUG "metric": { 07:13:23 DEBUG "status": "success", 07:13:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:13:23 DEBUG "data": { 07:13:23 DEBUG "data": { 07:13:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:13:23 DEBUG "data": { 07:13:23 DEBUG }, 07:13:23 DEBUG "resultType": "vector", 07:13:23 DEBUG "resultType": "vector", 07:13:23 DEBUG }, 07:13:23 DEBUG "resultType": "vector", 07:13:23 DEBUG "value": [ 07:13:23 DEBUG "result": [ 07:13:23 DEBUG "result": [ 07:13:23 DEBUG "value": [ 07:13:23 DEBUG "result": [ 07:13:23 DEBUG 07:13:23 DEBUG { 07:13:23 DEBUG { 07:13:23 DEBUG 07:13:23 DEBUG { 07:13:23 DEBUG 07:13:23 DEBUG "metric": { 07:13:23 DEBUG "metric": { 07:13:23 DEBUG 07:13:23 DEBUG "metric": { 07:13:23 DEBUG 07:13:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:13:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:13:23 DEBUG 07:13:23 DEBUG "instance": "10.142.0.107:9100" 07:13:23 DEBUG [...] 07:13:23 DEBUG }, 07:13:23 DEBUG }, 07:13:23 DEBUG [...] 07:13:23 DEBUG }, 07:13:23 DEBUG 07:13:23 DEBUG "value": [ 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:13:23 DEBUG 07:13:23 DEBUG }, 07:13:23 DEBUG [...] 07:13:23 DEBUG [...] 07:13:23 DEBUG }, 07:13:23 DEBUG [...] 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 07:13:23 DEBUG 1712816003, 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 1712816003, 07:13:23 DEBUG 07:13:23 DEBUG "0.023" 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG "8.000556036102086" 07:13:23 DEBUG 07:13:23 DEBUG ] 07:13:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:13:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:13:23 DEBUG ] 07:13:23 DEBUG "instance": "10.142.0.77:9100" 07:13:23 DEBUG } 07:13:23 DEBUG }, 07:13:23 DEBUG }, 07:13:23 DEBUG } 07:13:23 DEBUG }, 07:13:23 DEBUG ] 07:13:23 DEBUG "value": [ 07:13:23 DEBUG "value": [ 07:13:23 DEBUG ] 07:13:23 DEBUG "value": [ 07:13:23 DEBUG } 07:13:23 DEBUG 1712816003, 07:13:23 DEBUG 1712816003, 07:13:23 DEBUG } 07:13:23 DEBUG 1712816003, 07:13:23 DEBUG } 07:13:23 DEBUG "3395702.3495491077" 07:13:23 DEBUG "215436.5" 07:13:23 DEBUG } 07:13:23 DEBUG "72740978.16666667" 07:13:23 DEBUG ] 07:13:23 DEBUG ] 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG } 07:13:23 DEBUG } 07:13:23 DEBUG ] 07:13:23 DEBUG ] 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG } 07:13:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816003" 07:13:23 DEBUG } 07:13:23 DEBUG } 07:13:23 DEBUG } 07:13:23 DEBUG } 07:13:23 INFO [http_cmd]: http status code OK 07:13:23 DEBUG --- status code --- 07:13:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:13:23 DEBUG --- http response --- 07:13:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:13:23 DEBUG ----- output ----- 07:13:23 DEBUG { 07:13:23 DEBUG "status": "success", 07:13:23 DEBUG "data": { 07:13:23 DEBUG "resultType": "vector", 07:13:23 DEBUG "result": [ 07:13:23 DEBUG { 07:13:23 DEBUG "metric": { 07:13:23 DEBUG "instance": "10.142.0.107:9100" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG [...] 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG "instance": "10.142.0.77:9100" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 1712816003, 07:13:23 DEBUG "0.001966666666666583" 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG } 07:13:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816003" 07:13:23 INFO [http_cmd]: http status code OK 07:13:23 DEBUG --- status code --- 07:13:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:13:23 DEBUG --- http response --- 07:13:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 07:13:23 DEBUG ----- output ----- 07:13:23 DEBUG { 07:13:23 DEBUG "status": "success", 07:13:23 DEBUG "data": { 07:13:23 DEBUG "resultType": "vector", 07:13:23 DEBUG "result": [ 07:13:23 DEBUG { 07:13:23 DEBUG "metric": { 07:13:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG [...] 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 1712816003, 07:13:23 DEBUG "0" 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG } 07:13:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816003" 07:13:23 INFO [http_cmd]: http status code OK 07:13:23 DEBUG --- status code --- 07:13:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:13:23 DEBUG --- http response --- 07:13:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 07:13:23 DEBUG ----- output ----- 07:13:23 DEBUG { 07:13:23 DEBUG "status": "success", 07:13:23 DEBUG "data": { 07:13:23 DEBUG "resultType": "vector", 07:13:23 DEBUG "result": [ 07:13:23 DEBUG { 07:13:23 DEBUG "metric": { 07:13:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG [...] 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 1712816003, 07:13:23 DEBUG "0" 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG } 07:13:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816003" 07:13:23 INFO [http_cmd]: http status code OK 07:13:23 DEBUG --- status code --- 07:13:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:13:23 DEBUG --- http response --- 07:13:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:13:23 DEBUG ----- output ----- 07:13:23 DEBUG { 07:13:23 DEBUG "status": "success", 07:13:23 DEBUG "data": { 07:13:23 DEBUG "resultType": "vector", 07:13:23 DEBUG "result": [ 07:13:23 DEBUG { 07:13:23 DEBUG "metric": { 07:13:23 DEBUG "instance": "10.142.0.107:9100" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG [...] 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG "instance": "10.142.0.77:9100" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 1712816003, 07:13:23 DEBUG "72742872" 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG } 07:13:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816003" 07:13:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712816003" 07:13:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816003" 07:13:23 INFO [http_cmd]: http status code OK 07:13:23 INFO [http_cmd]: http status code OK 07:13:23 DEBUG --- status code --- 07:13:23 DEBUG --- status code --- 07:13:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:13:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:13:23 DEBUG --- http response --- 07:13:23 DEBUG --- http response --- 07:13:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:13:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:13:23 DEBUG ----- output ----- 07:13:23 DEBUG ----- output ----- 07:13:23 DEBUG { 07:13:23 DEBUG { 07:13:23 DEBUG "status": "success", 07:13:23 DEBUG "status": "success", 07:13:23 DEBUG "data": { 07:13:23 DEBUG "data": { 07:13:23 DEBUG "resultType": "vector", 07:13:23 DEBUG "resultType": "vector", 07:13:23 DEBUG "result": [ 07:13:23 DEBUG "result": [ 07:13:23 DEBUG { 07:13:23 DEBUG { 07:13:23 DEBUG "metric": { 07:13:23 DEBUG "metric": { 07:13:23 DEBUG "pod": "ds-idrepo-2" 07:13:23 DEBUG "pod": "ds-idrepo-2" 07:13:23 DEBUG }, 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG [...] 07:13:23 DEBUG [...] 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG "pod": "ds-cts-2" 07:13:23 DEBUG "pod": "ds-cts-2" 07:13:23 DEBUG }, 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 1712816003, 07:13:23 DEBUG 1712816003, 07:13:23 DEBUG "0" 07:13:23 DEBUG "0" 07:13:23 DEBUG ] 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG } 07:13:23 DEBUG ] 07:13:23 DEBUG ] 07:13:23 INFO [http_cmd]: http status code OK 07:13:23 DEBUG } 07:13:23 DEBUG } 07:13:23 DEBUG --- status code --- 07:13:23 DEBUG } 07:13:23 DEBUG } 07:13:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:13:23 DEBUG --- http response --- 07:13:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:13:23 DEBUG ----- output ----- 07:13:23 DEBUG { 07:13:23 DEBUG "status": "success", 07:13:23 DEBUG "data": { 07:13:23 DEBUG "resultType": "vector", 07:13:23 DEBUG "result": [ 07:13:23 DEBUG { 07:13:23 DEBUG "metric": { 07:13:23 DEBUG "pod": "ds-idrepo-2" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG [...] 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG "pod": "ds-idrepo-0" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 1712816003, 07:13:23 DEBUG "0" 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG } 07:13:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816003" 07:13:23 INFO [http_cmd]: http status code OK 07:13:23 DEBUG --- status code --- 07:13:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:13:23 DEBUG --- http response --- 07:13:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:13:23 DEBUG ----- output ----- 07:13:23 DEBUG { 07:13:23 DEBUG "status": "success", 07:13:23 DEBUG "data": { 07:13:23 DEBUG "resultType": "vector", 07:13:23 DEBUG "result": [ 07:13:23 DEBUG { 07:13:23 DEBUG "metric": { 07:13:23 DEBUG "instance": "10.142.0.107:9100" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG [...] 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG "instance": "10.142.0.77:9100" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 1712816003, 07:13:23 DEBUG "52993491968" 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG } 07:13:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816003" 07:13:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816003" 07:13:23 INFO [http_cmd]: http status code OK 07:13:23 DEBUG --- status code --- 07:13:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:13:23 DEBUG --- http response --- 07:13:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:13:23 DEBUG ----- output ----- 07:13:23 DEBUG { 07:13:23 DEBUG "status": "success", 07:13:23 DEBUG "data": { 07:13:23 DEBUG "resultType": "vector", 07:13:23 DEBUG "result": [ 07:13:23 DEBUG { 07:13:23 DEBUG "metric": { 07:13:23 DEBUG "pod": "ds-idrepo-2" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG [...] 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG "pod": "ds-cts-2" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 1712816003, 07:13:23 DEBUG "0.00023333333333333333" 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG } 07:13:23 INFO [http_cmd]: http status code OK 07:13:23 DEBUG --- status code --- 07:13:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:13:23 DEBUG --- http response --- 07:13:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:13:23 DEBUG ----- output ----- 07:13:23 DEBUG { 07:13:23 DEBUG "status": "success", 07:13:23 DEBUG "data": { 07:13:23 DEBUG "resultType": "vector", 07:13:23 DEBUG "result": [ 07:13:23 DEBUG { 07:13:23 DEBUG "metric": { 07:13:23 DEBUG "pod": "ds-idrepo-2" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG [...] 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG "pod": "ds-cts-2" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 1712816003, 07:13:23 DEBUG "0" 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG } 07:13:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816003" 07:13:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816003" 07:13:23 INFO [http_cmd]: http status code OK 07:13:23 DEBUG --- status code --- 07:13:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:13:23 DEBUG --- http response --- 07:13:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:13:23 DEBUG ----- output ----- 07:13:23 DEBUG { 07:13:23 DEBUG "status": "success", 07:13:23 DEBUG "data": { 07:13:23 DEBUG "resultType": "vector", 07:13:23 DEBUG "result": [ 07:13:23 DEBUG { 07:13:23 DEBUG "metric": { 07:13:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG [...] 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 1712816003, 07:13:23 DEBUG "1178.0666666666666" 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG } 07:13:23 INFO [http_cmd]: http status code OK 07:13:23 DEBUG --- status code --- 07:13:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:13:23 DEBUG --- http response --- 07:13:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:13:23 DEBUG ----- output ----- 07:13:23 DEBUG { 07:13:23 DEBUG "status": "success", 07:13:23 DEBUG "data": { 07:13:23 DEBUG "resultType": "vector", 07:13:23 DEBUG "result": [ 07:13:23 DEBUG { 07:13:23 DEBUG "metric": { 07:13:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG [...] 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 1712816003, 07:13:23 DEBUG "1178.4333333333334" 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG } 07:13:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816003" 07:13:23 INFO [http_cmd]: http status code OK 07:13:23 DEBUG --- status code --- 07:13:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:13:23 DEBUG --- http response --- 07:13:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:13:23 DEBUG ----- output ----- 07:13:23 DEBUG { 07:13:23 DEBUG "status": "success", 07:13:23 DEBUG "data": { 07:13:23 DEBUG "resultType": "vector", 07:13:23 DEBUG "result": [ 07:13:23 DEBUG { 07:13:23 DEBUG "metric": { 07:13:23 DEBUG "pod": "ds-idrepo-2" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG [...] 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG "pod": "ds-idrepo-0" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 1712816003, 07:13:23 DEBUG "0" 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG } 07:13:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712816003" 07:13:23 INFO [http_cmd]: http status code OK 07:13:23 DEBUG --- status code --- 07:13:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:13:23 DEBUG --- http response --- 07:13:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:13:23 DEBUG ----- output ----- 07:13:23 DEBUG { 07:13:23 DEBUG "status": "success", 07:13:23 DEBUG "data": { 07:13:23 DEBUG "resultType": "vector", 07:13:23 DEBUG "result": [ 07:13:23 DEBUG { 07:13:23 DEBUG "metric": { 07:13:23 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG [...] 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 1712816003, 07:13:23 DEBUG "13566226432" 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG } 07:13:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816003" 07:13:23 INFO [http_cmd]: http status code OK 07:13:23 DEBUG --- status code --- 07:13:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:13:23 DEBUG --- http response --- 07:13:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:13:23 DEBUG ----- output ----- 07:13:23 DEBUG { 07:13:23 DEBUG "status": "success", 07:13:23 DEBUG "data": { 07:13:23 DEBUG "resultType": "vector", 07:13:23 DEBUG "result": [ 07:13:23 DEBUG { 07:13:23 DEBUG "metric": { 07:13:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG [...] 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 1712816003, 07:13:23 DEBUG "1180.1333333333334" 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG } 07:13:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816003" 07:13:23 INFO [http_cmd]: http status code OK 07:13:23 DEBUG --- status code --- 07:13:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:13:23 DEBUG --- http response --- 07:13:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:13:23 DEBUG ----- output ----- 07:13:23 DEBUG { 07:13:23 DEBUG "status": "success", 07:13:23 DEBUG "data": { 07:13:23 DEBUG "resultType": "vector", 07:13:23 DEBUG "result": [ 07:13:23 DEBUG { 07:13:23 DEBUG "metric": { 07:13:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG [...] 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 1712816003, 07:13:23 DEBUG "0" 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG } 07:13:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816003" 07:13:23 INFO [http_cmd]: http status code OK 07:13:23 DEBUG --- status code --- 07:13:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:13:23 DEBUG --- http response --- 07:13:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:13:23 DEBUG ----- output ----- 07:13:23 DEBUG { 07:13:23 DEBUG "status": "success", 07:13:23 DEBUG "data": { 07:13:23 DEBUG "resultType": "vector", 07:13:23 DEBUG "result": [ 07:13:23 DEBUG { 07:13:23 DEBUG "metric": { 07:13:23 DEBUG "instance": "10.142.0.107:9100" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG [...] 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG 07:13:23 DEBUG "instance": "10.142.0.77:9100" 07:13:23 DEBUG }, 07:13:23 DEBUG "value": [ 07:13:23 DEBUG 1712816003, 07:13:23 DEBUG "0" 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG ] 07:13:23 DEBUG } 07:13:23 DEBUG } 07:13:35 INFO 07:13:35 INFO [loop_until]: kubectl --namespace=xlou top pods 07:13:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:13:35 INFO [loop_until]: OK (rc = 0) 07:13:35 DEBUG --- stdout --- 07:13:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 8211m 8675Mi am-5b5bd8567b-8h2w4 7758m 8870Mi am-5b5bd8567b-qlkw6 6747m 8532Mi ds-cts-0 7239m 13026Mi ds-cts-1 7498m 13084Mi ds-cts-2 7383m 13147Mi ds-idrepo-0 9940m 5503Mi ds-idrepo-1 295m 5480Mi ds-idrepo-2 321m 5785Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 4m 1010Mi idm-668d898cfd-st2t7 3m 1135Mi lodemon-ccc964465-p7gdm 3m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 7990m 7765Mi 07:13:35 DEBUG --- stderr --- 07:13:35 DEBUG 07:13:39 INFO 07:13:39 INFO [loop_until]: kubectl --namespace=xlou top node 07:13:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:13:39 INFO [loop_until]: OK (rc = 0) 07:13:39 DEBUG --- stdout --- 07:13:39 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 2724Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 62m 0% 2784Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7054m 23% 10211Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 11312m 37% 3679Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 8638m 28% 10296Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 140m 0% 3835Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 8589m 28% 10518Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 382m 2% 6473Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 10002m 62% 6207Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 7618m 47% 13777Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 7195m 45% 13791Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 7560m 47% 13646Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 361m 2% 6164Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 8210m 51% 11160Mi 19% 07:13:39 DEBUG --- stderr --- 07:13:39 DEBUG 07:14:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712816063" 07:14:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712816063" 07:14:23 INFO [http_cmd]: http status code OK 07:14:23 DEBUG --- status code --- 07:14:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:14:23 DEBUG --- http response --- 07:14:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:14:23 DEBUG ----- output ----- 07:14:23 DEBUG { 07:14:23 DEBUG "status": "success", 07:14:23 DEBUG "data": { 07:14:23 DEBUG "resultType": "vector", 07:14:23 DEBUG "result": [ 07:14:23 DEBUG { 07:14:23 DEBUG "metric": { 07:14:23 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG [...] 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 1712816063, 07:14:23 DEBUG "8.032140834659424" 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG } 07:14:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816063" 07:14:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816063" 07:14:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816063" 07:14:23 ERROR [http_cmd]: ERROR 07:14:23 ERROR --- status code --- 07:14:23 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:14:23 ERROR --- http response --- 07:14:23 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:14:23 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:14:23 INFO [http_cmd]: http status code OK 07:14:23 DEBUG --- status code --- 07:14:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:14:23 DEBUG --- http response --- 07:14:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:14:23 DEBUG ----- output ----- 07:14:23 DEBUG { 07:14:23 DEBUG "status": "success", 07:14:23 DEBUG "data": { 07:14:23 DEBUG "resultType": "vector", 07:14:23 DEBUG "result": [ 07:14:23 DEBUG { 07:14:23 DEBUG "metric": { 07:14:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG [...] 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 1712816063, 07:14:23 DEBUG "1231.9" 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG } 07:14:23 INFO [http_cmd]: http status code OK 07:14:23 INFO [http_cmd]: http status code OK 07:14:23 DEBUG --- status code --- 07:14:23 DEBUG --- status code --- 07:14:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:14:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:14:23 DEBUG --- http response --- 07:14:23 DEBUG --- http response --- 07:14:23 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:14:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 07:14:23 DEBUG ----- output ----- 07:14:23 DEBUG { 07:14:23 DEBUG "status": "success", 07:14:23 DEBUG "data": { 07:14:23 DEBUG "resultType": "vector", 07:14:23 DEBUG "result": [ 07:14:23 DEBUG { 07:14:23 DEBUG "metric": { 07:14:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG [...] 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG "pod": "ds-idrepo-1" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 1712816063, 07:14:23 DEBUG "0.20551928501768033" 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG } 07:14:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816063" 07:14:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816063" 07:14:23 INFO [http_cmd]: http status code OK 07:14:23 DEBUG --- status code --- 07:14:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:14:23 DEBUG --- http response --- 07:14:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:14:23 DEBUG ----- output ----- 07:14:23 DEBUG { 07:14:23 DEBUG "status": "success", 07:14:23 DEBUG "data": { 07:14:23 DEBUG "resultType": "vector", 07:14:23 DEBUG "result": [ 07:14:23 DEBUG { 07:14:23 DEBUG "metric": { 07:14:23 DEBUG "instance": "10.142.0.107:9100" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG [...] 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG "instance": "10.142.0.77:9100" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 1712816063, 07:14:23 DEBUG "73978502.6" 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG } 07:14:23 INFO [http_cmd]: http status code OK 07:14:23 DEBUG --- status code --- 07:14:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:14:23 DEBUG --- http response --- 07:14:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:14:23 DEBUG ----- output ----- 07:14:23 DEBUG { 07:14:23 DEBUG "status": "success", 07:14:23 DEBUG "data": { 07:14:23 DEBUG "resultType": "vector", 07:14:23 DEBUG "result": [ 07:14:23 DEBUG { 07:14:23 DEBUG "metric": { 07:14:23 DEBUG "pod": "ds-idrepo-2" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG [...] 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG "pod": "ds-idrepo-0" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 1712816063, 07:14:23 DEBUG "0" 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG } 07:14:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712816063" 07:14:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816063" 07:14:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816063" 07:14:23 INFO [http_cmd]: http status code OK 07:14:23 DEBUG --- status code --- 07:14:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:14:23 DEBUG --- http response --- 07:14:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:14:23 DEBUG ----- output ----- 07:14:23 DEBUG { 07:14:23 DEBUG "status": "success", 07:14:23 DEBUG "data": { 07:14:23 DEBUG "resultType": "vector", 07:14:23 DEBUG "result": [ 07:14:23 DEBUG { 07:14:23 DEBUG "metric": { 07:14:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG [...] 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 1712816063, 07:14:23 DEBUG "0.023" 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG } 07:14:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816063" 07:14:23 INFO [http_cmd]: http status code OK 07:14:23 INFO [http_cmd]: http status code OK 07:14:23 DEBUG --- status code --- 07:14:23 DEBUG --- status code --- 07:14:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:14:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:14:23 DEBUG --- http response --- 07:14:23 DEBUG --- http response --- 07:14:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:14:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 07:14:23 DEBUG ----- output ----- 07:14:23 DEBUG ----- output ----- 07:14:23 DEBUG { 07:14:23 DEBUG { 07:14:23 DEBUG "status": "success", 07:14:23 DEBUG "status": "success", 07:14:23 DEBUG "data": { 07:14:23 DEBUG "data": { 07:14:23 DEBUG "resultType": "vector", 07:14:23 DEBUG "resultType": "vector", 07:14:23 DEBUG "result": [ 07:14:23 DEBUG "result": [ 07:14:23 DEBUG { 07:14:23 DEBUG { 07:14:23 DEBUG "metric": { 07:14:23 DEBUG "metric": { 07:14:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:14:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:14:23 DEBUG }, 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG [...] 07:14:23 DEBUG [...] 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:14:23 DEBUG "pod": "ds-idrepo-1" 07:14:23 DEBUG }, 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 1712816063, 07:14:23 DEBUG 1712816063, 07:14:23 DEBUG "215111.5" 07:14:23 DEBUG "158957144.24291775" 07:14:23 DEBUG ] 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG } 07:14:23 DEBUG ] 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG } 07:14:23 DEBUG } 07:14:23 DEBUG } 07:14:23 INFO [http_cmd]: http status code OK 07:14:23 DEBUG --- status code --- 07:14:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:14:23 DEBUG --- http response --- 07:14:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:14:23 DEBUG ----- output ----- 07:14:23 DEBUG { 07:14:23 DEBUG "status": "success", 07:14:23 DEBUG "data": { 07:14:23 DEBUG "resultType": "vector", 07:14:23 DEBUG "result": [ 07:14:23 DEBUG { 07:14:23 DEBUG "metric": { 07:14:23 DEBUG "instance": "10.142.0.107:9100" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG [...] 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG "instance": "10.142.0.77:9100" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 1712816063, 07:14:23 DEBUG "0.002233333333333339" 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG } 07:14:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816063" 07:14:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816063" 07:14:23 INFO [http_cmd]: http status code OK 07:14:23 DEBUG --- status code --- 07:14:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:14:23 DEBUG --- http response --- 07:14:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 07:14:23 DEBUG ----- output ----- 07:14:23 DEBUG { 07:14:23 DEBUG "status": "success", 07:14:23 DEBUG "data": { 07:14:23 DEBUG "resultType": "vector", 07:14:23 DEBUG "result": [ 07:14:23 DEBUG { 07:14:23 DEBUG "metric": { 07:14:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG [...] 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG "pod": "ds-idrepo-1" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 1712816063, 07:14:23 DEBUG "2605.2253358505022" 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG } 07:14:23 INFO [http_cmd]: http status code OK 07:14:23 DEBUG --- status code --- 07:14:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:14:23 DEBUG --- http response --- 07:14:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 07:14:23 DEBUG ----- output ----- 07:14:23 DEBUG { 07:14:23 DEBUG "status": "success", 07:14:23 DEBUG "data": { 07:14:23 DEBUG "resultType": "vector", 07:14:23 DEBUG "result": [ 07:14:23 DEBUG { 07:14:23 DEBUG "metric": { 07:14:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG [...] 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG "pod": "ds-idrepo-1" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 1712816063, 07:14:23 DEBUG "0" 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG } 07:14:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816063" 07:14:23 INFO [http_cmd]: http status code OK 07:14:23 DEBUG --- status code --- 07:14:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:14:23 DEBUG --- http response --- 07:14:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:14:23 DEBUG ----- output ----- 07:14:23 DEBUG { 07:14:23 DEBUG "status": "success", 07:14:23 DEBUG "data": { 07:14:23 DEBUG "resultType": "vector", 07:14:23 DEBUG "result": [ 07:14:23 DEBUG { 07:14:23 DEBUG "metric": { 07:14:23 DEBUG "instance": "10.142.0.107:9100" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG [...] 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG "instance": "10.142.0.77:9100" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 1712816063, 07:14:23 DEBUG "73981384" 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG } 07:14:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816063" 07:14:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712816063" 07:14:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816063" 07:14:23 INFO [http_cmd]: http status code OK 07:14:23 INFO [http_cmd]: http status code OK 07:14:23 DEBUG --- status code --- 07:14:23 DEBUG --- status code --- 07:14:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:14:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:14:23 DEBUG --- http response --- 07:14:23 DEBUG --- http response --- 07:14:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:14:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:14:23 DEBUG ----- output ----- 07:14:23 DEBUG ----- output ----- 07:14:23 DEBUG { 07:14:23 DEBUG { 07:14:23 DEBUG "status": "success", 07:14:23 DEBUG "status": "success", 07:14:23 DEBUG "data": { 07:14:23 DEBUG "data": { 07:14:23 DEBUG "resultType": "vector", 07:14:23 DEBUG "resultType": "vector", 07:14:23 DEBUG "result": [ 07:14:23 DEBUG "result": [ 07:14:23 INFO [http_cmd]: http status code OK 07:14:23 DEBUG { 07:14:23 DEBUG { 07:14:23 DEBUG --- status code --- 07:14:23 DEBUG "metric": { 07:14:23 DEBUG "metric": { 07:14:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:14:23 DEBUG "pod": "ds-idrepo-2" 07:14:23 DEBUG "pod": "ds-idrepo-2" 07:14:23 DEBUG --- http response --- 07:14:23 DEBUG }, 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 07:14:23 DEBUG ----- output ----- 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG { 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG "status": "success", 07:14:23 DEBUG 07:14:23 DEBUG [...] 07:14:23 DEBUG "data": { 07:14:23 DEBUG [...] 07:14:23 DEBUG 07:14:23 DEBUG "resultType": "vector", 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG "result": [ 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG { 07:14:23 DEBUG 07:14:23 DEBUG "pod": "ds-cts-2" 07:14:23 DEBUG "metric": { 07:14:23 DEBUG "pod": "ds-cts-2" 07:14:23 DEBUG }, 07:14:23 DEBUG "pod": "ds-idrepo-2" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 1712816063, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 1712816063, 07:14:23 DEBUG "0" 07:14:23 DEBUG 07:14:23 DEBUG "0" 07:14:23 DEBUG ] 07:14:23 DEBUG 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG 07:14:23 DEBUG } 07:14:23 DEBUG ] 07:14:23 DEBUG [...] 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG 07:14:23 DEBUG } 07:14:23 DEBUG } 07:14:23 DEBUG 07:14:23 DEBUG } 07:14:23 DEBUG 07:14:23 DEBUG "pod": "ds-idrepo-0" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 1712816063, 07:14:23 DEBUG "0" 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG } 07:14:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816063" 07:14:23 INFO [http_cmd]: http status code OK 07:14:23 DEBUG --- status code --- 07:14:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:14:23 DEBUG --- http response --- 07:14:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:14:23 DEBUG ----- output ----- 07:14:23 DEBUG { 07:14:23 DEBUG "status": "success", 07:14:23 DEBUG "data": { 07:14:23 DEBUG "resultType": "vector", 07:14:23 DEBUG "result": [ 07:14:23 DEBUG { 07:14:23 DEBUG "metric": { 07:14:23 DEBUG "instance": "10.142.0.107:9100" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG [...] 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG "instance": "10.142.0.77:9100" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 1712816063, 07:14:23 DEBUG "52994765824" 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG } 07:14:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816063" 07:14:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816063" 07:14:23 INFO [http_cmd]: http status code OK 07:14:23 DEBUG --- status code --- 07:14:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:14:23 DEBUG --- http response --- 07:14:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:14:23 DEBUG ----- output ----- 07:14:23 DEBUG { 07:14:23 DEBUG "status": "success", 07:14:23 DEBUG "data": { 07:14:23 DEBUG "resultType": "vector", 07:14:23 DEBUG "result": [ 07:14:23 DEBUG { 07:14:23 DEBUG "metric": { 07:14:23 DEBUG "pod": "ds-idrepo-2" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG [...] 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG "pod": "ds-cts-2" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 1712816063, 07:14:23 DEBUG "0" 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG } 07:14:23 INFO [http_cmd]: http status code OK 07:14:23 DEBUG --- status code --- 07:14:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:14:23 DEBUG --- http response --- 07:14:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:14:23 DEBUG ----- output ----- 07:14:23 DEBUG { 07:14:23 DEBUG "status": "success", 07:14:23 DEBUG "data": { 07:14:23 DEBUG "resultType": "vector", 07:14:23 DEBUG "result": [ 07:14:23 DEBUG { 07:14:23 DEBUG "metric": { 07:14:23 DEBUG "pod": "ds-idrepo-2" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG [...] 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG "pod": "ds-cts-2" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 1712816063, 07:14:23 DEBUG "0" 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG } 07:14:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816063" 07:14:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816063" 07:14:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816063" 07:14:23 INFO [http_cmd]: http status code OK 07:14:23 INFO [http_cmd]: http status code OK 07:14:23 DEBUG --- status code --- 07:14:23 DEBUG --- status code --- 07:14:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:14:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:14:23 DEBUG --- http response --- 07:14:23 DEBUG --- http response --- 07:14:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:14:23 DEBUG ----- output ----- 07:14:23 DEBUG { 07:14:23 DEBUG "status": "success", 07:14:23 DEBUG "data": { 07:14:23 DEBUG "resultType": "vector", 07:14:23 DEBUG "result": [ 07:14:23 DEBUG { 07:14:23 DEBUG "metric": { 07:14:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG [...] 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 1712816063, 07:14:23 DEBUG "1195.5666666666666" 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG } 07:14:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:14:23 DEBUG ----- output ----- 07:14:23 DEBUG { 07:14:23 DEBUG "status": "success", 07:14:23 DEBUG "data": { 07:14:23 DEBUG "resultType": "vector", 07:14:23 DEBUG "result": [ 07:14:23 DEBUG { 07:14:23 DEBUG "metric": { 07:14:23 DEBUG "pod": "ds-idrepo-2" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG [...] 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG "pod": "ds-idrepo-0" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 1712816063, 07:14:23 DEBUG "0" 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG } 07:14:23 INFO [http_cmd]: http status code OK 07:14:23 DEBUG --- status code --- 07:14:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:14:23 DEBUG --- http response --- 07:14:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:14:23 DEBUG ----- output ----- 07:14:23 DEBUG { 07:14:23 DEBUG "status": "success", 07:14:23 DEBUG "data": { 07:14:23 DEBUG "resultType": "vector", 07:14:23 DEBUG "result": [ 07:14:23 DEBUG { 07:14:23 DEBUG "metric": { 07:14:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG [...] 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 1712816063, 07:14:23 DEBUG "1196" 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG } 07:14:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712816063" 07:14:23 INFO [http_cmd]: http status code OK 07:14:23 DEBUG --- status code --- 07:14:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:14:23 DEBUG --- http response --- 07:14:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:14:23 DEBUG ----- output ----- 07:14:23 DEBUG { 07:14:23 DEBUG "status": "success", 07:14:23 DEBUG "data": { 07:14:23 DEBUG "resultType": "vector", 07:14:23 DEBUG "result": [ 07:14:23 DEBUG { 07:14:23 DEBUG "metric": { 07:14:23 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG [...] 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 1712816063, 07:14:23 DEBUG "13866696704" 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG } 07:14:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816063" 07:14:23 INFO [http_cmd]: http status code OK 07:14:23 DEBUG --- status code --- 07:14:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:14:23 DEBUG --- http response --- 07:14:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:14:23 DEBUG ----- output ----- 07:14:23 DEBUG { 07:14:23 DEBUG "status": "success", 07:14:23 DEBUG "data": { 07:14:23 DEBUG "resultType": "vector", 07:14:23 DEBUG "result": [ 07:14:23 DEBUG { 07:14:23 DEBUG "metric": { 07:14:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG [...] 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 1712816063, 07:14:23 DEBUG "1182" 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG } 07:14:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816063" 07:14:23 INFO [http_cmd]: http status code OK 07:14:23 DEBUG --- status code --- 07:14:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:14:23 DEBUG --- http response --- 07:14:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:14:23 DEBUG ----- output ----- 07:14:23 DEBUG { 07:14:23 DEBUG "status": "success", 07:14:23 DEBUG "data": { 07:14:23 DEBUG "resultType": "vector", 07:14:23 DEBUG "result": [ 07:14:23 DEBUG { 07:14:23 DEBUG "metric": { 07:14:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG [...] 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 1712816063, 07:14:23 DEBUG "0" 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG } 07:14:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816063" 07:14:23 INFO [http_cmd]: http status code OK 07:14:23 DEBUG --- status code --- 07:14:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:14:23 DEBUG --- http response --- 07:14:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:14:23 DEBUG ----- output ----- 07:14:23 DEBUG { 07:14:23 DEBUG "status": "success", 07:14:23 DEBUG "data": { 07:14:23 DEBUG "resultType": "vector", 07:14:23 DEBUG "result": [ 07:14:23 DEBUG { 07:14:23 DEBUG "metric": { 07:14:23 DEBUG "instance": "10.142.0.107:9100" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG [...] 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG 07:14:23 DEBUG "instance": "10.142.0.77:9100" 07:14:23 DEBUG }, 07:14:23 DEBUG "value": [ 07:14:23 DEBUG 1712816063, 07:14:23 DEBUG "0.0006666666666666524" 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG ] 07:14:23 DEBUG } 07:14:23 DEBUG } 07:14:35 INFO 07:14:35 INFO [loop_until]: kubectl --namespace=xlou top pods 07:14:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:14:36 INFO [loop_until]: OK (rc = 0) 07:14:36 DEBUG --- stdout --- 07:14:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 8810m 8674Mi am-5b5bd8567b-8h2w4 7604m 8859Mi am-5b5bd8567b-qlkw6 6805m 8528Mi ds-cts-0 7366m 13276Mi ds-cts-1 7482m 13066Mi ds-cts-2 7802m 13348Mi ds-idrepo-0 10281m 5504Mi ds-idrepo-1 303m 5480Mi ds-idrepo-2 323m 5784Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 4m 1011Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 2m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 7982m 7912Mi 07:14:36 DEBUG --- stderr --- 07:14:36 DEBUG 07:14:39 INFO 07:14:39 INFO [loop_until]: kubectl --namespace=xlou top node 07:14:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:14:40 INFO [loop_until]: OK (rc = 0) 07:14:40 DEBUG --- stdout --- 07:14:40 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 2731Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 63m 0% 2786Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7152m 23% 10216Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 11977m 40% 3655Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 8789m 29% 10296Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 124m 0% 3856Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 8616m 28% 10502Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 388m 2% 6474Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 10494m 66% 6207Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 7743m 48% 13948Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 7368m 46% 13950Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 7746m 48% 13732Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 419m 2% 6166Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 8234m 51% 11163Mi 19% 07:14:40 DEBUG --- stderr --- 07:14:40 DEBUG 07:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816123" 07:15:23 INFO [http_cmd]: http status code OK 07:15:23 DEBUG --- status code --- 07:15:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:15:23 DEBUG --- http response --- 07:15:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:15:23 DEBUG ----- output ----- 07:15:23 DEBUG { 07:15:23 DEBUG "status": "success", 07:15:23 DEBUG "data": { 07:15:23 DEBUG "resultType": "vector", 07:15:23 DEBUG "result": [ 07:15:23 DEBUG { 07:15:23 DEBUG "metric": { 07:15:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG [...] 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 1712816123, 07:15:23 DEBUG "1207.0333333333333" 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG } 07:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712816123" 07:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816123" 07:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816123" 07:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712816123" 07:15:23 INFO [http_cmd]: http status code OK 07:15:23 DEBUG --- status code --- 07:15:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:15:23 DEBUG --- http response --- 07:15:23 INFO [http_cmd]: http status code OK 07:15:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 07:15:23 DEBUG --- status code --- 07:15:23 DEBUG ----- output ----- 07:15:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:15:23 DEBUG { 07:15:23 DEBUG --- http response --- 07:15:23 DEBUG "status": "success", 07:15:23 DEBUG "data": { 07:15:23 DEBUG "resultType": "vector", 07:15:23 DEBUG "result": [ 07:15:23 DEBUG { 07:15:23 DEBUG "metric": { 07:15:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG [...] 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 1712816123, 07:15:23 DEBUG "4.245522505154718" 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG } 07:15:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:15:23 DEBUG ----- output ----- 07:15:23 DEBUG { 07:15:23 DEBUG "status": "success", 07:15:23 DEBUG "data": { 07:15:23 DEBUG "resultType": "vector", 07:15:23 DEBUG "result": [ 07:15:23 DEBUG { 07:15:23 DEBUG "metric": { 07:15:23 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG [...] 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 1712816123, 07:15:23 DEBUG "8.037272597332484" 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG } 07:15:23 ERROR [http_cmd]: ERROR 07:15:23 ERROR --- status code --- 07:15:23 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:15:23 ERROR --- http response --- 07:15:23 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:15:23 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:15:23 INFO [http_cmd]: http status code OK 07:15:23 DEBUG --- status code --- 07:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816123" 07:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816123" 07:15:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:15:23 DEBUG --- http response --- 07:15:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:15:23 DEBUG ----- output ----- 07:15:23 DEBUG { 07:15:23 DEBUG "status": "success", 07:15:23 DEBUG "data": { 07:15:23 DEBUG "resultType": "vector", 07:15:23 DEBUG "result": [ 07:15:23 DEBUG { 07:15:23 DEBUG "metric": { 07:15:23 DEBUG "instance": "10.142.0.107:9100" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG [...] 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG "instance": "10.142.0.77:9100" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 1712816123, 07:15:23 DEBUG "69352717.23333333" 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG } 07:15:23 INFO [http_cmd]: http status code OK 07:15:23 DEBUG --- status code --- 07:15:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:15:23 DEBUG --- http response --- 07:15:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:15:23 DEBUG ----- output ----- 07:15:23 DEBUG { 07:15:23 DEBUG "status": "success", 07:15:23 DEBUG "data": { 07:15:23 DEBUG "resultType": "vector", 07:15:23 DEBUG "result": [ 07:15:23 DEBUG { 07:15:23 DEBUG "metric": { 07:15:23 DEBUG "pod": "ds-idrepo-2" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG [...] 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG "pod": "ds-idrepo-0" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 1712816123, 07:15:23 DEBUG "0" 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG } 07:15:23 INFO [http_cmd]: http status code OK 07:15:23 DEBUG --- status code --- 07:15:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:15:23 DEBUG --- http response --- 07:15:23 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712816123" 07:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816123" 07:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816123" 07:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816123" 07:15:23 INFO [http_cmd]: http status code OK 07:15:23 INFO [http_cmd]: http status code OK 07:15:23 DEBUG --- status code --- 07:15:23 DEBUG --- status code --- 07:15:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:15:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:15:23 DEBUG --- http response --- 07:15:23 DEBUG --- http response --- 07:15:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:15:23 DEBUG ----- output ----- 07:15:23 DEBUG { 07:15:23 DEBUG "status": "success", 07:15:23 DEBUG "data": { 07:15:23 DEBUG "resultType": "vector", 07:15:23 DEBUG "result": [ 07:15:23 DEBUG { 07:15:23 DEBUG "metric": { 07:15:23 DEBUG "instance": "10.142.0.107:9100" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG [...] 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG "instance": "10.142.0.77:9100" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 1712816123, 07:15:23 DEBUG "0.0021999999999998466" 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG } 07:15:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:15:23 DEBUG ----- output ----- 07:15:23 DEBUG { 07:15:23 DEBUG "status": "success", 07:15:23 DEBUG "data": { 07:15:23 DEBUG "resultType": "vector", 07:15:23 DEBUG "result": [ 07:15:23 DEBUG { 07:15:23 DEBUG "metric": { 07:15:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG [...] 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 1712816123, 07:15:23 DEBUG "0.023" 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG } 07:15:23 INFO [http_cmd]: http status code OK 07:15:23 DEBUG --- status code --- 07:15:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:15:23 DEBUG --- http response --- 07:15:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:15:23 DEBUG ----- output ----- 07:15:23 DEBUG { 07:15:23 DEBUG "status": "success", 07:15:23 DEBUG "data": { 07:15:23 DEBUG "resultType": "vector", 07:15:23 DEBUG "result": [ 07:15:23 DEBUG { 07:15:23 DEBUG "metric": { 07:15:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG [...] 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 1712816123, 07:15:23 DEBUG "215353.5" 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG } 07:15:23 INFO [http_cmd]: http status code OK 07:15:23 DEBUG --- status code --- 07:15:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:15:23 DEBUG --- http response --- 07:15:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 07:15:23 DEBUG ----- output ----- 07:15:23 DEBUG { 07:15:23 DEBUG "status": "success", 07:15:23 DEBUG "data": { 07:15:23 DEBUG "resultType": "vector", 07:15:23 DEBUG "result": [ 07:15:23 DEBUG { 07:15:23 DEBUG "metric": { 07:15:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG [...] 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 1712816123, 07:15:23 DEBUG "286675438.248701" 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG } 07:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712816123" 07:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816123" 07:15:23 INFO [http_cmd]: http status code OK 07:15:23 DEBUG --- status code --- 07:15:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:15:23 DEBUG --- http response --- 07:15:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:15:23 DEBUG ----- output ----- 07:15:23 DEBUG { 07:15:23 DEBUG "status": "success", 07:15:23 DEBUG "data": { 07:15:23 DEBUG "resultType": "vector", 07:15:23 DEBUG "result": [ 07:15:23 DEBUG { 07:15:23 DEBUG "metric": { 07:15:23 DEBUG "pod": "ds-idrepo-2" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG [...] 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG "pod": "ds-cts-2" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 1712816123, 07:15:23 DEBUG "0" 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG } 07:15:23 INFO [http_cmd]: http status code OK 07:15:23 DEBUG --- status code --- 07:15:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:15:23 DEBUG --- http response --- 07:15:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:15:23 DEBUG ----- output ----- 07:15:23 DEBUG { 07:15:23 DEBUG "status": "success", 07:15:23 DEBUG "data": { 07:15:23 DEBUG "resultType": "vector", 07:15:23 DEBUG "result": [ 07:15:23 DEBUG { 07:15:23 DEBUG "metric": { 07:15:23 DEBUG "pod": "ds-idrepo-2" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG [...] 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG "pod": "ds-cts-2" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 1712816123, 07:15:23 DEBUG "15.4" 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG } 07:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816123" 07:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816123" 07:15:23 INFO [http_cmd]: http status code OK 07:15:23 DEBUG --- status code --- 07:15:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:15:23 DEBUG --- http response --- 07:15:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:15:23 DEBUG ----- output ----- 07:15:23 DEBUG { 07:15:23 DEBUG "status": "success", 07:15:23 DEBUG "data": { 07:15:23 DEBUG "resultType": "vector", 07:15:23 DEBUG "result": [ 07:15:23 DEBUG { 07:15:23 DEBUG "metric": { 07:15:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG [...] 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 1712816123, 07:15:23 DEBUG "0" 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG } 07:15:23 INFO [http_cmd]: http status code OK 07:15:23 DEBUG --- status code --- 07:15:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:15:23 DEBUG --- http response --- 07:15:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:15:23 DEBUG ----- output ----- 07:15:23 DEBUG { 07:15:23 DEBUG "status": "success", 07:15:23 DEBUG "data": { 07:15:23 DEBUG "resultType": "vector", 07:15:23 DEBUG "result": [ 07:15:23 DEBUG { 07:15:23 DEBUG "metric": { 07:15:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG [...] 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 1712816123, 07:15:23 DEBUG "0" 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG } 07:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816123" 07:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816123" 07:15:23 INFO [http_cmd]: http status code OK 07:15:23 DEBUG --- status code --- 07:15:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:15:23 DEBUG --- http response --- 07:15:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:15:23 DEBUG ----- output ----- 07:15:23 DEBUG { 07:15:23 DEBUG "status": "success", 07:15:23 DEBUG "data": { 07:15:23 DEBUG "resultType": "vector", 07:15:23 DEBUG "result": [ 07:15:23 DEBUG { 07:15:23 DEBUG "metric": { 07:15:23 DEBUG "instance": "10.142.0.107:9100" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG [...] 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG "instance": "10.142.0.77:9100" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 1712816123, 07:15:23 DEBUG "69354548.53333333" 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG } 07:15:23 INFO [http_cmd]: http status code OK 07:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816123" 07:15:23 DEBUG --- status code --- 07:15:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:15:23 DEBUG --- http response --- 07:15:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:15:23 DEBUG ----- output ----- 07:15:23 DEBUG { 07:15:23 DEBUG "status": "success", 07:15:23 DEBUG "data": { 07:15:23 DEBUG "resultType": "vector", 07:15:23 DEBUG "result": [ 07:15:23 DEBUG { 07:15:23 DEBUG "metric": { 07:15:23 DEBUG "instance": "10.142.0.107:9100" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG [...] 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG "instance": "10.142.0.77:9100" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 1712816123, 07:15:23 DEBUG "52996039680" 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG } 07:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816123" 07:15:23 INFO [http_cmd]: http status code OK 07:15:23 DEBUG --- status code --- 07:15:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:15:23 DEBUG --- http response --- 07:15:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:15:23 DEBUG ----- output ----- 07:15:23 DEBUG { 07:15:23 DEBUG "status": "success", 07:15:23 DEBUG "data": { 07:15:23 DEBUG "resultType": "vector", 07:15:23 DEBUG "result": [ 07:15:23 DEBUG { 07:15:23 DEBUG "metric": { 07:15:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG [...] 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 1712816123, 07:15:23 DEBUG "1164.5" 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG } 07:15:23 INFO [http_cmd]: http status code OK 07:15:23 DEBUG --- status code --- 07:15:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:15:23 DEBUG --- http response --- 07:15:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:15:23 DEBUG ----- output ----- 07:15:23 DEBUG { 07:15:23 DEBUG "status": "success", 07:15:23 DEBUG "data": { 07:15:23 DEBUG "resultType": "vector", 07:15:23 DEBUG "result": [ 07:15:23 DEBUG { 07:15:23 DEBUG "metric": { 07:15:23 DEBUG "pod": "ds-idrepo-2" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG [...] 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG "pod": "ds-idrepo-0" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 1712816123, 07:15:23 DEBUG "0" 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG } 07:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816123" 07:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816123" 07:15:23 INFO [http_cmd]: http status code OK 07:15:23 INFO [http_cmd]: http status code OK 07:15:23 DEBUG --- status code --- 07:15:23 DEBUG --- status code --- 07:15:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:15:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:15:23 DEBUG --- http response --- 07:15:23 DEBUG --- http response --- 07:15:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:15:23 DEBUG ----- output ----- 07:15:23 DEBUG { 07:15:23 DEBUG "status": "success", 07:15:23 DEBUG "data": { 07:15:23 DEBUG "resultType": "vector", 07:15:23 DEBUG "result": [ 07:15:23 DEBUG { 07:15:23 DEBUG "metric": { 07:15:23 DEBUG "pod": "ds-idrepo-2" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG [...] 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG "pod": "ds-idrepo-0" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 1712816123, 07:15:23 DEBUG "0" 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG } 07:15:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:15:23 DEBUG ----- output ----- 07:15:23 DEBUG { 07:15:23 DEBUG "status": "success", 07:15:23 DEBUG "data": { 07:15:23 DEBUG "resultType": "vector", 07:15:23 DEBUG "result": [ 07:15:23 DEBUG { 07:15:23 DEBUG "metric": { 07:15:23 DEBUG "pod": "ds-idrepo-2" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG [...] 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG "pod": "ds-cts-2" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 1712816123, 07:15:23 DEBUG "0.0005666666666666666" 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG } 07:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816123" 07:15:23 INFO [http_cmd]: http status code OK 07:15:23 DEBUG --- status code --- 07:15:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:15:23 DEBUG --- http response --- 07:15:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:15:23 DEBUG ----- output ----- 07:15:23 DEBUG { 07:15:23 DEBUG "status": "success", 07:15:23 DEBUG "data": { 07:15:23 DEBUG "resultType": "vector", 07:15:23 DEBUG "result": [ 07:15:23 DEBUG { 07:15:23 DEBUG "metric": { 07:15:23 DEBUG "pod": "ds-idrepo-2" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG [...] 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG "pod": "ds-cts-2" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 1712816123, 07:15:23 DEBUG "0" 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG } 07:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816123" 07:15:23 INFO [http_cmd]: http status code OK 07:15:23 DEBUG --- status code --- 07:15:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:15:23 DEBUG --- http response --- 07:15:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:15:23 DEBUG ----- output ----- 07:15:23 DEBUG { 07:15:23 DEBUG "status": "success", 07:15:23 DEBUG "data": { 07:15:23 DEBUG "resultType": "vector", 07:15:23 DEBUG "result": [ 07:15:23 DEBUG { 07:15:23 DEBUG "metric": { 07:15:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG [...] 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 1712816123, 07:15:23 DEBUG "1165.0666666666666" 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG } 07:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712816123" 07:15:23 INFO [http_cmd]: http status code OK 07:15:23 DEBUG --- status code --- 07:15:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:15:23 DEBUG --- http response --- 07:15:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:15:23 DEBUG ----- output ----- 07:15:23 DEBUG { 07:15:23 DEBUG "status": "success", 07:15:23 DEBUG "data": { 07:15:23 DEBUG "resultType": "vector", 07:15:23 DEBUG "result": [ 07:15:23 DEBUG { 07:15:23 DEBUG "metric": { 07:15:23 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG [...] 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 1712816123, 07:15:23 DEBUG "14025756672" 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG } 07:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816123" 07:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816123" 07:15:23 INFO [http_cmd]: http status code OK 07:15:23 DEBUG --- status code --- 07:15:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:15:23 DEBUG --- http response --- 07:15:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:15:23 DEBUG ----- output ----- 07:15:23 DEBUG { 07:15:23 DEBUG "status": "success", 07:15:23 DEBUG "data": { 07:15:23 DEBUG "resultType": "vector", 07:15:23 DEBUG "result": [ 07:15:23 DEBUG { 07:15:23 DEBUG "metric": { 07:15:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG [...] 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 1712816123, 07:15:23 DEBUG "1162.9666666666667" 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG } 07:15:23 INFO [http_cmd]: http status code OK 07:15:23 DEBUG --- status code --- 07:15:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:15:23 DEBUG --- http response --- 07:15:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:15:23 DEBUG ----- output ----- 07:15:23 DEBUG { 07:15:23 DEBUG "status": "success", 07:15:23 DEBUG "data": { 07:15:23 DEBUG "resultType": "vector", 07:15:23 DEBUG "result": [ 07:15:23 DEBUG { 07:15:23 DEBUG "metric": { 07:15:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG [...] 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 1712816123, 07:15:23 DEBUG "0" 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG } 07:15:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816123" 07:15:23 INFO [http_cmd]: http status code OK 07:15:23 DEBUG --- status code --- 07:15:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:15:23 DEBUG --- http response --- 07:15:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:15:23 DEBUG ----- output ----- 07:15:23 DEBUG { 07:15:23 DEBUG "status": "success", 07:15:23 DEBUG "data": { 07:15:23 DEBUG "resultType": "vector", 07:15:23 DEBUG "result": [ 07:15:23 DEBUG { 07:15:23 DEBUG "metric": { 07:15:23 DEBUG "instance": "10.142.0.107:9100" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG [...] 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG 07:15:23 DEBUG "instance": "10.142.0.77:9100" 07:15:23 DEBUG }, 07:15:23 DEBUG "value": [ 07:15:23 DEBUG 1712816123, 07:15:23 DEBUG "0.0006666666666667708" 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG ] 07:15:23 DEBUG } 07:15:23 DEBUG } 07:15:36 INFO 07:15:36 INFO [loop_until]: kubectl --namespace=xlou top pods 07:15:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:15:36 INFO [loop_until]: OK (rc = 0) 07:15:36 DEBUG --- stdout --- 07:15:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 7729m 8673Mi am-5b5bd8567b-8h2w4 7740m 8861Mi am-5b5bd8567b-qlkw6 6668m 8529Mi ds-cts-0 6829m 13400Mi ds-cts-1 7531m 13342Mi ds-cts-2 7313m 12466Mi ds-idrepo-0 9710m 5503Mi ds-idrepo-1 300m 5480Mi ds-idrepo-2 337m 5785Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 3m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 2m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 8073m 7808Mi 07:15:36 DEBUG --- stderr --- 07:15:36 DEBUG 07:15:40 INFO 07:15:40 INFO [loop_until]: kubectl --namespace=xlou top node 07:15:40 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:15:40 INFO [loop_until]: OK (rc = 0) 07:15:40 DEBUG --- stdout --- 07:15:40 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 2733Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 65m 0% 2787Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6797m 22% 10213Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 11396m 38% 3658Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 8133m 27% 10296Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 141m 0% 3837Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 8208m 27% 10512Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 417m 2% 6473Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 9672m 60% 6212Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 7424m 46% 13121Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 6811m 42% 13106Mi 22% gke-xlou-cdm-ds-32e4dcb1-j2hk 7687m 48% 13960Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 357m 2% 6165Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 8280m 52% 11052Mi 18% 07:15:40 DEBUG --- stderr --- 07:15:40 DEBUG 07:16:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816183" 07:16:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712816183" 07:16:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712816183" 07:16:23 INFO [http_cmd]: http status code OK 07:16:23 DEBUG --- status code --- 07:16:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:16:23 DEBUG --- http response --- 07:16:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:16:23 DEBUG ----- output ----- 07:16:23 DEBUG { 07:16:23 DEBUG "status": "success", 07:16:23 DEBUG "data": { 07:16:23 DEBUG "resultType": "vector", 07:16:23 DEBUG "result": [ 07:16:23 DEBUG { 07:16:23 DEBUG "metric": { 07:16:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG [...] 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 1712816183, 07:16:23 DEBUG "1178.0666666666666" 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG } 07:16:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816183" 07:16:23 ERROR [http_cmd]: ERROR 07:16:23 ERROR --- status code --- 07:16:23 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:16:23 ERROR --- http response --- 07:16:23 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:16:23 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:16:23 INFO [http_cmd]: http status code OK 07:16:23 DEBUG --- status code --- 07:16:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:16:23 DEBUG --- http response --- 07:16:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:16:23 DEBUG ----- output ----- 07:16:23 DEBUG { 07:16:23 DEBUG "status": "success", 07:16:23 DEBUG "data": { 07:16:23 DEBUG "resultType": "vector", 07:16:23 DEBUG "result": [ 07:16:23 DEBUG { 07:16:23 DEBUG "metric": { 07:16:23 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG [...] 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 1712816183, 07:16:23 DEBUG "8.02629506544805" 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG } 07:16:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816183" 07:16:23 INFO [http_cmd]: http status code OK 07:16:23 DEBUG --- status code --- 07:16:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:16:23 DEBUG --- http response --- 07:16:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:16:23 DEBUG ----- output ----- 07:16:23 DEBUG { 07:16:23 DEBUG "status": "success", 07:16:23 DEBUG "data": { 07:16:23 DEBUG "resultType": "vector", 07:16:23 DEBUG "result": [ 07:16:23 DEBUG { 07:16:23 DEBUG "metric": { 07:16:23 DEBUG "pod": "ds-idrepo-2" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG [...] 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG "pod": "ds-idrepo-0" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 1712816183, 07:16:23 DEBUG "0" 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG } 07:16:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816183" 07:16:23 INFO [http_cmd]: http status code OK 07:16:23 DEBUG --- status code --- 07:16:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:16:23 DEBUG --- http response --- 07:16:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:16:23 DEBUG ----- output ----- 07:16:23 DEBUG { 07:16:23 DEBUG "status": "success", 07:16:23 DEBUG "data": { 07:16:23 DEBUG "resultType": "vector", 07:16:23 DEBUG "result": [ 07:16:23 DEBUG { 07:16:23 DEBUG "metric": { 07:16:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG [...] 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG "pod": "ds-idrepo-1" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 1712816183, 07:16:23 DEBUG "0.12698517631800915" 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG } 07:16:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816183" 07:16:23 INFO [http_cmd]: http status code OK 07:16:23 DEBUG --- status code --- 07:16:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:16:23 DEBUG --- http response --- 07:16:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:16:23 DEBUG ----- output ----- 07:16:23 DEBUG { 07:16:23 DEBUG "status": "success", 07:16:23 DEBUG "data": { 07:16:23 DEBUG "resultType": "vector", 07:16:23 DEBUG "result": [ 07:16:23 DEBUG { 07:16:23 DEBUG "metric": { 07:16:23 DEBUG "instance": "10.142.0.107:9100" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG [...] 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG "instance": "10.142.0.77:9100" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 1712816183, 07:16:23 DEBUG "68923421.03333333" 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG } 07:16:23 INFO [http_cmd]: http status code OK 07:16:23 DEBUG --- status code --- 07:16:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:16:23 DEBUG --- http response --- 07:16:23 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:16:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816183" 07:16:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712816183" 07:16:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816183" 07:16:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816183" 07:16:23 INFO [http_cmd]: http status code OK 07:16:23 DEBUG --- status code --- 07:16:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:16:23 DEBUG --- http response --- 07:16:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:16:23 DEBUG ----- output ----- 07:16:23 DEBUG { 07:16:23 DEBUG "status": "success", 07:16:23 DEBUG "data": { 07:16:23 DEBUG "resultType": "vector", 07:16:23 DEBUG "result": [ 07:16:23 DEBUG { 07:16:23 DEBUG "metric": { 07:16:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG [...] 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 1712816183, 07:16:23 DEBUG "0.024" 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG } 07:16:23 INFO [http_cmd]: http status code OK 07:16:23 DEBUG --- status code --- 07:16:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:16:23 DEBUG --- http response --- 07:16:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:16:23 DEBUG ----- output ----- 07:16:23 DEBUG { 07:16:23 DEBUG "status": "success", 07:16:23 DEBUG "data": { 07:16:23 DEBUG "resultType": "vector", 07:16:23 DEBUG "result": [ 07:16:23 DEBUG { 07:16:23 DEBUG "metric": { 07:16:23 DEBUG "instance": "10.142.0.107:9100" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG [...] 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG "instance": "10.142.0.77:9100" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 1712816183, 07:16:23 DEBUG "0.0021333333333338087" 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG } 07:16:23 INFO [http_cmd]: http status code OK 07:16:23 DEBUG --- status code --- 07:16:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:16:23 DEBUG --- http response --- 07:16:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:16:23 DEBUG ----- output ----- 07:16:23 DEBUG { 07:16:23 DEBUG "status": "success", 07:16:23 DEBUG "data": { 07:16:23 DEBUG "resultType": "vector", 07:16:23 DEBUG "result": [ 07:16:23 DEBUG { 07:16:23 DEBUG "metric": { 07:16:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG [...] 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 1712816183, 07:16:23 DEBUG "210549.5" 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG } 07:16:23 INFO [http_cmd]: http status code OK 07:16:23 DEBUG --- status code --- 07:16:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:16:23 DEBUG --- http response --- 07:16:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:16:23 DEBUG ----- output ----- 07:16:23 DEBUG { 07:16:23 DEBUG "status": "success", 07:16:23 DEBUG "data": { 07:16:23 DEBUG "resultType": "vector", 07:16:23 DEBUG "result": [ 07:16:23 DEBUG { 07:16:23 DEBUG "metric": { 07:16:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG [...] 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG "pod": "ds-idrepo-1" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 1712816183, 07:16:23 DEBUG "28590.28323737303" 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG } 07:16:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816183" 07:16:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816183" 07:16:23 INFO [http_cmd]: http status code OK 07:16:23 DEBUG --- status code --- 07:16:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:16:23 DEBUG --- http response --- 07:16:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:16:23 DEBUG ----- output ----- 07:16:23 DEBUG { 07:16:23 DEBUG "status": "success", 07:16:23 DEBUG "data": { 07:16:23 DEBUG "resultType": "vector", 07:16:23 DEBUG "result": [ 07:16:23 DEBUG { 07:16:23 DEBUG "metric": { 07:16:23 DEBUG "pod": "ds-idrepo-2" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG [...] 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG "pod": "ds-cts-2" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 1712816183, 07:16:23 DEBUG "27.8" 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG } 07:16:23 INFO [http_cmd]: http status code OK 07:16:23 DEBUG --- status code --- 07:16:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:16:23 DEBUG --- http response --- 07:16:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:16:23 DEBUG ----- output ----- 07:16:23 DEBUG { 07:16:23 DEBUG "status": "success", 07:16:23 DEBUG "data": { 07:16:23 DEBUG "resultType": "vector", 07:16:23 DEBUG "result": [ 07:16:23 DEBUG { 07:16:23 DEBUG "metric": { 07:16:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG [...] 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG "pod": "ds-idrepo-1" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 1712816183, 07:16:23 DEBUG "0" 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG } 07:16:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712816183" 07:16:23 INFO [http_cmd]: http status code OK 07:16:23 DEBUG --- status code --- 07:16:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:16:23 DEBUG --- http response --- 07:16:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:16:23 DEBUG ----- output ----- 07:16:23 DEBUG { 07:16:23 DEBUG "status": "success", 07:16:23 DEBUG "data": { 07:16:23 DEBUG "resultType": "vector", 07:16:23 DEBUG "result": [ 07:16:23 DEBUG { 07:16:23 DEBUG "metric": { 07:16:23 DEBUG "pod": "ds-idrepo-2" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG [...] 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG "pod": "ds-cts-2" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 1712816183, 07:16:23 DEBUG "0" 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG } 07:16:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816183" 07:16:23 INFO [http_cmd]: http status code OK 07:16:23 DEBUG --- status code --- 07:16:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:16:23 DEBUG --- http response --- 07:16:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:16:23 DEBUG ----- output ----- 07:16:23 DEBUG { 07:16:23 DEBUG "status": "success", 07:16:23 DEBUG "data": { 07:16:23 DEBUG "resultType": "vector", 07:16:23 DEBUG "result": [ 07:16:23 DEBUG { 07:16:23 DEBUG "metric": { 07:16:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG [...] 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG "pod": "ds-idrepo-1" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 1712816183, 07:16:23 DEBUG "1518.068136497683" 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG } 07:16:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816183" 07:16:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816183" 07:16:23 INFO [http_cmd]: http status code OK 07:16:23 DEBUG --- status code --- 07:16:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:16:23 DEBUG --- http response --- 07:16:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:16:23 DEBUG ----- output ----- 07:16:23 DEBUG { 07:16:23 DEBUG "status": "success", 07:16:23 DEBUG "data": { 07:16:23 DEBUG "resultType": "vector", 07:16:23 DEBUG "result": [ 07:16:23 DEBUG { 07:16:23 DEBUG "metric": { 07:16:23 DEBUG "instance": "10.142.0.107:9100" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG [...] 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG "instance": "10.142.0.77:9100" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 1712816183, 07:16:23 DEBUG "68925409.26666667" 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG } 07:16:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816183" 07:16:23 INFO [http_cmd]: http status code OK 07:16:23 DEBUG --- status code --- 07:16:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:16:23 DEBUG --- http response --- 07:16:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:16:23 DEBUG ----- output ----- 07:16:23 DEBUG { 07:16:23 DEBUG "status": "success", 07:16:23 DEBUG "data": { 07:16:23 DEBUG "resultType": "vector", 07:16:23 DEBUG "result": [ 07:16:23 DEBUG { 07:16:23 DEBUG "metric": { 07:16:23 DEBUG "instance": "10.142.0.107:9100" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG [...] 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG "instance": "10.142.0.77:9100" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 1712816183, 07:16:23 DEBUG "52997368832" 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG } 07:16:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816183" 07:16:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816183" 07:16:23 INFO [http_cmd]: http status code OK 07:16:23 DEBUG --- status code --- 07:16:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:16:23 DEBUG --- http response --- 07:16:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:16:23 DEBUG ----- output ----- 07:16:23 DEBUG { 07:16:23 DEBUG "status": "success", 07:16:23 DEBUG "data": { 07:16:23 DEBUG "resultType": "vector", 07:16:23 DEBUG "result": [ 07:16:23 DEBUG { 07:16:23 DEBUG "metric": { 07:16:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG [...] 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 1712816183, 07:16:23 DEBUG "1112.5333333333333" 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG } 07:16:23 INFO [http_cmd]: http status code OK 07:16:23 DEBUG --- status code --- 07:16:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:16:23 DEBUG --- http response --- 07:16:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:16:23 DEBUG ----- output ----- 07:16:23 DEBUG { 07:16:23 DEBUG "status": "success", 07:16:23 DEBUG "data": { 07:16:23 DEBUG "resultType": "vector", 07:16:23 DEBUG "result": [ 07:16:23 DEBUG { 07:16:23 DEBUG "metric": { 07:16:23 DEBUG "pod": "ds-idrepo-2" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG [...] 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG "pod": "ds-idrepo-0" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 1712816183, 07:16:23 DEBUG "0" 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG } 07:16:23 INFO [http_cmd]: http status code OK 07:16:23 DEBUG --- status code --- 07:16:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:16:23 DEBUG --- http response --- 07:16:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:16:23 DEBUG ----- output ----- 07:16:23 DEBUG { 07:16:23 DEBUG "status": "success", 07:16:23 DEBUG "data": { 07:16:23 DEBUG "resultType": "vector", 07:16:23 DEBUG "result": [ 07:16:23 DEBUG { 07:16:23 DEBUG "metric": { 07:16:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG [...] 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 1712816183, 07:16:23 DEBUG "1112.6333333333332" 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG } 07:16:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816183" 07:16:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816183" 07:16:23 INFO [http_cmd]: http status code OK 07:16:23 INFO [http_cmd]: http status code OK 07:16:23 DEBUG --- status code --- 07:16:23 DEBUG --- status code --- 07:16:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:16:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:16:23 DEBUG --- http response --- 07:16:23 DEBUG --- http response --- 07:16:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:16:23 DEBUG ----- output ----- 07:16:23 DEBUG { 07:16:23 DEBUG "status": "success", 07:16:23 DEBUG "data": { 07:16:23 DEBUG "resultType": "vector", 07:16:23 DEBUG "result": [ 07:16:23 DEBUG { 07:16:23 DEBUG "metric": { 07:16:23 DEBUG "pod": "ds-idrepo-2" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG [...] 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG "pod": "ds-cts-2" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 1712816183, 07:16:23 DEBUG "0.00006252111111111111" 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG } 07:16:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:16:23 DEBUG ----- output ----- 07:16:23 DEBUG { 07:16:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816183" 07:16:23 DEBUG "status": "success", 07:16:23 DEBUG "data": { 07:16:23 DEBUG "resultType": "vector", 07:16:23 DEBUG "result": [ 07:16:23 DEBUG { 07:16:23 DEBUG "metric": { 07:16:23 DEBUG "pod": "ds-idrepo-2" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG [...] 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG "pod": "ds-idrepo-0" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 1712816183, 07:16:23 DEBUG "0" 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG } 07:16:23 INFO [http_cmd]: http status code OK 07:16:23 DEBUG --- status code --- 07:16:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:16:23 DEBUG --- http response --- 07:16:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:16:23 DEBUG ----- output ----- 07:16:23 DEBUG { 07:16:23 DEBUG "status": "success", 07:16:23 DEBUG "data": { 07:16:23 DEBUG "resultType": "vector", 07:16:23 DEBUG "result": [ 07:16:23 DEBUG { 07:16:23 DEBUG "metric": { 07:16:23 DEBUG "pod": "ds-idrepo-2" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG [...] 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG "pod": "ds-cts-2" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 1712816183, 07:16:23 DEBUG "0" 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG } 07:16:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712816183" 07:16:23 INFO [http_cmd]: http status code OK 07:16:23 DEBUG --- status code --- 07:16:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:16:23 DEBUG --- http response --- 07:16:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:16:23 DEBUG ----- output ----- 07:16:23 DEBUG { 07:16:23 DEBUG "status": "success", 07:16:23 DEBUG "data": { 07:16:23 DEBUG "resultType": "vector", 07:16:23 DEBUG "result": [ 07:16:23 DEBUG { 07:16:23 DEBUG "metric": { 07:16:23 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG [...] 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 1712816183, 07:16:23 DEBUG "13498109952" 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG } 07:16:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816183" 07:16:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816183" 07:16:23 INFO [http_cmd]: http status code OK 07:16:23 DEBUG --- status code --- 07:16:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:16:23 DEBUG --- http response --- 07:16:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:16:23 DEBUG ----- output ----- 07:16:23 DEBUG { 07:16:23 DEBUG "status": "success", 07:16:23 DEBUG "data": { 07:16:23 DEBUG "resultType": "vector", 07:16:23 DEBUG "result": [ 07:16:23 DEBUG { 07:16:23 DEBUG "metric": { 07:16:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG [...] 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 1712816183, 07:16:23 DEBUG "1112.0666666666666" 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG } 07:16:23 INFO [http_cmd]: http status code OK 07:16:23 DEBUG --- status code --- 07:16:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:16:23 DEBUG --- http response --- 07:16:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:16:23 DEBUG ----- output ----- 07:16:23 DEBUG { 07:16:23 DEBUG "status": "success", 07:16:23 DEBUG "data": { 07:16:23 DEBUG "resultType": "vector", 07:16:23 DEBUG "result": [ 07:16:23 DEBUG { 07:16:23 DEBUG "metric": { 07:16:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG [...] 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 1712816183, 07:16:23 DEBUG "0" 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG } 07:16:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816183" 07:16:23 INFO [http_cmd]: http status code OK 07:16:23 DEBUG --- status code --- 07:16:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:16:23 DEBUG --- http response --- 07:16:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:16:23 DEBUG ----- output ----- 07:16:23 DEBUG { 07:16:23 DEBUG "status": "success", 07:16:23 DEBUG "data": { 07:16:23 DEBUG "resultType": "vector", 07:16:23 DEBUG "result": [ 07:16:23 DEBUG { 07:16:23 DEBUG "metric": { 07:16:23 DEBUG "instance": "10.142.0.107:9100" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG [...] 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG 07:16:23 DEBUG "instance": "10.142.0.77:9100" 07:16:23 DEBUG }, 07:16:23 DEBUG "value": [ 07:16:23 DEBUG 1712816183, 07:16:23 DEBUG "0.000333333333333267" 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG ] 07:16:23 DEBUG } 07:16:23 DEBUG } 07:16:36 INFO 07:16:36 INFO [loop_until]: kubectl --namespace=xlou top pods 07:16:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:16:36 INFO [loop_until]: OK (rc = 0) 07:16:36 DEBUG --- stdout --- 07:16:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 7250m 8679Mi am-5b5bd8567b-8h2w4 6755m 8860Mi am-5b5bd8567b-qlkw6 6083m 8535Mi ds-cts-0 6639m 13018Mi ds-cts-1 6437m 12787Mi ds-cts-2 7017m 13029Mi ds-idrepo-0 9502m 5504Mi ds-idrepo-1 330m 5480Mi ds-idrepo-2 305m 5784Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 3m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 1m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 8258m 7868Mi 07:16:36 DEBUG --- stderr --- 07:16:36 DEBUG 07:16:40 INFO 07:16:40 INFO [loop_until]: kubectl --namespace=xlou top node 07:16:40 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:16:40 INFO [loop_until]: OK (rc = 0) 07:16:40 DEBUG --- stdout --- 07:16:40 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 2735Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 64m 0% 2790Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6552m 21% 10211Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 11034m 36% 3632Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 7988m 26% 10300Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 142m 0% 3866Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 8034m 26% 10504Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 357m 2% 6474Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 9386m 59% 6205Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 7126m 44% 13645Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 6558m 41% 13655Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 7074m 44% 13498Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 389m 2% 6168Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 8232m 51% 10918Mi 18% 07:16:40 DEBUG --- stderr --- 07:16:40 DEBUG 07:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816243" 07:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816243" 07:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712816243" 07:17:23 ERROR [http_cmd]: ERROR 07:17:23 INFO [http_cmd]: http status code OK 07:17:23 INFO [http_cmd]: http status code OK 07:17:23 ERROR --- status code --- 07:17:23 DEBUG --- status code --- 07:17:23 DEBUG --- status code --- 07:17:23 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:17:23 ERROR --- http response --- 07:17:23 DEBUG --- http response --- 07:17:23 DEBUG --- http response --- 07:17:23 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:17:23 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:17:23 DEBUG ----- output ----- 07:17:23 DEBUG ----- output ----- 07:17:23 DEBUG { 07:17:23 DEBUG { 07:17:23 DEBUG "status": "success", 07:17:23 DEBUG "status": "success", 07:17:23 DEBUG "data": { 07:17:23 DEBUG "data": { 07:17:23 DEBUG "resultType": "vector", 07:17:23 DEBUG "resultType": "vector", 07:17:23 DEBUG "result": [ 07:17:23 DEBUG "result": [ 07:17:23 DEBUG { 07:17:23 DEBUG { 07:17:23 DEBUG "metric": { 07:17:23 DEBUG "metric": { 07:17:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:17:23 DEBUG "pod": "ds-idrepo-2" 07:17:23 DEBUG }, 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG [...] 07:17:23 DEBUG [...] 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:17:23 DEBUG "pod": "ds-idrepo-0" 07:17:23 DEBUG }, 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 1712816243, 07:17:23 DEBUG 1712816243, 07:17:23 DEBUG "8.001606813180187" 07:17:23 DEBUG "0" 07:17:23 DEBUG ] 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG } 07:17:23 DEBUG ] 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG } 07:17:23 DEBUG } 07:17:23 DEBUG } 07:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816243" 07:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712816243" 07:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816243" 07:17:23 INFO [http_cmd]: http status code OK 07:17:23 DEBUG --- status code --- 07:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:17:23 DEBUG --- http response --- 07:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:17:23 DEBUG ----- output ----- 07:17:23 DEBUG { 07:17:23 DEBUG "status": "success", 07:17:23 DEBUG "data": { 07:17:23 DEBUG "resultType": "vector", 07:17:23 DEBUG "result": [ 07:17:23 DEBUG { 07:17:23 DEBUG "metric": { 07:17:23 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 INFO [http_cmd]: http status code OK 07:17:23 DEBUG 07:17:23 DEBUG --- status code --- 07:17:23 DEBUG [...] 07:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:17:23 DEBUG 07:17:23 DEBUG --- http response --- 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 1712816243, 07:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:17:23 DEBUG "8.036039707461763" 07:17:23 DEBUG ----- output ----- 07:17:23 DEBUG ] 07:17:23 DEBUG { 07:17:23 DEBUG } 07:17:23 DEBUG "status": "success", 07:17:23 DEBUG ] 07:17:23 DEBUG "data": { 07:17:23 DEBUG } 07:17:23 DEBUG "resultType": "vector", 07:17:23 DEBUG } 07:17:23 DEBUG "result": [ 07:17:23 DEBUG { 07:17:23 DEBUG "metric": { 07:17:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG [...] 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 1712816243, 07:17:23 DEBUG "1197.0333333333333" 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG } 07:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816243" 07:17:23 INFO [http_cmd]: http status code OK 07:17:23 DEBUG --- status code --- 07:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:17:23 DEBUG --- http response --- 07:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:17:23 DEBUG ----- output ----- 07:17:23 DEBUG { 07:17:23 DEBUG "status": "success", 07:17:23 DEBUG "data": { 07:17:23 DEBUG "resultType": "vector", 07:17:23 DEBUG "result": [ 07:17:23 DEBUG { 07:17:23 DEBUG "metric": { 07:17:23 DEBUG "instance": "10.142.0.107:9100" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG [...] 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG "instance": "10.142.0.77:9100" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 1712816243, 07:17:23 DEBUG "70458716.56666666" 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG } 07:17:23 INFO [http_cmd]: http status code OK 07:17:23 DEBUG --- status code --- 07:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:17:23 DEBUG --- http response --- 07:17:23 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712816243" 07:17:23 INFO [http_cmd]: http status code OK 07:17:23 DEBUG --- status code --- 07:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:17:23 DEBUG --- http response --- 07:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:17:23 DEBUG ----- output ----- 07:17:23 DEBUG { 07:17:23 DEBUG "status": "success", 07:17:23 DEBUG "data": { 07:17:23 DEBUG "resultType": "vector", 07:17:23 DEBUG "result": [ 07:17:23 DEBUG { 07:17:23 DEBUG "metric": { 07:17:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG [...] 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 1712816243, 07:17:23 DEBUG "0.024" 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG } 07:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816243" 07:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816243" 07:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816243" 07:17:23 INFO [http_cmd]: http status code OK 07:17:23 INFO [http_cmd]: http status code OK 07:17:23 DEBUG --- status code --- 07:17:23 DEBUG --- status code --- 07:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:17:23 DEBUG --- http response --- 07:17:23 DEBUG --- http response --- 07:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:17:23 DEBUG ----- output ----- 07:17:23 DEBUG ----- output ----- 07:17:23 DEBUG { 07:17:23 DEBUG { 07:17:23 DEBUG "status": "success", 07:17:23 DEBUG "status": "success", 07:17:23 DEBUG "data": { 07:17:23 DEBUG "data": { 07:17:23 DEBUG "resultType": "vector", 07:17:23 DEBUG "resultType": "vector", 07:17:23 DEBUG "result": [ 07:17:23 DEBUG "result": [ 07:17:23 DEBUG { 07:17:23 DEBUG { 07:17:23 DEBUG "metric": { 07:17:23 DEBUG "metric": { 07:17:23 DEBUG "instance": "10.142.0.107:9100" 07:17:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:17:23 DEBUG }, 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG [...] 07:17:23 DEBUG [...] 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG "instance": "10.142.0.77:9100" 07:17:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:17:23 DEBUG }, 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 1712816243, 07:17:23 DEBUG 1712816243, 07:17:23 DEBUG "0.0020333333333333314" 07:17:23 DEBUG "206265.5" 07:17:23 DEBUG ] 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG } 07:17:23 DEBUG ] 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG } 07:17:23 DEBUG } 07:17:23 DEBUG } 07:17:23 INFO [http_cmd]: http status code OK 07:17:23 DEBUG --- status code --- 07:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:17:23 DEBUG --- http response --- 07:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:17:23 DEBUG ----- output ----- 07:17:23 DEBUG { 07:17:23 DEBUG "status": "success", 07:17:23 DEBUG "data": { 07:17:23 DEBUG "resultType": "vector", 07:17:23 DEBUG "result": [ 07:17:23 DEBUG { 07:17:23 DEBUG "metric": { 07:17:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG [...] 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 1712816243, 07:17:23 DEBUG "2689581.6069342843" 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG } 07:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816243" 07:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816243" 07:17:23 INFO [http_cmd]: http status code OK 07:17:23 DEBUG --- status code --- 07:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:17:23 DEBUG --- http response --- 07:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:17:23 DEBUG ----- output ----- 07:17:23 DEBUG { 07:17:23 DEBUG "status": "success", 07:17:23 DEBUG "data": { 07:17:23 DEBUG "resultType": "vector", 07:17:23 DEBUG "result": [ 07:17:23 DEBUG { 07:17:23 DEBUG "metric": { 07:17:23 DEBUG "pod": "ds-idrepo-2" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG [...] 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG "pod": "ds-cts-2" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 1712816243, 07:17:23 DEBUG "28.333333333333332" 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG } 07:17:23 INFO [http_cmd]: http status code OK 07:17:23 DEBUG --- status code --- 07:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:17:23 DEBUG --- http response --- 07:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:17:23 DEBUG ----- output ----- 07:17:23 DEBUG { 07:17:23 DEBUG "status": "success", 07:17:23 DEBUG "data": { 07:17:23 DEBUG "resultType": "vector", 07:17:23 DEBUG "result": [ 07:17:23 DEBUG { 07:17:23 DEBUG "metric": { 07:17:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG [...] 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 1712816243, 07:17:23 DEBUG "0" 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG } 07:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816243" 07:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712816243" 07:17:23 INFO [http_cmd]: http status code OK 07:17:23 DEBUG --- status code --- 07:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:17:23 DEBUG --- http response --- 07:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:17:23 DEBUG ----- output ----- 07:17:23 DEBUG { 07:17:23 DEBUG "status": "success", 07:17:23 DEBUG "data": { 07:17:23 DEBUG "resultType": "vector", 07:17:23 DEBUG "result": [ 07:17:23 DEBUG { 07:17:23 DEBUG "metric": { 07:17:23 DEBUG "pod": "ds-idrepo-2" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG [...] 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG "pod": "ds-cts-2" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 1712816243, 07:17:23 DEBUG "0" 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG } 07:17:23 INFO [http_cmd]: http status code OK 07:17:23 DEBUG --- status code --- 07:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:17:23 DEBUG --- http response --- 07:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:17:23 DEBUG ----- output ----- 07:17:23 DEBUG { 07:17:23 DEBUG "status": "success", 07:17:23 DEBUG "data": { 07:17:23 DEBUG "resultType": "vector", 07:17:23 DEBUG "result": [ 07:17:23 DEBUG { 07:17:23 DEBUG "metric": { 07:17:23 DEBUG "pod": "ds-idrepo-2" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG [...] 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG "pod": "ds-idrepo-0" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 1712816243, 07:17:23 DEBUG "0" 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG } 07:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816243" 07:17:23 INFO [http_cmd]: http status code OK 07:17:23 DEBUG --- status code --- 07:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:17:23 DEBUG --- http response --- 07:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:17:23 DEBUG ----- output ----- 07:17:23 DEBUG { 07:17:23 DEBUG "status": "success", 07:17:23 DEBUG "data": { 07:17:23 DEBUG "resultType": "vector", 07:17:23 DEBUG "result": [ 07:17:23 DEBUG { 07:17:23 DEBUG "metric": { 07:17:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG [...] 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 1712816243, 07:17:23 DEBUG "0" 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG } 07:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816243" 07:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816243" 07:17:23 INFO [http_cmd]: http status code OK 07:17:23 DEBUG --- status code --- 07:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:17:23 DEBUG --- http response --- 07:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:17:23 DEBUG ----- output ----- 07:17:23 DEBUG { 07:17:23 DEBUG "status": "success", 07:17:23 DEBUG "data": { 07:17:23 DEBUG "resultType": "vector", 07:17:23 DEBUG "result": [ 07:17:23 DEBUG { 07:17:23 DEBUG "metric": { 07:17:23 DEBUG "instance": "10.142.0.107:9100" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG [...] 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG "instance": "10.142.0.77:9100" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 1712816243, 07:17:23 DEBUG "70460362.1" 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG } 07:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816243" 07:17:23 INFO [http_cmd]: http status code OK 07:17:23 DEBUG --- status code --- 07:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:17:23 DEBUG --- http response --- 07:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:17:23 DEBUG ----- output ----- 07:17:23 DEBUG { 07:17:23 DEBUG "status": "success", 07:17:23 DEBUG "data": { 07:17:23 DEBUG "resultType": "vector", 07:17:23 DEBUG "result": [ 07:17:23 DEBUG { 07:17:23 DEBUG "metric": { 07:17:23 DEBUG "instance": "10.142.0.107:9100" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG [...] 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG "instance": "10.142.0.77:9100" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 1712816243, 07:17:23 DEBUG "52998792192" 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG } 07:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816243" 07:17:23 INFO [http_cmd]: http status code OK 07:17:23 DEBUG --- status code --- 07:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:17:23 DEBUG --- http response --- 07:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:17:23 DEBUG ----- output ----- 07:17:23 DEBUG { 07:17:23 DEBUG "status": "success", 07:17:23 DEBUG "data": { 07:17:23 DEBUG "resultType": "vector", 07:17:23 DEBUG "result": [ 07:17:23 DEBUG { 07:17:23 DEBUG "metric": { 07:17:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG [...] 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 1712816243, 07:17:23 DEBUG "1167.5333333333333" 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG } 07:17:23 INFO [http_cmd]: http status code OK 07:17:23 DEBUG --- status code --- 07:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:17:23 DEBUG --- http response --- 07:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:17:23 DEBUG ----- output ----- 07:17:23 DEBUG { 07:17:23 DEBUG "status": "success", 07:17:23 DEBUG "data": { 07:17:23 DEBUG "resultType": "vector", 07:17:23 DEBUG "result": [ 07:17:23 DEBUG { 07:17:23 DEBUG "metric": { 07:17:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG [...] 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 1712816243, 07:17:23 DEBUG "1166.5666666666666" 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG } 07:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816243" 07:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816243" 07:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816243" 07:17:23 INFO [http_cmd]: http status code OK 07:17:23 DEBUG --- status code --- 07:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:17:23 DEBUG --- http response --- 07:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:17:23 DEBUG ----- output ----- 07:17:23 DEBUG { 07:17:23 DEBUG "status": "success", 07:17:23 DEBUG "data": { 07:17:23 DEBUG "resultType": "vector", 07:17:23 DEBUG "result": [ 07:17:23 DEBUG { 07:17:23 DEBUG "metric": { 07:17:23 DEBUG "pod": "ds-idrepo-2" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG [...] 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG "pod": "ds-cts-2" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 1712816243, 07:17:23 DEBUG "0.00016666666666666663" 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG } 07:17:23 INFO [http_cmd]: http status code OK 07:17:23 DEBUG --- status code --- 07:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:17:23 DEBUG --- http response --- 07:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:17:23 DEBUG ----- output ----- 07:17:23 DEBUG { 07:17:23 DEBUG "status": "success", 07:17:23 DEBUG "data": { 07:17:23 DEBUG "resultType": "vector", 07:17:23 DEBUG "result": [ 07:17:23 DEBUG { 07:17:23 DEBUG "metric": { 07:17:23 DEBUG "pod": "ds-idrepo-2" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG [...] 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG "pod": "ds-cts-2" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 1712816243, 07:17:23 DEBUG "0" 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG } 07:17:23 INFO [http_cmd]: http status code OK 07:17:23 DEBUG --- status code --- 07:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:17:23 DEBUG --- http response --- 07:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:17:23 DEBUG ----- output ----- 07:17:23 DEBUG { 07:17:23 DEBUG "status": "success", 07:17:23 DEBUG "data": { 07:17:23 DEBUG "resultType": "vector", 07:17:23 DEBUG "result": [ 07:17:23 DEBUG { 07:17:23 DEBUG "metric": { 07:17:23 DEBUG "pod": "ds-idrepo-2" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG [...] 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG "pod": "ds-idrepo-0" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 1712816243, 07:17:23 DEBUG "0" 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG } 07:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712816243" 07:17:23 INFO [http_cmd]: http status code OK 07:17:23 DEBUG --- status code --- 07:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:17:23 DEBUG --- http response --- 07:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:17:23 DEBUG ----- output ----- 07:17:23 DEBUG { 07:17:23 DEBUG "status": "success", 07:17:23 DEBUG "data": { 07:17:23 DEBUG "resultType": "vector", 07:17:23 DEBUG "result": [ 07:17:23 DEBUG { 07:17:23 DEBUG "metric": { 07:17:23 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG [...] 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 1712816243, 07:17:23 DEBUG "13976047616" 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG } 07:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816243" 07:17:23 INFO [http_cmd]: http status code OK 07:17:23 DEBUG --- status code --- 07:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:17:23 DEBUG --- http response --- 07:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:17:23 DEBUG ----- output ----- 07:17:23 DEBUG { 07:17:23 DEBUG "status": "success", 07:17:23 DEBUG "data": { 07:17:23 DEBUG "resultType": "vector", 07:17:23 DEBUG "result": [ 07:17:23 DEBUG { 07:17:23 DEBUG "metric": { 07:17:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG [...] 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 1712816243, 07:17:23 DEBUG "1174.1" 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG } 07:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816243" 07:17:23 INFO [http_cmd]: http status code OK 07:17:23 DEBUG --- status code --- 07:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:17:23 DEBUG --- http response --- 07:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:17:23 DEBUG ----- output ----- 07:17:23 DEBUG { 07:17:23 DEBUG "status": "success", 07:17:23 DEBUG "data": { 07:17:23 DEBUG "resultType": "vector", 07:17:23 DEBUG "result": [ 07:17:23 DEBUG { 07:17:23 DEBUG "metric": { 07:17:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG [...] 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 1712816243, 07:17:23 DEBUG "0" 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG } 07:17:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816243" 07:17:23 INFO [http_cmd]: http status code OK 07:17:23 DEBUG --- status code --- 07:17:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:17:23 DEBUG --- http response --- 07:17:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:17:23 DEBUG ----- output ----- 07:17:23 DEBUG { 07:17:23 DEBUG "status": "success", 07:17:23 DEBUG "data": { 07:17:23 DEBUG "resultType": "vector", 07:17:23 DEBUG "result": [ 07:17:23 DEBUG { 07:17:23 DEBUG "metric": { 07:17:23 DEBUG "instance": "10.142.0.107:9100" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG [...] 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG 07:17:23 DEBUG "instance": "10.142.0.77:9100" 07:17:23 DEBUG }, 07:17:23 DEBUG "value": [ 07:17:23 DEBUG 1712816243, 07:17:23 DEBUG "0" 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG ] 07:17:23 DEBUG } 07:17:23 DEBUG } 07:17:36 INFO 07:17:36 INFO [loop_until]: kubectl --namespace=xlou top pods 07:17:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:17:36 INFO [loop_until]: OK (rc = 0) 07:17:36 DEBUG --- stdout --- 07:17:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 8240m 8683Mi am-5b5bd8567b-8h2w4 7470m 8859Mi am-5b5bd8567b-qlkw6 6416m 8527Mi ds-cts-0 6917m 13098Mi ds-cts-1 7417m 13006Mi ds-cts-2 7638m 12987Mi ds-idrepo-0 9824m 5503Mi ds-idrepo-1 318m 5480Mi ds-idrepo-2 314m 5784Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 4m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 4m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 7980m 8059Mi 07:17:36 DEBUG --- stderr --- 07:17:36 DEBUG 07:17:40 INFO 07:17:40 INFO [loop_until]: kubectl --namespace=xlou top node 07:17:40 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:17:40 INFO [loop_until]: OK (rc = 0) 07:17:40 DEBUG --- stdout --- 07:17:40 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 2716Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 63m 0% 2787Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6828m 22% 10206Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10951m 36% 3644Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 8272m 27% 10294Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 142m 0% 3868Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 8376m 28% 10507Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 379m 2% 6475Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 9998m 62% 6210Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 7593m 47% 13723Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 7251m 45% 13765Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 7542m 47% 13581Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 370m 2% 6167Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 8206m 51% 11124Mi 18% 07:17:40 DEBUG --- stderr --- 07:17:40 DEBUG 07:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816303" 07:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816303" 07:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712816303" 07:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816303" 07:18:23 INFO [http_cmd]: http status code OK 07:18:23 DEBUG --- status code --- 07:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:18:23 DEBUG --- http response --- 07:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:18:23 DEBUG ----- output ----- 07:18:23 DEBUG { 07:18:23 DEBUG "status": "success", 07:18:23 DEBUG "data": { 07:18:23 DEBUG "resultType": "vector", 07:18:23 DEBUG "result": [ 07:18:23 DEBUG { 07:18:23 DEBUG "metric": { 07:18:23 DEBUG "pod": "ds-idrepo-2" 07:18:23 DEBUG }, 07:18:23 DEBUG[0m "value": [ 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG [...] 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG "pod": "ds-idrepo-0" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 1712816303, 07:18:23 DEBUG "0" 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG } 07:18:23 ERROR [http_cmd]: ERROR 07:18:23 ERROR --- status code --- 07:18:23 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:18:23 ERROR --- http response --- 07:18:23 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:18:23 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:18:23 INFO [http_cmd]: http status code OK 07:18:23 DEBUG --- status code --- 07:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:18:23 DEBUG --- http response --- 07:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 07:18:23 DEBUG ----- output ----- 07:18:23 DEBUG { 07:18:23 DEBUG "status": "success", 07:18:23 DEBUG "data": { 07:18:23 DEBUG "resultType": "vector", 07:18:23 DEBUG "result": [ 07:18:23 DEBUG { 07:18:23 DEBUG "metric": { 07:18:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG [...] 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 1712816303, 07:18:23 DEBUG "5.39727488073288" 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG } 07:18:23 INFO [http_cmd]: http status code OK 07:18:23 DEBUG --- status code --- 07:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:18:23 DEBUG --- http response --- 07:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:18:23 DEBUG ----- output ----- 07:18:23 DEBUG { 07:18:23 DEBUG "status": "success", 07:18:23 DEBUG "data": { 07:18:23 DEBUG "resultType": "vector", 07:18:23 DEBUG "result": [ 07:18:23 DEBUG { 07:18:23 DEBUG "metric": { 07:18:23 DEBUG "instance": "10.142.0.107:9100" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG [...] 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG "instance": "10.142.0.77:9100" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 1712816303, 07:18:23 DEBUG "48323034.8" 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG } 07:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816303" 07:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712816303" 07:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816303" 07:18:23 INFO [http_cmd]: http status code OK 07:18:23 DEBUG --- status code --- 07:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:18:23 DEBUG --- http response --- 07:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:18:23 DEBUG ----- output ----- 07:18:23 DEBUG { 07:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712816303" 07:18:23 DEBUG "status": "success", 07:18:23 DEBUG "data": { 07:18:23 DEBUG "resultType": "vector", 07:18:23 DEBUG "result": [ 07:18:23 DEBUG { 07:18:23 DEBUG "metric": { 07:18:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG [...] 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 1712816303, 07:18:23 DEBUG "1166.0666666666666" 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG } 07:18:23 INFO [http_cmd]: http status code OK 07:18:23 DEBUG --- status code --- 07:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:18:23 DEBUG --- http response --- 07:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:18:23 DEBUG ----- output ----- 07:18:23 DEBUG { 07:18:23 DEBUG "status": "success", 07:18:23 DEBUG "data": { 07:18:23 DEBUG "resultType": "vector", 07:18:23 DEBUG "result": [ 07:18:23 DEBUG { 07:18:23 DEBUG "metric": { 07:18:23 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG [...] 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 1712816303, 07:18:23 DEBUG "8.033992727635441" 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG } 07:18:23 INFO [http_cmd]: http status code OK 07:18:23 DEBUG --- status code --- 07:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:18:23 DEBUG --- http response --- 07:18:23 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816303" 07:18:23 INFO [http_cmd]: http status code OK 07:18:23 DEBUG --- status code --- 07:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:18:23 DEBUG --- http response --- 07:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:18:23 DEBUG ----- output ----- 07:18:23 DEBUG { 07:18:23 DEBUG "status": "success", 07:18:23 DEBUG "data": { 07:18:23 DEBUG "resultType": "vector", 07:18:23 DEBUG "result": [ 07:18:23 DEBUG { 07:18:23 DEBUG "metric": { 07:18:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG [...] 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 1712816303, 07:18:23 DEBUG "0.024" 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG } 07:18:23 INFO [http_cmd]: http status code OK 07:18:23 DEBUG --- status code --- 07:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:18:23 DEBUG --- http response --- 07:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:18:23 DEBUG ----- output ----- 07:18:23 DEBUG { 07:18:23 DEBUG "status": "success", 07:18:23 DEBUG "data": { 07:18:23 DEBUG "resultType": "vector", 07:18:23 DEBUG "result": [ 07:18:23 DEBUG { 07:18:23 DEBUG "metric": { 07:18:23 DEBUG "instance": "10.142.0.107:9100" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG [...] 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG "instance": "10.142.0.77:9100" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 1712816303, 07:18:23 DEBUG "0.0022666666666668316" 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG } 07:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816303" 07:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816303" 07:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816303" 07:18:23 INFO [http_cmd]: http status code OK 07:18:23 DEBUG --- status code --- 07:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:18:23 DEBUG --- http response --- 07:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:18:23 DEBUG ----- output ----- 07:18:23 DEBUG { 07:18:23 DEBUG "status": "success", 07:18:23 DEBUG "data": { 07:18:23 DEBUG "resultType": "vector", 07:18:23 DEBUG "result": [ 07:18:23 DEBUG { 07:18:23 DEBUG "metric": { 07:18:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG [...] 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 1712816303, 07:18:23 DEBUG "198338.5" 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG } 07:18:23 INFO [http_cmd]: http status code OK 07:18:23 DEBUG --- status code --- 07:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:18:23 DEBUG --- http response --- 07:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 07:18:23 DEBUG ----- output ----- 07:18:23 DEBUG { 07:18:23 DEBUG "status": "success", 07:18:23 DEBUG "data": { 07:18:23 DEBUG "resultType": "vector", 07:18:23 DEBUG "result": [ 07:18:23 DEBUG { 07:18:23 DEBUG "metric": { 07:18:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG [...] 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 1712816303, 07:18:23 DEBUG "1498266.9814186303" 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG } 07:18:23 INFO [http_cmd]: http status code OK 07:18:23 DEBUG --- status code --- 07:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:18:23 DEBUG --- http response --- 07:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 07:18:23 DEBUG ----- output ----- 07:18:23 DEBUG { 07:18:23 DEBUG "status": "success", 07:18:23 DEBUG "data": { 07:18:23 DEBUG "resultType": "vector", 07:18:23 DEBUG "result": [ 07:18:23 DEBUG { 07:18:23 DEBUG "metric": { 07:18:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG [...] 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 1712816303, 07:18:23 DEBUG "112.62624832132829" 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG } 07:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816303" 07:18:23 INFO [http_cmd]: http status code OK 07:18:23 DEBUG --- status code --- 07:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:18:23 DEBUG --- http response --- 07:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:18:23 DEBUG ----- output ----- 07:18:23 DEBUG { 07:18:23 DEBUG "status": "success", 07:18:23 DEBUG "data": { 07:18:23 DEBUG "resultType": "vector", 07:18:23 DEBUG "result": [ 07:18:23 DEBUG { 07:18:23 DEBUG "metric": { 07:18:23 DEBUG "pod": "ds-idrepo-2" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG [...] 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG "pod": "ds-cts-2" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 1712816303, 07:18:23 DEBUG "29.366666666666667" 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG } 07:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712816303" 07:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816303" 07:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816303" 07:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816303" 07:18:23 INFO [http_cmd]: http status code OK 07:18:23 DEBUG --- status code --- 07:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:18:23 DEBUG --- http response --- 07:18:23 INFO [http_cmd]: http status code OK 07:18:23 INFO [http_cmd]: http status code OK 07:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:18:23 DEBUG --- status code --- 07:18:23 DEBUG --- status code --- 07:18:23 DEBUG ----- output ----- 07:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:18:23 DEBUG { 07:18:23 DEBUG --- http response --- 07:18:23 DEBUG --- http response --- 07:18:23 DEBUG "status": "success", 07:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:18:23 DEBUG "data": { 07:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:18:23 DEBUG ----- output ----- 07:18:23 DEBUG "resultType": "vector", 07:18:23 DEBUG ----- output ----- 07:18:23 DEBUG { 07:18:23 DEBUG "result": [ 07:18:23 DEBUG { 07:18:23 DEBUG "status": "success", 07:18:23 DEBUG { 07:18:23 DEBUG "status": "success", 07:18:23 DEBUG "data": { 07:18:23 DEBUG "metric": { 07:18:23 DEBUG "data": { 07:18:23 DEBUG "resultType": "vector", 07:18:23 DEBUG "pod": "ds-idrepo-2" 07:18:23 DEBUG "resultType": "vector", 07:18:23 DEBUG "result": [ 07:18:23 DEBUG }, 07:18:23 DEBUG "result": [ 07:18:23 DEBUG { 07:18:23 DEBUG "value": [ 07:18:23 DEBUG { 07:18:23 DEBUG "metric": { 07:18:23 DEBUG 07:18:23 DEBUG "metric": { 07:18:23 DEBUG "instance": "10.142.0.107:9100" 07:18:23 DEBUG 07:18:23 DEBUG "pod": "ds-idrepo-2" 07:18:23 DEBUG }, 07:18:23 DEBUG 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG [...] 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG [...] 07:18:23 DEBUG "pod": "ds-idrepo-0" 07:18:23 DEBUG [...] 07:18:23 DEBUG 07:18:23 DEBUG }, 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 1712816303, 07:18:23 DEBUG 07:18:23 DEBUG "instance": "10.142.0.77:9100" 07:18:23 DEBUG "0" 07:18:23 DEBUG "pod": "ds-cts-2" 07:18:23 DEBUG }, 07:18:23 DEBUG ] 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG } 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 1712816303, 07:18:23 DEBUG ] 07:18:23 DEBUG 1712816303, 07:18:23 DEBUG "53000111104" 07:18:23 DEBUG } 07:18:23 DEBUG "0" 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG } 07:18:23 DEBUG ] 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG } 07:18:23 DEBUG } 07:18:23 DEBUG } 07:18:23 INFO [http_cmd]: http status code OK 07:18:23 DEBUG --- status code --- 07:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:18:23 DEBUG --- http response --- 07:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 07:18:23 DEBUG ----- output ----- 07:18:23 DEBUG { 07:18:23 DEBUG "status": "success", 07:18:23 DEBUG "data": { 07:18:23 DEBUG "resultType": "vector", 07:18:23 DEBUG "result": [ 07:18:23 DEBUG { 07:18:23 DEBUG "metric": { 07:18:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG [...] 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 1712816303, 07:18:23 DEBUG "0" 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG } 07:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816303" 07:18:23 INFO [http_cmd]: http status code OK 07:18:23 DEBUG --- status code --- 07:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:18:23 DEBUG --- http response --- 07:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816303" 07:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:18:23 DEBUG ----- output ----- 07:18:23 DEBUG { 07:18:23 DEBUG "status": "success", 07:18:23 DEBUG "data": { 07:18:23 DEBUG "resultType": "vector", 07:18:23 DEBUG "result": [ 07:18:23 DEBUG { 07:18:23 DEBUG "metric": { 07:18:23 DEBUG "pod": "ds-idrepo-2" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG [...] 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG "pod": "ds-idrepo-0" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 1712816303, 07:18:23 DEBUG "0" 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG } 07:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816303" 07:18:23 INFO [http_cmd]: http status code OK 07:18:23 DEBUG --- status code --- 07:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:18:23 DEBUG --- http response --- 07:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:18:23 DEBUG ----- output ----- 07:18:23 DEBUG { 07:18:23 DEBUG "status": "success", 07:18:23 DEBUG "data": { 07:18:23 DEBUG "resultType": "vector", 07:18:23 DEBUG "result": [ 07:18:23 DEBUG { 07:18:23 DEBUG "metric": { 07:18:23 DEBUG "instance": "10.142.0.107:9100" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG [...] 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG "instance": "10.142.0.77:9100" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 1712816303, 07:18:23 DEBUG "48324623.83333333" 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG } 07:18:23 INFO [http_cmd]: http status code OK 07:18:23 DEBUG --- status code --- 07:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:18:23 DEBUG --- http response --- 07:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:18:23 DEBUG ----- output ----- 07:18:23 DEBUG { 07:18:23 DEBUG "status": "success", 07:18:23 DEBUG "data": { 07:18:23 DEBUG "resultType": "vector", 07:18:23 DEBUG "result": [ 07:18:23 DEBUG { 07:18:23 DEBUG "metric": { 07:18:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG [...] 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 1712816303, 07:18:23 DEBUG "886.9666666666667" 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG } 07:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816303" 07:18:23 INFO [http_cmd]: http status code OK 07:18:23 DEBUG --- status code --- 07:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:18:23 DEBUG --- http response --- 07:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:18:23 DEBUG ----- output ----- 07:18:23 DEBUG { 07:18:23 DEBUG "status": "success", 07:18:23 DEBUG "data": { 07:18:23 DEBUG "resultType": "vector", 07:18:23 DEBUG "result": [ 07:18:23 DEBUG { 07:18:23 DEBUG "metric": { 07:18:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG [...] 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 1712816303, 07:18:23 DEBUG "887" 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG } 07:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816303" 07:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816303" 07:18:23 INFO [http_cmd]: http status code OK 07:18:23 DEBUG --- status code --- 07:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:18:23 DEBUG --- http response --- 07:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:18:23 DEBUG ----- output ----- 07:18:23 DEBUG { 07:18:23 DEBUG "status": "success", 07:18:23 DEBUG "data": { 07:18:23 DEBUG "resultType": "vector", 07:18:23 DEBUG "result": [ 07:18:23 DEBUG { 07:18:23 DEBUG "metric": { 07:18:23 DEBUG "pod": "ds-idrepo-2" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG [...] 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG "pod": "ds-cts-2" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 1712816303, 07:18:23 DEBUG "0.000031260555555555555" 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG } 07:18:23 INFO [http_cmd]: http status code OK 07:18:23 DEBUG --- status code --- 07:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:18:23 DEBUG --- http response --- 07:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:18:23 DEBUG ----- output ----- 07:18:23 DEBUG { 07:18:23 DEBUG "status": "success", 07:18:23 DEBUG "data": { 07:18:23 DEBUG "resultType": "vector", 07:18:23 DEBUG "result": [ 07:18:23 DEBUG { 07:18:23 DEBUG "metric": { 07:18:23 DEBUG "pod": "ds-idrepo-2" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG [...] 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG "pod": "ds-cts-2" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 1712816303, 07:18:23 DEBUG "0" 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG } 07:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816303" 07:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816303" 07:18:23 INFO [http_cmd]: http status code OK 07:18:23 DEBUG --- status code --- 07:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:18:23 DEBUG --- http response --- 07:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:18:23 DEBUG ----- output ----- 07:18:23 DEBUG { 07:18:23 DEBUG "status": "success", 07:18:23 DEBUG "data": { 07:18:23 DEBUG "resultType": "vector", 07:18:23 DEBUG "result": [ 07:18:23 DEBUG { 07:18:23 DEBUG "metric": { 07:18:23 DEBUG "instance": "10.142.0.107:9100" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG [...] 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG "instance": "10.142.0.77:9100" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 1712816303, 07:18:23 DEBUG "0.0006666666666662973" 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG } 07:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712816303" 07:18:23 INFO [http_cmd]: http status code OK 07:18:23 DEBUG --- status code --- 07:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:18:23 DEBUG --- http response --- 07:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:18:23 DEBUG ----- output ----- 07:18:23 DEBUG { 07:18:23 DEBUG "status": "success", 07:18:23 DEBUG "data": { 07:18:23 DEBUG "resultType": "vector", 07:18:23 DEBUG "result": [ 07:18:23 DEBUG { 07:18:23 DEBUG "metric": { 07:18:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG [...] 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 1712816303, 07:18:23 DEBUG "864.8" 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG } 07:18:23 INFO [http_cmd]: http status code OK 07:18:23 DEBUG --- status code --- 07:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:18:23 DEBUG --- http response --- 07:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:18:23 DEBUG ----- output ----- 07:18:23 DEBUG { 07:18:23 DEBUG "status": "success", 07:18:23 DEBUG "data": { 07:18:23 DEBUG "resultType": "vector", 07:18:23 DEBUG "result": [ 07:18:23 DEBUG { 07:18:23 DEBUG "metric": { 07:18:23 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG [...] 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 1712816303, 07:18:23 DEBUG "13780344832" 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG } 07:18:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816303" 07:18:23 INFO [http_cmd]: http status code OK 07:18:23 DEBUG --- status code --- 07:18:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:18:23 DEBUG --- http response --- 07:18:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:18:23 DEBUG ----- output ----- 07:18:23 DEBUG { 07:18:23 DEBUG "status": "success", 07:18:23 DEBUG "data": { 07:18:23 DEBUG "resultType": "vector", 07:18:23 DEBUG "result": [ 07:18:23 DEBUG { 07:18:23 DEBUG "metric": { 07:18:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG [...] 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG 07:18:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:18:23 DEBUG }, 07:18:23 DEBUG "value": [ 07:18:23 DEBUG 1712816303, 07:18:23 DEBUG "0" 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG ] 07:18:23 DEBUG } 07:18:23 DEBUG } 07:18:36 INFO 07:18:36 INFO [loop_until]: kubectl --namespace=xlou top pods 07:18:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:18:36 INFO [loop_until]: OK (rc = 0) 07:18:36 DEBUG --- stdout --- 07:18:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 5471m 8687Mi am-5b5bd8567b-8h2w4 6193m 8775Mi am-5b5bd8567b-qlkw6 4732m 8546Mi ds-cts-0 6215m 13146Mi ds-cts-1 5248m 13244Mi ds-cts-2 5712m 13373Mi ds-idrepo-0 5816m 5503Mi ds-idrepo-1 186m 5480Mi ds-idrepo-2 248m 5785Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 3m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 2m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4349m 16Mi 07:18:36 DEBUG --- stderr --- 07:18:36 DEBUG 07:18:40 INFO 07:18:40 INFO [loop_until]: kubectl --namespace=xlou top node 07:18:40 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:18:40 INFO [loop_until]: OK (rc = 0) 07:18:40 DEBUG --- stdout --- 07:18:40 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 2732Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 61m 0% 2780Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 4709m 15% 10236Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 7113m 23% 3147Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 5763m 19% 10306Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 142m 0% 3844Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 5793m 19% 10440Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 316m 1% 6475Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 7373m 46% 6209Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 5535m 34% 14011Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 4779m 30% 13812Mi 23% gke-xlou-cdm-ds-32e4dcb1-j2hk 6072m 38% 13907Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 187m 1% 6164Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 5926m 37% 3963Mi 6% 07:18:40 DEBUG --- stderr --- 07:18:40 DEBUG 07:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712816363" 07:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712816363" 07:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816363" 07:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816363" 07:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816363" 07:19:23 INFO [http_cmd]: http status code OK 07:19:23 DEBUG --- status code --- 07:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:19:23 DEBUG --- http response --- 07:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:19:23 DEBUG ----- output ----- 07:19:23 DEBUG { 07:19:23 DEBUG "status": "success", 07:19:23 DEBUG "data": { 07:19:23 DEBUG "resultType": "vector", 07:19:23 DEBUG "result": [ 07:19:23 DEBUG { 07:19:23 DEBUG "metric": { 07:19:23 DEBUG "pod": "ds-idrepo-2" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG [...] 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG "pod": "ds-idrepo-0" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 1712816363, 07:19:23 DEBUG "0" 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG } 07:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712816363" 07:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816363" 07:19:23 ERROR [http_cmd]: ERROR 07:19:23 INFO [http_cmd]: http status code OK 07:19:23 INFO [http_cmd]: http status code OK 07:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816363" 07:19:23 ERROR --- status code --- 07:19:23 DEBUG --- status code --- 07:19:23 DEBUG --- status code --- 07:19:23 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:19:23 ERROR --- http response --- 07:19:23 DEBUG --- http response --- 07:19:23 DEBUG --- http response --- 07:19:23 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:19:23 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:19:23 DEBUG ----- output ----- 07:19:23 INFO [http_cmd]: http status code OK 07:19:23 DEBUG ----- output ----- 07:19:23 DEBUG { 07:19:23 DEBUG --- status code --- 07:19:23 DEBUG { 07:19:23 DEBUG "status": "success", 07:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:19:23 DEBUG "status": "success", 07:19:23 DEBUG "data": { 07:19:23 DEBUG --- http response --- 07:19:23 DEBUG "data": { 07:19:23 DEBUG "resultType": "vector", 07:19:23 DEBUG "resultType": "vector", 07:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 07:19:23 DEBUG "result": [ 07:19:23 DEBUG "result": [ 07:19:23 DEBUG ----- output ----- 07:19:23 DEBUG { 07:19:23 DEBUG { 07:19:23 DEBUG { 07:19:23 DEBUG "metric": { 07:19:23 DEBUG "metric": { 07:19:23 DEBUG "status": "success", 07:19:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:19:23 DEBUG "instance": "10.142.0.107:9100" 07:19:23 DEBUG "data": { 07:19:23 DEBUG }, 07:19:23 DEBUG }, 07:19:23 DEBUG "resultType": "vector", 07:19:23 DEBUG "value": [ 07:19:23 DEBUG "value": [ 07:19:23 DEBUG "result": [ 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG { 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG "metric": { 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:19:23 DEBUG [...] 07:19:23 DEBUG [...] 07:19:23 DEBUG }, 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:19:23 DEBUG "instance": "10.142.0.77:9100" 07:19:23 DEBUG 07:19:23 DEBUG }, 07:19:23 DEBUG }, 07:19:23 DEBUG [...] 07:19:23 DEBUG "value": [ 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 07:19:23 DEBUG 1712816363, 07:19:23 DEBUG 1712816363, 07:19:23 DEBUG 07:19:23 DEBUG "0.025500000000000002" 07:19:23 DEBUG "9885227.2" 07:19:23 DEBUG 07:19:23 DEBUG ] 07:19:23 DEBUG ] 07:19:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:19:23 DEBUG } 07:19:23 DEBUG } 07:19:23 DEBUG }, 07:19:23 DEBUG ] 07:19:23 DEBUG ] 07:19:23 DEBUG "value": [ 07:19:23 DEBUG } 07:19:23 DEBUG } 07:19:23 DEBUG 1712816363, 07:19:23 DEBUG } 07:19:23 DEBUG } 07:19:23 DEBUG "0.0003098688018122234" 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG } 07:19:23 INFO [http_cmd]: http status code OK 07:19:23 DEBUG --- status code --- 07:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:19:23 DEBUG --- http response --- 07:19:23 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:19:23 INFO [http_cmd]: http status code OK 07:19:23 DEBUG --- status code --- 07:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:19:23 DEBUG --- http response --- 07:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:19:23 DEBUG ----- output ----- 07:19:23 DEBUG { 07:19:23 DEBUG "status": "success", 07:19:23 DEBUG "data": { 07:19:23 DEBUG "resultType": "vector", 07:19:23 DEBUG "result": [ 07:19:23 DEBUG { 07:19:23 DEBUG "metric": { 07:19:23 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG [...] 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 1712816363, 07:19:23 DEBUG "0.03141415804623062" 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG } 07:19:23 INFO [http_cmd]: http status code OK 07:19:23 DEBUG --- status code --- 07:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:19:23 DEBUG --- http response --- 07:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:19:23 DEBUG ----- output ----- 07:19:23 DEBUG { 07:19:23 DEBUG "status": "success", 07:19:23 DEBUG "data": { 07:19:23 DEBUG "resultType": "vector", 07:19:23 DEBUG "result": [ 07:19:23 DEBUG { 07:19:23 DEBUG "metric": { 07:19:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG [...] 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 1712816363, 07:19:23 DEBUG "1112.2" 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG } 07:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816363" 07:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816363" 07:19:23 INFO [http_cmd]: http status code OK 07:19:23 DEBUG --- status code --- 07:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:19:23 DEBUG --- http response --- 07:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 07:19:23 DEBUG ----- output ----- 07:19:23 DEBUG { 07:19:23 DEBUG "status": "success", 07:19:23 DEBUG "data": { 07:19:23 DEBUG "resultType": "vector", 07:19:23 DEBUG "result": [ 07:19:23 DEBUG { 07:19:23 DEBUG "metric": { 07:19:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG [...] 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 1712816363, 07:19:23 DEBUG "0" 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG } 07:19:23 INFO [http_cmd]: http status code OK 07:19:23 DEBUG --- status code --- 07:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:19:23 DEBUG --- http response --- 07:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:19:23 DEBUG ----- output ----- 07:19:23 DEBUG { 07:19:23 DEBUG "status": "success", 07:19:23 DEBUG "data": { 07:19:23 DEBUG "resultType": "vector", 07:19:23 DEBUG "result": [ 07:19:23 DEBUG { 07:19:23 DEBUG "metric": { 07:19:23 DEBUG "instance": "10.142.0.107:9100" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG [...] 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG "instance": "10.142.0.77:9100" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 1712816363, 07:19:23 DEBUG "0.002133333333333572" 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG } 07:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816363" 07:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712816363" 07:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816363" 07:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816363" 07:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816363" 07:19:23 INFO [http_cmd]: http status code OK 07:19:23 DEBUG --- status code --- 07:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:19:23 DEBUG --- http response --- 07:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:19:23 DEBUG ----- output ----- 07:19:23 DEBUG { 07:19:23 DEBUG "status": "success", 07:19:23 DEBUG "data": { 07:19:23 DEBUG "resultType": "vector", 07:19:23 DEBUG "result": [ 07:19:23 DEBUG { 07:19:23 DEBUG "metric": { 07:19:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG [...] 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 1712816363, 07:19:23 DEBUG "143986" 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG } 07:19:23 INFO [http_cmd]: http status code OK 07:19:23 DEBUG --- status code --- 07:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:19:23 DEBUG --- http response --- 07:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:19:23 DEBUG ----- output ----- 07:19:23 DEBUG { 07:19:23 DEBUG "status": "success", 07:19:23 DEBUG "data": { 07:19:23 DEBUG "resultType": "vector", 07:19:23 DEBUG "result": [ 07:19:23 DEBUG { 07:19:23 DEBUG "metric": { 07:19:23 DEBUG "pod": "ds-idrepo-2" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG [...] 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG "pod": "ds-cts-2" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 1712816363, 07:19:23 DEBUG "0" 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG } 07:19:23 INFO [http_cmd]: http status code OK 07:19:23 DEBUG --- status code --- 07:19:23 INFO [http_cmd]: http status code OK 07:19:23 INFO [http_cmd]: http status code OK 07:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:19:23 DEBUG --- status code --- 07:19:23 DEBUG --- status code --- 07:19:23 DEBUG --- http response --- 07:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:19:23 DEBUG --- http response --- 07:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:19:23 DEBUG --- http response --- 07:19:23 DEBUG ----- output ----- 07:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:19:23 DEBUG { 07:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 07:19:23 DEBUG ----- output ----- 07:19:23 DEBUG "status": "success", 07:19:23 DEBUG ----- output ----- 07:19:23 DEBUG { 07:19:23 DEBUG "data": { 07:19:23 DEBUG { 07:19:23 DEBUG "status": "success", 07:19:23 DEBUG "resultType": "vector", 07:19:23 DEBUG "status": "success", 07:19:23 DEBUG "data": { 07:19:23 DEBUG "result": [ 07:19:23 DEBUG "data": { 07:19:23 DEBUG "resultType": "vector", 07:19:23 DEBUG { 07:19:23 DEBUG "resultType": "vector", 07:19:23 DEBUG "result": [ 07:19:23 DEBUG "metric": { 07:19:23 DEBUG "result": [ 07:19:23 DEBUG { 07:19:23 DEBUG "pod": "ds-idrepo-2" 07:19:23 DEBUG { 07:19:23 DEBUG "metric": { 07:19:23 DEBUG }, 07:19:23 DEBUG "metric": { 07:19:23 DEBUG "instance": "10.142.0.107:9100" 07:19:23 DEBUG "value": [ 07:19:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:19:23 DEBUG }, 07:19:23 DEBUG 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG [...] 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG [...] 07:19:23 DEBUG 07:19:23 DEBUG [...] 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG "pod": "ds-cts-2" 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG }, 07:19:23 DEBUG 07:19:23 DEBUG "instance": "10.142.0.77:9100" 07:19:23 DEBUG "value": [ 07:19:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:19:23 DEBUG }, 07:19:23 DEBUG 1712816363, 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG "29.133333333333333" 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 1712816363, 07:19:23 DEBUG ] 07:19:23 DEBUG 1712816363, 07:19:23 DEBUG "53001554944" 07:19:23 DEBUG } 07:19:23 DEBUG "0" 07:19:23 DEBUG ] 07:19:23 DEBUG ] 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG } 07:19:23 DEBUG } 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG } 07:19:23 DEBUG } 07:19:23 DEBUG } 07:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816363" 07:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816363" 07:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816363" 07:19:23 INFO [http_cmd]: http status code OK 07:19:23 INFO [http_cmd]: http status code OK 07:19:23 DEBUG --- status code --- 07:19:23 DEBUG --- status code --- 07:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:19:23 DEBUG --- http response --- 07:19:23 DEBUG --- http response --- 07:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:19:23 DEBUG ----- output ----- 07:19:23 DEBUG { 07:19:23 DEBUG "status": "success", 07:19:23 DEBUG "data": { 07:19:23 DEBUG "resultType": "vector", 07:19:23 DEBUG "result": [ 07:19:23 DEBUG { 07:19:23 DEBUG "metric": { 07:19:23 DEBUG "pod": "ds-idrepo-2" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG [...] 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG "pod": "ds-idrepo-0" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 1712816363, 07:19:23 DEBUG "0" 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG } 07:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:19:23 DEBUG ----- output ----- 07:19:23 DEBUG { 07:19:23 DEBUG "status": "success", 07:19:23 DEBUG "data": { 07:19:23 DEBUG "resultType": "vector", 07:19:23 DEBUG "result": [ 07:19:23 DEBUG { 07:19:23 DEBUG "metric": { 07:19:23 DEBUG "pod": "ds-idrepo-2" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG [...] 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG "pod": "ds-idrepo-0" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 1712816363, 07:19:23 DEBUG "0" 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG } 07:19:23 INFO [http_cmd]: http status code OK 07:19:23 DEBUG --- status code --- 07:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:19:23 DEBUG --- http response --- 07:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 07:19:23 DEBUG ----- output ----- 07:19:23 DEBUG { 07:19:23 DEBUG "status": "success", 07:19:23 DEBUG "data": { 07:19:23 DEBUG "resultType": "vector", 07:19:23 DEBUG "result": [ 07:19:23 DEBUG { 07:19:23 DEBUG "metric": { 07:19:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG [...] 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 1712816363, 07:19:23 DEBUG "0" 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG } 07:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816363" 07:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816363" 07:19:23 INFO [http_cmd]: http status code OK 07:19:23 DEBUG --- status code --- 07:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:19:23 DEBUG --- http response --- 07:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:19:23 DEBUG ----- output ----- 07:19:23 DEBUG { 07:19:23 DEBUG "status": "success", 07:19:23 DEBUG "data": { 07:19:23 DEBUG "resultType": "vector", 07:19:23 DEBUG "result": [ 07:19:23 DEBUG { 07:19:23 DEBUG "metric": { 07:19:23 DEBUG "instance": "10.142.0.107:9100" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG [...] 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG "instance": "10.142.0.77:9100" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 1712816363, 07:19:23 DEBUG "9886210.8" 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG } 07:19:23 INFO [http_cmd]: http status code OK 07:19:23 DEBUG --- status code --- 07:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:19:23 DEBUG --- http response --- 07:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:19:23 DEBUG ----- output ----- 07:19:23 DEBUG { 07:19:23 DEBUG "status": "success", 07:19:23 DEBUG "data": { 07:19:23 DEBUG "resultType": "vector", 07:19:23 DEBUG "result": [ 07:19:23 DEBUG { 07:19:23 DEBUG "metric": { 07:19:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG [...] 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 1712816363, 07:19:23 DEBUG "0" 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG } 07:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816363" 07:19:23 INFO [http_cmd]: http status code OK 07:19:23 DEBUG --- status code --- 07:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:19:23 DEBUG --- http response --- 07:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:19:23 DEBUG ----- output ----- 07:19:23 DEBUG { 07:19:23 DEBUG "status": "success", 07:19:23 DEBUG "data": { 07:19:23 DEBUG "resultType": "vector", 07:19:23 DEBUG "result": [ 07:19:23 DEBUG { 07:19:23 DEBUG "metric": { 07:19:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG [...] 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 1712816363, 07:19:23 DEBUG "0" 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG } 07:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816363" 07:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816363" 07:19:23 INFO [http_cmd]: http status code OK 07:19:23 DEBUG --- status code --- 07:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:19:23 DEBUG --- http response --- 07:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:19:23 DEBUG ----- output ----- 07:19:23 DEBUG { 07:19:23 DEBUG "status": "success", 07:19:23 DEBUG "data": { 07:19:23 DEBUG "resultType": "vector", 07:19:23 DEBUG "result": [ 07:19:23 DEBUG { 07:19:23 DEBUG "metric": { 07:19:23 DEBUG "pod": "ds-idrepo-2" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG [...] 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG "pod": "ds-cts-2" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 1712816363, 07:19:23 DEBUG "0" 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG } 07:19:23 INFO [http_cmd]: http status code OK 07:19:23 DEBUG --- status code --- 07:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:19:23 DEBUG --- http response --- 07:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:19:23 DEBUG ----- output ----- 07:19:23 DEBUG { 07:19:23 DEBUG "status": "success", 07:19:23 DEBUG "data": { 07:19:23 DEBUG "resultType": "vector", 07:19:23 DEBUG "result": [ 07:19:23 DEBUG { 07:19:23 DEBUG "metric": { 07:19:23 DEBUG "pod": "ds-idrepo-2" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG [...] 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG "pod": "ds-cts-2" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 1712816363, 07:19:23 DEBUG "0" 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG } 07:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816363" 07:19:23 INFO [http_cmd]: http status code OK 07:19:23 DEBUG --- status code --- 07:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:19:23 DEBUG --- http response --- 07:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:19:23 DEBUG ----- output ----- 07:19:23 DEBUG { 07:19:23 DEBUG "status": "success", 07:19:23 DEBUG "data": { 07:19:23 DEBUG "resultType": "vector", 07:19:23 DEBUG "result": [ 07:19:23 DEBUG { 07:19:23 DEBUG "metric": { 07:19:23 DEBUG "instance": "10.142.0.107:9100" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG [...] 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG "instance": "10.142.0.77:9100" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 1712816363, 07:19:23 DEBUG "0.000333333333333267" 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG } 07:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816363" 07:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712816363" 07:19:23 INFO [http_cmd]: http status code OK 07:19:23 DEBUG --- status code --- 07:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:19:23 DEBUG --- http response --- 07:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:19:23 DEBUG ----- output ----- 07:19:23 DEBUG { 07:19:23 DEBUG "status": "success", 07:19:23 DEBUG "data": { 07:19:23 DEBUG "resultType": "vector", 07:19:23 DEBUG "result": [ 07:19:23 DEBUG { 07:19:23 DEBUG "metric": { 07:19:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG [...] 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 1712816363, 07:19:23 DEBUG "0" 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG } 07:19:23 INFO [http_cmd]: http status code OK 07:19:23 DEBUG --- status code --- 07:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:19:23 DEBUG --- http response --- 07:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:19:23 DEBUG ----- output ----- 07:19:23 DEBUG { 07:19:23 DEBUG "status": "success", 07:19:23 DEBUG "data": { 07:19:23 DEBUG "resultType": "vector", 07:19:23 DEBUG "result": [ 07:19:23 DEBUG { 07:19:23 DEBUG "metric": { 07:19:23 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG [...] 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 1712816363, 07:19:23 DEBUG "13899055104" 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG } 07:19:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816363" 07:19:23 INFO [http_cmd]: http status code OK 07:19:23 DEBUG --- status code --- 07:19:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:19:23 DEBUG --- http response --- 07:19:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:19:23 DEBUG ----- output ----- 07:19:23 DEBUG { 07:19:23 DEBUG "status": "success", 07:19:23 DEBUG "data": { 07:19:23 DEBUG "resultType": "vector", 07:19:23 DEBUG "result": [ 07:19:23 DEBUG { 07:19:23 DEBUG "metric": { 07:19:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG [...] 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG 07:19:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:19:23 DEBUG }, 07:19:23 DEBUG "value": [ 07:19:23 DEBUG 1712816363, 07:19:23 DEBUG "0" 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG ] 07:19:23 DEBUG } 07:19:23 DEBUG } 07:19:36 INFO 07:19:36 INFO [loop_until]: kubectl --namespace=xlou top pods 07:19:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:19:36 INFO [loop_until]: OK (rc = 0) 07:19:36 DEBUG --- stdout --- 07:19:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 1243m 8171Mi am-5b5bd8567b-8h2w4 1259m 8684Mi am-5b5bd8567b-qlkw6 1148m 8432Mi ds-cts-0 3246m 12180Mi ds-cts-1 3354m 13246Mi ds-cts-2 3299m 13371Mi ds-idrepo-0 1135m 5503Mi ds-idrepo-1 10m 5480Mi ds-idrepo-2 11m 5784Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 4m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 2m 67Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 1m 16Mi 07:19:36 DEBUG --- stderr --- 07:19:36 DEBUG 07:19:40 INFO 07:19:40 INFO [loop_until]: kubectl --namespace=xlou top node 07:19:40 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:19:40 INFO [loop_until]: OK (rc = 0) 07:19:40 DEBUG --- stdout --- 07:19:40 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 2733Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 60m 0% 2782Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1168m 3% 10182Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 113m 0% 3067Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1381m 4% 9854Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 126m 0% 3871Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1349m 4% 10391Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 64m 0% 6476Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 1184m 7% 6210Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 3312m 20% 14045Mi 23% gke-xlou-cdm-ds-32e4dcb1-gxzj 3300m 20% 12880Mi 21% gke-xlou-cdm-ds-32e4dcb1-j2hk 3400m 21% 13943Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 66m 0% 6165Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 70m 0% 3879Mi 6% 07:19:40 DEBUG --- stderr --- 07:19:40 DEBUG 07:19:52 WARNING File /tmp/lodemon_data-20240411_022725/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [11/Apr/2024 07:19:52] "GET /monitoring/average?start_time=20240411_060313&stop_time=20240411_061751 HTTP/1.1" 200 - 07:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712816423" 07:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816423" 07:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816423" 07:20:23 ERROR [http_cmd]: ERROR 07:20:23 INFO [http_cmd]: http status code OK 07:20:23 ERROR --- status code --- 07:20:23 DEBUG --- status code --- 07:20:23 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:20:23 ERROR --- http response --- 07:20:23 DEBUG --- http response --- 07:20:23 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:20:23 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:20:23 DEBUG ----- output ----- 07:20:23 DEBUG { 07:20:23 DEBUG "status": "success", 07:20:23 DEBUG "data": { 07:20:23 DEBUG "resultType": "vector", 07:20:23 DEBUG "result": [ 07:20:23 DEBUG { 07:20:23 DEBUG "metric": { 07:20:23 DEBUG "pod": "ds-idrepo-2" 07:20:23 DEBUG }, 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG [...] 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG "pod": "ds-idrepo-0" 07:20:23 DEBUG }, 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 1712816423, 07:20:23 DEBUG "0" 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG } 07:20:23 INFO [http_cmd]: http status code OK 07:20:23 DEBUG --- status code --- 07:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:20:23 DEBUG --- http response --- 07:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 07:20:23 DEBUG ----- output ----- 07:20:23 DEBUG { 07:20:23 DEBUG "status": "success", 07:20:23 DEBUG "data": { 07:20:23 DEBUG "resultType": "vector", 07:20:23 DEBUG "result": [ 07:20:23 DEBUG { 07:20:23 DEBUG "metric": { 07:20:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:20:23 DEBUG }, 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG [...] 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG "pod": "ds-cts-0" 07:20:23 DEBUG }, 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 1712816423, 07:20:23 DEBUG "3.024410664772383" 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG } 07:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712816423" 07:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816423" 07:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816423" 07:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816423" 07:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712816423" 07:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816423" 07:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816423" 07:20:23 INFO [http_cmd]: http status code OK 07:20:23 INFO [http_cmd]: http status code OK 07:20:23 INFO [http_cmd]: http status code OK 07:20:23 INFO [http_cmd]: http status code OK 07:20:23 DEBUG --- status code --- 07:20:23 INFO [http_cmd]: http status code OK 07:20:23 DEBUG --- status code --- 07:20:23 DEBUG --- status code --- 07:20:23 DEBUG --- status code --- 07:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:20:23 DEBUG --- status code --- 07:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:20:23 DEBUG --- http response --- 07:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:20:23 DEBUG --- http response --- 07:20:23 DEBUG --- http response --- 07:20:23 DEBUG --- http response --- 07:20:23 DEBUG --- http response --- 07:20:23 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 07:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:20:23 DEBUG ----- output ----- 07:20:23 DEBUG ----- output ----- 07:20:23 DEBUG ----- output ----- 07:20:23 DEBUG ----- output ----- 07:20:23 DEBUG { 07:20:23 DEBUG { 07:20:23 DEBUG { 07:20:23 DEBUG { 07:20:23 DEBUG "status": "success", 07:20:23 DEBUG "status": "success", 07:20:23 DEBUG "status": "success", 07:20:23 DEBUG "status": "success", 07:20:23 INFO [http_cmd]: http status code OK 07:20:23 DEBUG "data": { 07:20:23 DEBUG "data": { 07:20:23 DEBUG "data": { 07:20:23 DEBUG "data": { 07:20:23 DEBUG --- status code --- 07:20:23 DEBUG "resultType": "vector", 07:20:23 DEBUG "resultType": "vector", 07:20:23 DEBUG "resultType": "vector", 07:20:23 DEBUG "resultType": "vector", 07:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:20:23 DEBUG "result": [ 07:20:23 DEBUG "result": [ 07:20:23 DEBUG "result": [ 07:20:23 DEBUG "result": [ 07:20:23 DEBUG --- http response --- 07:20:23 DEBUG { 07:20:23 DEBUG { 07:20:23 DEBUG { 07:20:23 DEBUG { 07:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:20:23 DEBUG "metric": { 07:20:23 DEBUG "metric": { 07:20:23 DEBUG "metric": { 07:20:23 DEBUG "metric": { 07:20:23 DEBUG ----- output ----- 07:20:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:20:23 DEBUG "instance": "10.142.0.107:9100" 07:20:23 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:20:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:20:23 DEBUG { 07:20:23 DEBUG }, 07:20:23 DEBUG }, 07:20:23 DEBUG }, 07:20:23 DEBUG }, 07:20:23 DEBUG "status": "success", 07:20:23 DEBUG "value": [ 07:20:23 DEBUG "value": [ 07:20:23 DEBUG "value": [ 07:20:23 DEBUG "value": [ 07:20:23 DEBUG "data": { 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG "resultType": "vector", 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG "result": [ 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG { 07:20:23 DEBUG [...] 07:20:23 DEBUG [...] 07:20:23 DEBUG [...] 07:20:23 DEBUG [...] 07:20:23 DEBUG "metric": { 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG }, 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG "value": [ 07:20:23 DEBUG "pod": "ds-cts-0" 07:20:23 DEBUG "instance": "10.142.0.77:9100" 07:20:23 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:20:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:20:23 DEBUG 07:20:23 DEBUG }, 07:20:23 DEBUG }, 07:20:23 DEBUG }, 07:20:23 DEBUG }, 07:20:23 DEBUG 07:20:23 DEBUG "value": [ 07:20:23 DEBUG "value": [ 07:20:23 DEBUG "value": [ 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 07:20:23 DEBUG 1712816423, 07:20:23 DEBUG 1712816423, 07:20:23 DEBUG 1712816423, 07:20:23 DEBUG 1712816423, 07:20:23 DEBUG [...] 07:20:23 DEBUG "618132.9267447501" 07:20:23 DEBUG "10261233.233333332" 07:20:23 DEBUG "0.029591483625887413" 07:20:23 DEBUG "0.026" 07:20:23 DEBUG 07:20:23 DEBUG ] 07:20:23 DEBUG ] 07:20:23 DEBUG ] 07:20:23 DEBUG ] 07:20:23 DEBUG 07:20:23 DEBUG } 07:20:23 DEBUG } 07:20:23 DEBUG } 07:20:23 DEBUG } 07:20:23 DEBUG 07:20:23 DEBUG ] 07:20:23 DEBUG ] 07:20:23 DEBUG ] 07:20:23 DEBUG ] 07:20:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:20:23 DEBUG } 07:20:23 DEBUG } 07:20:23 DEBUG } 07:20:23 DEBUG } 07:20:23 DEBUG }, 07:20:23 DEBUG } 07:20:23 DEBUG } 07:20:23 DEBUG } 07:20:23 DEBUG } 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 1712816423, 07:20:23 DEBUG "1166.5333333333333" 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG } 07:20:23 INFO [http_cmd]: http status code OK 07:20:23 DEBUG --- status code --- 07:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:20:23 DEBUG --- http response --- 07:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:20:23 DEBUG ----- output ----- 07:20:23 DEBUG { 07:20:23 DEBUG "status": "success", 07:20:23 DEBUG "data": { 07:20:23 DEBUG "resultType": "vector", 07:20:23 DEBUG "result": [ 07:20:23 DEBUG { 07:20:23 DEBUG "metric": { 07:20:23 DEBUG "instance": "10.142.0.107:9100" 07:20:23 DEBUG }, 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG [...] 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG "instance": "10.142.0.77:9100" 07:20:23 DEBUG }, 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 1712816423, 07:20:23 DEBUG "0.002066666666666587" 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG } 07:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816423" 07:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712816423" 07:20:23 INFO [http_cmd]: http status code OK 07:20:23 DEBUG --- status code --- 07:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:20:23 DEBUG --- http response --- 07:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:20:23 DEBUG ----- output ----- 07:20:23 DEBUG { 07:20:23 DEBUG "status": "success", 07:20:23 DEBUG "data": { 07:20:23 DEBUG "resultType": "vector", 07:20:23 DEBUG "result": [ 07:20:23 DEBUG { 07:20:23 DEBUG "metric": { 07:20:23 DEBUG "pod": "ds-idrepo-2" 07:20:23 DEBUG }, 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG [...] 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG "pod": "ds-cts-2" 07:20:23 DEBUG }, 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 1712816423, 07:20:23 DEBUG "0" 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG } 07:20:23 INFO [http_cmd]: http status code OK 07:20:23 DEBUG --- status code --- 07:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:20:23 DEBUG --- http response --- 07:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:20:23 DEBUG ----- output ----- 07:20:23 DEBUG { 07:20:23 DEBUG "status": "success", 07:20:23 DEBUG "data": { 07:20:23 DEBUG "resultType": "vector", 07:20:23 DEBUG "result": [ 07:20:23 DEBUG { 07:20:23 DEBUG "metric": { 07:20:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:20:23 DEBUG }, 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG [...] 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:20:23 DEBUG }, 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 1712816423, 07:20:23 DEBUG "76488" 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG } 07:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816423" 07:20:23 INFO [http_cmd]: http status code OK 07:20:23 DEBUG --- status code --- 07:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:20:23 DEBUG --- http response --- 07:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:20:23 DEBUG ----- output ----- 07:20:23 DEBUG { 07:20:23 DEBUG "status": "success", 07:20:23 DEBUG "data": { 07:20:23 DEBUG "resultType": "vector", 07:20:23 DEBUG "result": [ 07:20:23 DEBUG { 07:20:23 DEBUG "metric": { 07:20:23 DEBUG "pod": "ds-idrepo-2" 07:20:23 DEBUG }, 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG [...] 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG "pod": "ds-cts-2" 07:20:23 DEBUG }, 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 1712816423, 07:20:23 DEBUG "29.3" 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG } 07:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816423" 07:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816423" 07:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816423" 07:20:23 INFO [http_cmd]: http status code OK 07:20:23 INFO [http_cmd]: http status code OK 07:20:23 DEBUG --- status code --- 07:20:23 DEBUG --- status code --- 07:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:20:23 DEBUG --- http response --- 07:20:23 DEBUG --- http response --- 07:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 07:20:23 DEBUG ----- output ----- 07:20:23 DEBUG { 07:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:20:23 DEBUG "status": "success", 07:20:23 DEBUG ----- output ----- 07:20:23 DEBUG "data": { 07:20:23 DEBUG { 07:20:23 DEBUG "resultType": "vector", 07:20:23 DEBUG "status": "success", 07:20:23 DEBUG "result": [ 07:20:23 DEBUG "data": { 07:20:23 DEBUG { 07:20:23 DEBUG "resultType": "vector", 07:20:23 DEBUG "metric": { 07:20:23 DEBUG "result": [ 07:20:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:20:23 DEBUG { 07:20:23 DEBUG }, 07:20:23 DEBUG "metric": { 07:20:23 DEBUG "value": [ 07:20:23 DEBUG "instance": "10.142.0.107:9100" 07:20:23 DEBUG 07:20:23 DEBUG }, 07:20:23 DEBUG 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG [...] 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG [...] 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG "pod": "ds-cts-0" 07:20:23 DEBUG 07:20:23 DEBUG }, 07:20:23 DEBUG 07:20:23 DEBUG "value": [ 07:20:23 DEBUG "instance": "10.142.0.77:9100" 07:20:23 DEBUG 1712816423, 07:20:23 DEBUG }, 07:20:23 DEBUG "141.93637812738234" 07:20:23 DEBUG "value": [ 07:20:23 DEBUG ] 07:20:23 DEBUG 1712816423, 07:20:23 DEBUG } 07:20:23 DEBUG "53002902528" 07:20:23 DEBUG ] 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG } 07:20:23 DEBUG } 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG } 07:20:23 INFO [http_cmd]: http status code OK 07:20:23 DEBUG --- status code --- 07:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:20:23 DEBUG --- http response --- 07:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:20:23 DEBUG ----- output ----- 07:20:23 DEBUG { 07:20:23 DEBUG "status": "success", 07:20:23 DEBUG "data": { 07:20:23 DEBUG "resultType": "vector", 07:20:23 DEBUG "result": [ 07:20:23 DEBUG { 07:20:23 DEBUG "metric": { 07:20:23 DEBUG "pod": "ds-idrepo-2" 07:20:23 DEBUG }, 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG [...] 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG "pod": "ds-idrepo-0" 07:20:23 DEBUG }, 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 1712816423, 07:20:23 DEBUG "0" 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG } 07:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816423" 07:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816423" 07:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816423" 07:20:23 INFO [http_cmd]: http status code OK 07:20:23 DEBUG --- status code --- 07:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:20:23 DEBUG --- http response --- 07:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:20:23 INFO [http_cmd]: http status code OK 07:20:23 DEBUG ----- output ----- 07:20:23 DEBUG --- status code --- 07:20:23 DEBUG { 07:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:20:23 DEBUG "status": "success", 07:20:23 DEBUG --- http response --- 07:20:23 DEBUG "data": { 07:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 07:20:23 DEBUG "resultType": "vector", 07:20:23 DEBUG ----- output ----- 07:20:23 DEBUG "result": [ 07:20:23 DEBUG { 07:20:23 DEBUG { 07:20:23 DEBUG "status": "success", 07:20:23 DEBUG "metric": { 07:20:23 DEBUG "data": { 07:20:23 DEBUG "pod": "ds-idrepo-2" 07:20:23 DEBUG "resultType": "vector", 07:20:23 DEBUG }, 07:20:23 DEBUG "result": [ 07:20:23 DEBUG "value": [ 07:20:23 DEBUG { 07:20:23 DEBUG 07:20:23 DEBUG "metric": { 07:20:23 DEBUG 07:20:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:20:23 DEBUG 07:20:23 DEBUG }, 07:20:23 DEBUG [...] 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG "pod": "ds-idrepo-0" 07:20:23 DEBUG [...] 07:20:23 DEBUG }, 07:20:23 DEBUG 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 07:20:23 DEBUG 1712816423, 07:20:23 DEBUG 07:20:23 DEBUG "0" 07:20:23 DEBUG "pod": "ds-cts-0" 07:20:23 DEBUG ] 07:20:23 DEBUG }, 07:20:23 DEBUG } 07:20:23 DEBUG "value": [ 07:20:23 DEBUG ] 07:20:23 DEBUG 1712816423, 07:20:23 DEBUG } 07:20:23 DEBUG "2210517.1529558524" 07:20:23 DEBUG } 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG } 07:20:23 INFO [http_cmd]: http status code OK 07:20:23 DEBUG --- status code --- 07:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:20:23 DEBUG --- http response --- 07:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:20:23 DEBUG ----- output ----- 07:20:23 DEBUG { 07:20:23 DEBUG "status": "success", 07:20:23 DEBUG "data": { 07:20:23 DEBUG "resultType": "vector", 07:20:23 DEBUG "result": [ 07:20:23 DEBUG { 07:20:23 DEBUG "metric": { 07:20:23 DEBUG "instance": "10.142.0.107:9100" 07:20:23 DEBUG }, 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG [...] 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG "instance": "10.142.0.77:9100" 07:20:23 DEBUG }, 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 1712816423, 07:20:23 DEBUG "10244503.3" 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG } 07:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816423" 07:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816423" 07:20:23 INFO [http_cmd]: http status code OK 07:20:23 INFO [http_cmd]: http status code OK 07:20:23 DEBUG --- status code --- 07:20:23 DEBUG --- status code --- 07:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:20:23 DEBUG --- http response --- 07:20:23 DEBUG --- http response --- 07:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:20:23 DEBUG ----- output ----- 07:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:20:23 DEBUG { 07:20:23 DEBUG ----- output ----- 07:20:23 DEBUG "status": "success", 07:20:23 DEBUG { 07:20:23 DEBUG "data": { 07:20:23 DEBUG "status": "success", 07:20:23 DEBUG "resultType": "vector", 07:20:23 DEBUG "data": { 07:20:23 DEBUG "result": [ 07:20:23 DEBUG "resultType": "vector", 07:20:23 DEBUG { 07:20:23 DEBUG "result": [ 07:20:23 DEBUG "metric": { 07:20:23 DEBUG { 07:20:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:20:23 DEBUG "metric": { 07:20:23 DEBUG }, 07:20:23 DEBUG "pod": "ds-idrepo-2" 07:20:23 DEBUG "value": [ 07:20:23 DEBUG }, 07:20:23 DEBUG 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG [...] 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG [...] 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:20:23 DEBUG 07:20:23 DEBUG }, 07:20:23 DEBUG "pod": "ds-cts-2" 07:20:23 DEBUG "value": [ 07:20:23 DEBUG }, 07:20:23 DEBUG 1712816423, 07:20:23 DEBUG "value": [ 07:20:23 DEBUG "0" 07:20:23 DEBUG 1712816423, 07:20:23 DEBUG ] 07:20:23 DEBUG "0" 07:20:23 DEBUG } 07:20:23 DEBUG ] 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG } 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG } 07:20:23 DEBUG } 07:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816423" 07:20:23 INFO [http_cmd]: http status code OK 07:20:23 DEBUG --- status code --- 07:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:20:23 DEBUG --- http response --- 07:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:20:23 DEBUG ----- output ----- 07:20:23 DEBUG { 07:20:23 DEBUG "status": "success", 07:20:23 DEBUG "data": { 07:20:23 DEBUG "resultType": "vector", 07:20:23 DEBUG "result": [ 07:20:23 DEBUG { 07:20:23 DEBUG "metric": { 07:20:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:20:23 DEBUG }, 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG [...] 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:20:23 DEBUG }, 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 1712816423, 07:20:23 DEBUG "0" 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG } 07:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816423" 07:20:23 INFO [http_cmd]: http status code OK 07:20:23 DEBUG --- status code --- 07:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:20:23 DEBUG --- http response --- 07:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:20:23 DEBUG ----- output ----- 07:20:23 DEBUG { 07:20:23 DEBUG "status": "success", 07:20:23 DEBUG "data": { 07:20:23 DEBUG "resultType": "vector", 07:20:23 DEBUG "result": [ 07:20:23 DEBUG { 07:20:23 DEBUG "metric": { 07:20:23 DEBUG "pod": "ds-idrepo-2" 07:20:23 DEBUG }, 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG [...] 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG "pod": "ds-cts-2" 07:20:23 DEBUG }, 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 1712816423, 07:20:23 DEBUG "0.000033333333333333335" 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG } 07:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816423" 07:20:23 INFO [http_cmd]: http status code OK 07:20:23 DEBUG --- status code --- 07:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:20:23 DEBUG --- http response --- 07:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:20:23 DEBUG ----- output ----- 07:20:23 DEBUG { 07:20:23 DEBUG "status": "success", 07:20:23 DEBUG "data": { 07:20:23 DEBUG "resultType": "vector", 07:20:23 DEBUG "result": [ 07:20:23 DEBUG { 07:20:23 DEBUG "metric": { 07:20:23 DEBUG "instance": "10.142.0.107:9100" 07:20:23 DEBUG }, 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG [...] 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG "instance": "10.142.0.77:9100" 07:20:23 DEBUG }, 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 1712816423, 07:20:23 DEBUG "0" 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG } 07:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816423" 07:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712816423" 07:20:23 INFO [http_cmd]: http status code OK 07:20:23 DEBUG --- status code --- 07:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:20:23 DEBUG --- http response --- 07:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:20:23 DEBUG ----- output ----- 07:20:23 DEBUG { 07:20:23 DEBUG "status": "success", 07:20:23 DEBUG "data": { 07:20:23 DEBUG "resultType": "vector", 07:20:23 DEBUG "result": [ 07:20:23 DEBUG { 07:20:23 DEBUG "metric": { 07:20:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:20:23 DEBUG }, 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG [...] 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:20:23 DEBUG }, 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 1712816423, 07:20:23 DEBUG "0" 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG } 07:20:23 INFO [http_cmd]: http status code OK 07:20:23 DEBUG --- status code --- 07:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:20:23 DEBUG --- http response --- 07:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:20:23 DEBUG ----- output ----- 07:20:23 DEBUG { 07:20:23 DEBUG "status": "success", 07:20:23 DEBUG "data": { 07:20:23 DEBUG "resultType": "vector", 07:20:23 DEBUG "result": [ 07:20:23 DEBUG { 07:20:23 DEBUG "metric": { 07:20:23 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:20:23 DEBUG }, 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG [...] 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 07:20:23 DEBUG }, 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 1712816423, 07:20:23 DEBUG "12882407424" 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG } 07:20:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816423" 07:20:23 INFO [http_cmd]: http status code OK 07:20:23 DEBUG --- status code --- 07:20:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:20:23 DEBUG --- http response --- 07:20:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:20:23 DEBUG ----- output ----- 07:20:23 DEBUG { 07:20:23 DEBUG "status": "success", 07:20:23 DEBUG "data": { 07:20:23 DEBUG "resultType": "vector", 07:20:23 DEBUG "result": [ 07:20:23 DEBUG { 07:20:23 DEBUG "metric": { 07:20:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:20:23 DEBUG }, 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG [...] 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG 07:20:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:20:23 DEBUG }, 07:20:23 DEBUG "value": [ 07:20:23 DEBUG 1712816423, 07:20:23 DEBUG "0" 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG ] 07:20:23 DEBUG } 07:20:23 DEBUG } 07:20:36 INFO 07:20:36 INFO [loop_until]: kubectl --namespace=xlou top pods 07:20:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:20:36 INFO [loop_until]: OK (rc = 0) 07:20:36 DEBUG --- stdout --- 07:20:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 1346m 8163Mi am-5b5bd8567b-8h2w4 1291m 8673Mi am-5b5bd8567b-qlkw6 1119m 8425Mi ds-cts-0 3046m 12217Mi ds-cts-1 3224m 13278Mi ds-cts-2 3132m 12449Mi ds-idrepo-0 1224m 5503Mi ds-idrepo-1 11m 5480Mi ds-idrepo-2 11m 5784Mi end-user-ui-58878d7668-gjq6d 1m 3Mi idm-668d898cfd-2vqdm 4m 1010Mi idm-668d898cfd-st2t7 4m 1135Mi lodemon-ccc964465-p7gdm 9m 68Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 4m 16Mi 07:20:36 DEBUG --- stderr --- 07:20:36 DEBUG 07:20:40 INFO 07:20:40 INFO [loop_until]: kubectl --namespace=xlou top node 07:20:40 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:20:40 INFO [loop_until]: OK (rc = 0) 07:20:40 DEBUG --- stdout --- 07:20:40 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 77m 0% 2734Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 65m 0% 2793Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1194m 3% 10139Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 127m 0% 3062Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1428m 4% 9817Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 139m 0% 3850Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1400m 4% 10369Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 64m 0% 6474Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 1257m 7% 6209Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 3255m 20% 13120Mi 22% gke-xlou-cdm-ds-32e4dcb1-gxzj 3202m 20% 12901Mi 21% gke-xlou-cdm-ds-32e4dcb1-j2hk 3338m 21% 13974Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 69m 0% 6169Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 87m 0% 3877Mi 6% 07:20:40 DEBUG --- stderr --- 07:20:40 DEBUG 07:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712816483" 07:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816483" 07:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816483" 07:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816483" 07:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816483" 07:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712816483" 07:21:23 ERROR [http_cmd]: ERROR 07:21:23 INFO [http_cmd]: http status code OK 07:21:23 ERROR --- status code --- 07:21:23 INFO [http_cmd]: http status code OK 07:21:23 INFO [http_cmd]: http status code OK 07:21:23 DEBUG --- status code --- 07:21:23 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:21:23 DEBUG --- status code --- 07:21:23 DEBUG --- status code --- 07:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:21:23 DEBUG --- http response --- 07:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:21:23 ERROR --- http response --- 07:21:23 DEBUG --- http response --- 07:21:23 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:21:23 DEBUG --- http response --- 07:21:23 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:21:23 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:21:23 DEBUG ----- output ----- 07:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816483" 07:21:23 DEBUG ----- output ----- 07:21:23 DEBUG { 07:21:23 DEBUG { 07:21:23 DEBUG "status": "success", 07:21:23 DEBUG "status": "success", 07:21:23 DEBUG "data": { 07:21:23 DEBUG "data": { 07:21:23 DEBUG "resultType": "vector", 07:21:23 DEBUG "resultType": "vector", 07:21:23 DEBUG "result": [ 07:21:23 DEBUG "result": [ 07:21:23 DEBUG { 07:21:23 DEBUG { 07:21:23 DEBUG "metric": { 07:21:23 DEBUG "metric": { 07:21:23 DEBUG "pod": "ds-idrepo-2" 07:21:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:21:23 DEBUG }, 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 07:21:23 INFO [http_cmd]: http status code OK 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG --- status code --- 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:21:23 DEBUG 07:21:23 DEBUG [...] 07:21:23 DEBUG --- http response --- 07:21:23 DEBUG [...] 07:21:23 DEBUG 07:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG ----- output ----- 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG { 07:21:23 DEBUG 07:21:23 DEBUG "pod": "ds-idrepo-0" 07:21:23 DEBUG "status": "success", 07:21:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:21:23 DEBUG }, 07:21:23 DEBUG "data": { 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG "resultType": "vector", 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 1712816483, 07:21:23 DEBUG "result": [ 07:21:23 DEBUG 1712816483, 07:21:23 DEBUG "0" 07:21:23 DEBUG { 07:21:23 DEBUG "0.6897514184773065" 07:21:23 DEBUG ] 07:21:23 DEBUG "metric": { 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:21:23 DEBUG } 07:21:23 DEBUG ] 07:21:23 DEBUG }, 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG "value": [ 07:21:23 DEBUG } 07:21:23 DEBUG } 07:21:23 DEBUG 07:21:23 DEBUG } 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG [...] 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 1712816483, 07:21:23 DEBUG "0.026" 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG } 07:21:23 INFO [http_cmd]: http status code OK 07:21:23 DEBUG --- status code --- 07:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:21:23 DEBUG --- http response --- 07:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:21:23 DEBUG ----- output ----- 07:21:23 DEBUG { 07:21:23 DEBUG "status": "success", 07:21:23 DEBUG "data": { 07:21:23 DEBUG "resultType": "vector", 07:21:23 DEBUG "result": [ 07:21:23 DEBUG { 07:21:23 DEBUG "metric": { 07:21:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG [...] 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 1712816483, 07:21:23 DEBUG "1187185.6864580512" 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG } 07:21:23 INFO [http_cmd]: http status code OK 07:21:23 DEBUG --- status code --- 07:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:21:23 DEBUG --- http response --- 07:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:21:23 DEBUG ----- output ----- 07:21:23 DEBUG { 07:21:23 DEBUG "status": "success", 07:21:23 DEBUG "data": { 07:21:23 DEBUG "resultType": "vector", 07:21:23 DEBUG "result": [ 07:21:23 DEBUG { 07:21:23 DEBUG "metric": { 07:21:23 DEBUG "instance": "10.142.0.107:9100" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG [...] 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG "instance": "10.142.0.77:9100" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 1712816483, 07:21:23 DEBUG "3578108.8" 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG } 07:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712816483" 07:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816483" 07:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712816483" 07:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816483" 07:21:23 INFO [http_cmd]: http status code OK 07:21:23 DEBUG --- status code --- 07:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:21:23 DEBUG --- http response --- 07:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:21:23 DEBUG ----- output ----- 07:21:23 DEBUG { 07:21:23 DEBUG "status": "success", 07:21:23 DEBUG "data": { 07:21:23 DEBUG "resultType": "vector", 07:21:23 DEBUG "result": [ 07:21:23 DEBUG { 07:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816483" 07:21:23 DEBUG "metric": { 07:21:23 DEBUG "pod": "ds-idrepo-2" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG [...] 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG "pod": "ds-cts-2" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 1712816483, 07:21:23 DEBUG "0" 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG } 07:21:23 INFO [http_cmd]: http status code OK 07:21:23 DEBUG --- status code --- 07:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:21:23 DEBUG --- http response --- 07:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:21:23 DEBUG ----- output ----- 07:21:23 DEBUG { 07:21:23 DEBUG "status": "success", 07:21:23 DEBUG "data": { 07:21:23 DEBUG "resultType": "vector", 07:21:23 DEBUG "result": [ 07:21:23 DEBUG { 07:21:23 DEBUG "metric": { 07:21:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG [...] 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 1712816483, 07:21:23 DEBUG "888.2333333333333" 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG } 07:21:23 INFO [http_cmd]: http status code OK 07:21:23 DEBUG --- status code --- 07:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:21:23 DEBUG --- http response --- 07:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:21:23 DEBUG ----- output ----- 07:21:23 DEBUG { 07:21:23 DEBUG "status": "success", 07:21:23 DEBUG "data": { 07:21:23 DEBUG "resultType": "vector", 07:21:23 DEBUG "result": [ 07:21:23 DEBUG { 07:21:23 DEBUG "metric": { 07:21:23 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG [...] 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 1712816483, 07:21:23 DEBUG "0.8752700986493501" 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 INFO [http_cmd]: http status code OK 07:21:23 DEBUG ] 07:21:23 DEBUG --- status code --- 07:21:23 INFO [http_cmd]: http status code OK 07:21:23 DEBUG } 07:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:21:23 DEBUG --- status code --- 07:21:23 DEBUG } 07:21:23 DEBUG --- http response --- 07:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:21:23 DEBUG --- http response --- 07:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:21:23 DEBUG ----- output ----- 07:21:23 DEBUG ----- output ----- 07:21:23 DEBUG { 07:21:23 DEBUG { 07:21:23 DEBUG "status": "success", 07:21:23 DEBUG "status": "success", 07:21:23 DEBUG "data": { 07:21:23 DEBUG "data": { 07:21:23 DEBUG "resultType": "vector", 07:21:23 DEBUG "resultType": "vector", 07:21:23 DEBUG "result": [ 07:21:23 DEBUG "result": [ 07:21:23 DEBUG { 07:21:23 DEBUG { 07:21:23 DEBUG "metric": { 07:21:23 DEBUG "metric": { 07:21:23 DEBUG "instance": "10.142.0.107:9100" 07:21:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:21:23 DEBUG }, 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG [...] 07:21:23 DEBUG [...] 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG "instance": "10.142.0.77:9100" 07:21:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:21:23 DEBUG }, 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 1712816483, 07:21:23 DEBUG 1712816483, 07:21:23 DEBUG "0.0026999999999996285" 07:21:23 DEBUG "13323.5" 07:21:23 DEBUG ] 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG } 07:21:23 DEBUG ] 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG } 07:21:23 DEBUG } 07:21:23 DEBUG } 07:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816483" 07:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816483" 07:21:23 INFO [http_cmd]: http status code OK 07:21:23 DEBUG --- status code --- 07:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:21:23 DEBUG --- http response --- 07:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:21:23 DEBUG ----- output ----- 07:21:23 DEBUG { 07:21:23 DEBUG "status": "success", 07:21:23 DEBUG "data": { 07:21:23 DEBUG "resultType": "vector", 07:21:23 DEBUG "result": [ 07:21:23 DEBUG { 07:21:23 DEBUG "metric": { 07:21:23 DEBUG "pod": "ds-idrepo-2" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG [...] 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG "pod": "ds-idrepo-0" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 1712816483, 07:21:23 DEBUG "0" 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG } 07:21:23 INFO [http_cmd]: http status code OK 07:21:23 DEBUG --- status code --- 07:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:21:23 DEBUG --- http response --- 07:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 07:21:23 DEBUG ----- output ----- 07:21:23 DEBUG { 07:21:23 DEBUG "status": "success", 07:21:23 DEBUG "data": { 07:21:23 DEBUG "resultType": "vector", 07:21:23 DEBUG "result": [ 07:21:23 DEBUG { 07:21:23 DEBUG "metric": { 07:21:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG [...] 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 1712816483, 07:21:23 DEBUG "1254704.6782552637" 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG } 07:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816483" 07:21:23 INFO [http_cmd]: http status code OK 07:21:23 DEBUG --- status code --- 07:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:21:23 DEBUG --- http response --- 07:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:21:23 DEBUG ----- output ----- 07:21:23 DEBUG { 07:21:23 DEBUG "status": "success", 07:21:23 DEBUG "data": { 07:21:23 DEBUG "resultType": "vector", 07:21:23 DEBUG "result": [ 07:21:23 DEBUG { 07:21:23 DEBUG "metric": { 07:21:23 DEBUG "pod": "ds-idrepo-2" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG [...] 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG "pod": "ds-cts-2" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 1712816483, 07:21:23 DEBUG "29.4" 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG } 07:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816483" 07:21:23 INFO [http_cmd]: http status code OK 07:21:23 DEBUG --- status code --- 07:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:21:23 DEBUG --- http response --- 07:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:21:23 DEBUG ----- output ----- 07:21:23 DEBUG { 07:21:23 DEBUG "status": "success", 07:21:23 DEBUG "data": { 07:21:23 DEBUG "resultType": "vector", 07:21:23 DEBUG "result": [ 07:21:23 DEBUG { 07:21:23 DEBUG "metric": { 07:21:23 DEBUG "instance": "10.142.0.107:9100" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG [...] 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG "instance": "10.142.0.77:9100" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 1712816483, 07:21:23 DEBUG "53004786688" 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG } 07:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816483" 07:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816483" 07:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816483" 07:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816483" 07:21:23 INFO [http_cmd]: http status code OK 07:21:23 INFO [http_cmd]: http status code OK 07:21:23 DEBUG --- status code --- 07:21:23 INFO [http_cmd]: http status code OK 07:21:23 DEBUG --- status code --- 07:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:21:23 DEBUG --- status code --- 07:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:21:23 DEBUG --- http response --- 07:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:21:23 DEBUG --- http response --- 07:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:21:23 DEBUG --- http response --- 07:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:21:23 DEBUG ----- output ----- 07:21:23 DEBUG ----- output ----- 07:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 07:21:23 DEBUG { 07:21:23 DEBUG { 07:21:23 DEBUG ----- output ----- 07:21:23 DEBUG "status": "success", 07:21:23 DEBUG "status": "success", 07:21:23 DEBUG { 07:21:23 DEBUG "data": { 07:21:23 DEBUG "data": { 07:21:23 DEBUG "status": "success", 07:21:23 DEBUG "resultType": "vector", 07:21:23 DEBUG "resultType": "vector", 07:21:23 DEBUG "data": { 07:21:23 DEBUG "result": [ 07:21:23 DEBUG "result": [ 07:21:23 DEBUG "resultType": "vector", 07:21:23 DEBUG { 07:21:23 DEBUG { 07:21:23 DEBUG "result": [ 07:21:23 DEBUG "metric": { 07:21:23 DEBUG "metric": { 07:21:23 DEBUG { 07:21:23 DEBUG "pod": "ds-idrepo-2" 07:21:23 DEBUG "pod": "ds-idrepo-2" 07:21:23 DEBUG "metric": { 07:21:23 DEBUG }, 07:21:23 DEBUG }, 07:21:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:21:23 DEBUG "value": [ 07:21:23 DEBUG "value": [ 07:21:23 DEBUG }, 07:21:23 INFO [http_cmd]: http status code OK 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG "value": [ 07:21:23 DEBUG --- status code --- 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG --- http response --- 07:21:23 DEBUG [...] 07:21:23 DEBUG [...] 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:21:23 DEBUG 07:21:23 DEBUG [...] 07:21:23 DEBUG 07:21:23 DEBUG ----- output ----- 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG { 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG "pod": "ds-idrepo-0" 07:21:23 DEBUG "status": "success", 07:21:23 DEBUG "pod": "ds-cts-2" 07:21:23 DEBUG 07:21:23 DEBUG }, 07:21:23 DEBUG "data": { 07:21:23 DEBUG }, 07:21:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:21:23 DEBUG "value": [ 07:21:23 DEBUG "resultType": "vector", 07:21:23 DEBUG "value": [ 07:21:23 DEBUG }, 07:21:23 DEBUG 1712816483, 07:21:23 DEBUG "result": [ 07:21:23 DEBUG 1712816483, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG "0" 07:21:23 DEBUG { 07:21:23 DEBUG "0" 07:21:23 DEBUG 1712816483, 07:21:23 DEBUG ] 07:21:23 DEBUG "metric": { 07:21:23 DEBUG ] 07:21:23 DEBUG "4221.479905620237" 07:21:23 DEBUG } 07:21:23 DEBUG "instance": "10.142.0.107:9100" 07:21:23 DEBUG } 07:21:23 DEBUG ] 07:21:23 DEBUG ] 07:21:23 DEBUG }, 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG } 07:21:23 DEBUG "value": [ 07:21:23 DEBUG } 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG 07:21:23 DEBUG } 07:21:23 DEBUG } 07:21:23 DEBUG 07:21:23 DEBUG } 07:21:23 DEBUG 07:21:23 DEBUG [...] 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG "instance": "10.142.0.77:9100" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 1712816483, 07:21:23 DEBUG "3562070.2666666666" 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG } 07:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816483" 07:21:23 INFO [http_cmd]: http status code OK 07:21:23 DEBUG --- status code --- 07:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:21:23 DEBUG --- http response --- 07:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:21:23 DEBUG ----- output ----- 07:21:23 DEBUG { 07:21:23 DEBUG "status": "success", 07:21:23 DEBUG "data": { 07:21:23 DEBUG "resultType": "vector", 07:21:23 DEBUG "result": [ 07:21:23 DEBUG { 07:21:23 DEBUG "metric": { 07:21:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG [...] 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 1712816483, 07:21:23 DEBUG "0" 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG } 07:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816483" 07:21:23 INFO [http_cmd]: http status code OK 07:21:23 DEBUG --- status code --- 07:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:21:23 DEBUG --- http response --- 07:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:21:23 DEBUG ----- output ----- 07:21:23 DEBUG { 07:21:23 DEBUG "status": "success", 07:21:23 DEBUG "data": { 07:21:23 DEBUG "resultType": "vector", 07:21:23 DEBUG "result": [ 07:21:23 DEBUG { 07:21:23 DEBUG "metric": { 07:21:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG [...] 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 1712816483, 07:21:23 DEBUG "0" 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG } 07:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816483" 07:21:23 INFO [http_cmd]: http status code OK 07:21:23 DEBUG --- status code --- 07:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:21:23 DEBUG --- http response --- 07:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:21:23 DEBUG ----- output ----- 07:21:23 DEBUG { 07:21:23 DEBUG "status": "success", 07:21:23 DEBUG "data": { 07:21:23 DEBUG "resultType": "vector", 07:21:23 DEBUG "result": [ 07:21:23 DEBUG { 07:21:23 DEBUG "metric": { 07:21:23 DEBUG "pod": "ds-idrepo-2" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG [...] 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG "pod": "ds-cts-2" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 1712816483, 07:21:23 DEBUG "0.00023333333333333333" 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG } 07:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816483" 07:21:23 INFO [http_cmd]: http status code OK 07:21:23 DEBUG --- status code --- 07:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:21:23 DEBUG --- http response --- 07:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:21:23 DEBUG ----- output ----- 07:21:23 DEBUG { 07:21:23 DEBUG "status": "success", 07:21:23 DEBUG "data": { 07:21:23 DEBUG "resultType": "vector", 07:21:23 DEBUG "result": [ 07:21:23 DEBUG { 07:21:23 DEBUG "metric": { 07:21:23 DEBUG "instance": "10.142.0.107:9100" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG [...] 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG "instance": "10.142.0.77:9100" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 1712816483, 07:21:23 DEBUG "0.0006666666666666524" 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG } 07:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816483" 07:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712816483" 07:21:23 INFO [http_cmd]: http status code OK 07:21:23 DEBUG --- status code --- 07:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:21:23 DEBUG --- http response --- 07:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:21:23 DEBUG ----- output ----- 07:21:23 DEBUG { 07:21:23 DEBUG "status": "success", 07:21:23 DEBUG "data": { 07:21:23 DEBUG "resultType": "vector", 07:21:23 DEBUG "result": [ 07:21:23 DEBUG { 07:21:23 DEBUG "metric": { 07:21:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG [...] 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 1712816483, 07:21:23 DEBUG "0" 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG } 07:21:23 INFO [http_cmd]: http status code OK 07:21:23 DEBUG --- status code --- 07:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:21:23 DEBUG --- http response --- 07:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:21:23 DEBUG ----- output ----- 07:21:23 DEBUG { 07:21:23 DEBUG "status": "success", 07:21:23 DEBUG "data": { 07:21:23 DEBUG "resultType": "vector", 07:21:23 DEBUG "result": [ 07:21:23 DEBUG { 07:21:23 DEBUG "metric": { 07:21:23 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG [...] 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 1712816483, 07:21:23 DEBUG "12923891712" 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG } 07:21:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816483" 07:21:23 INFO [http_cmd]: http status code OK 07:21:23 DEBUG --- status code --- 07:21:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:21:23 DEBUG --- http response --- 07:21:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:21:23 DEBUG ----- output ----- 07:21:23 DEBUG { 07:21:23 DEBUG "status": "success", 07:21:23 DEBUG "data": { 07:21:23 DEBUG "resultType": "vector", 07:21:23 DEBUG "result": [ 07:21:23 DEBUG { 07:21:23 DEBUG "metric": { 07:21:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG [...] 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG 07:21:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:21:23 DEBUG }, 07:21:23 DEBUG "value": [ 07:21:23 DEBUG 1712816483, 07:21:23 DEBUG "0" 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG ] 07:21:23 DEBUG } 07:21:23 DEBUG } 07:21:36 INFO 07:21:36 INFO [loop_until]: kubectl --namespace=xlou top pods 07:21:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:21:36 INFO [loop_until]: OK (rc = 0) 07:21:36 DEBUG --- stdout --- 07:21:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 473m 8179Mi am-5b5bd8567b-8h2w4 904m 8669Mi am-5b5bd8567b-qlkw6 709m 8423Mi ds-cts-0 2485m 12242Mi ds-cts-1 1810m 13308Mi ds-cts-2 2287m 13437Mi ds-idrepo-0 1445m 5526Mi ds-idrepo-1 316m 5482Mi ds-idrepo-2 370m 5784Mi end-user-ui-58878d7668-gjq6d 1m 5Mi idm-668d898cfd-2vqdm 3m 1010Mi idm-668d898cfd-st2t7 4m 1136Mi lodemon-ccc964465-p7gdm 1m 68Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 593m 305Mi 07:21:36 DEBUG --- stderr --- 07:21:36 DEBUG 07:21:40 INFO 07:21:40 INFO [loop_until]: kubectl --namespace=xlou top node 07:21:40 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:21:40 INFO [loop_until]: OK (rc = 0) 07:21:40 DEBUG --- stdout --- 07:21:40 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 2742Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 68m 0% 2780Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 647m 2% 10126Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 120m 0% 3057Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 827m 2% 9843Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 146m 0% 3863Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 793m 2% 10355Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 466m 2% 6472Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 1195m 7% 6232Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 1814m 11% 14101Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 2309m 14% 12925Mi 22% gke-xlou-cdm-ds-32e4dcb1-j2hk 2148m 13% 14000Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 513m 3% 6175Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 607m 3% 4158Mi 7% 07:21:40 DEBUG --- stderr --- 07:21:40 DEBUG 07:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816543" 07:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712816543" 07:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816543" 07:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712816543" 07:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816543" 07:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816543" 07:22:23 INFO [http_cmd]: http status code OK 07:22:23 DEBUG --- status code --- 07:22:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:22:23 DEBUG --- http response --- 07:22:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:22:23 DEBUG ----- output ----- 07:22:23 DEBUG { 07:22:23 DEBUG "status": "success", 07:22:23 DEBUG "data": { 07:22:23 DEBUG "resultType": "vector", 07:22:23 DEBUG "result": [ 07:22:23 DEBUG { 07:22:23 DEBUG "metric": { 07:22:23 DEBUG "pod": "ds-idrepo-2" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG [...] 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG "pod": "ds-idrepo-0" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 1712816543, 07:22:23 DEBUG "0" 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG } 07:22:23 INFO [http_cmd]: http status code OK 07:22:23 DEBUG --- status code --- 07:22:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:22:23 DEBUG --- http response --- 07:22:23 ERROR [http_cmd]: ERROR 07:22:23 INFO [http_cmd]: http status code OK 07:22:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:22:23 ERROR --- status code --- 07:22:23 DEBUG --- status code --- 07:22:23 DEBUG ----- output ----- 07:22:23 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:22:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:22:23 DEBUG { 07:22:23 ERROR --- http response --- 07:22:23 DEBUG --- http response --- 07:22:23 DEBUG "status": "success", 07:22:23 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:22:23 DEBUG "data": { 07:22:23 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:22:23 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:22:23 DEBUG "resultType": "vector", 07:22:23 DEBUG "result": [ 07:22:23 DEBUG { 07:22:23 DEBUG "metric": { 07:22:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG [...] 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 1712816543, 07:22:23 DEBUG "0.026" 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG } 07:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816543" 07:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816543" 07:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712816543" 07:22:23 INFO [http_cmd]: http status code OK 07:22:23 INFO [http_cmd]: http status code OK 07:22:23 DEBUG --- status code --- 07:22:23 DEBUG --- status code --- 07:22:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:22:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:22:23 DEBUG --- http response --- 07:22:23 DEBUG --- http response --- 07:22:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:22:23 DEBUG ----- output ----- 07:22:23 DEBUG { 07:22:23 DEBUG "status": "success", 07:22:23 DEBUG "data": { 07:22:23 DEBUG "resultType": "vector", 07:22:23 DEBUG "result": [ 07:22:23 DEBUG { 07:22:23 DEBUG "metric": { 07:22:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG [...] 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG "pod": "ds-cts-0" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 1712816543, 07:22:23 DEBUG "0.016511502040125286" 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG } 07:22:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:22:23 DEBUG ----- output ----- 07:22:23 DEBUG { 07:22:23 DEBUG "status": "success", 07:22:23 DEBUG "data": { 07:22:23 DEBUG "resultType": "vector", 07:22:23 DEBUG "result": [ 07:22:23 DEBUG { 07:22:23 DEBUG "metric": { 07:22:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG [...] 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG "pod": "ds-cts-0" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 1712816543, 07:22:23 DEBUG "471994463.32585764" 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG } 07:22:23 INFO [http_cmd]: http status code OK 07:22:23 INFO [http_cmd]: http status code OK 07:22:23 DEBUG --- status code --- 07:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816543" 07:22:23 DEBUG --- status code --- 07:22:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:22:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:22:23 DEBUG --- http response --- 07:22:23 DEBUG --- http response --- 07:22:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:22:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:22:23 DEBUG ----- output ----- 07:22:23 DEBUG ----- output ----- 07:22:23 DEBUG { 07:22:23 DEBUG { 07:22:23 DEBUG "status": "success", 07:22:23 DEBUG "status": "success", 07:22:23 DEBUG "data": { 07:22:23 DEBUG "data": { 07:22:23 DEBUG "resultType": "vector", 07:22:23 DEBUG "resultType": "vector", 07:22:23 DEBUG "result": [ 07:22:23 DEBUG "result": [ 07:22:23 DEBUG { 07:22:23 DEBUG { 07:22:23 DEBUG "metric": { 07:22:23 DEBUG "metric": { 07:22:23 DEBUG "instance": "10.142.0.107:9100" 07:22:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:22:23 DEBUG }, 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG [...] 07:22:23 DEBUG [...] 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG "instance": "10.142.0.77:9100" 07:22:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:22:23 DEBUG }, 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 1712816543, 07:22:23 DEBUG 1712816543, 07:22:23 DEBUG "34308.53333333333" 07:22:23 DEBUG "0" 07:22:23 DEBUG ] 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG } 07:22:23 DEBUG ] 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG } 07:22:23 DEBUG } 07:22:23 DEBUG } 07:22:23 INFO [http_cmd]: http status code OK 07:22:23 DEBUG --- status code --- 07:22:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:22:23 DEBUG --- http response --- 07:22:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:22:23 DEBUG ----- output ----- 07:22:23 DEBUG { 07:22:23 DEBUG "status": "success", 07:22:23 DEBUG "data": { 07:22:23 DEBUG "resultType": "vector", 07:22:23 DEBUG "result": [ 07:22:23 DEBUG { 07:22:23 DEBUG "metric": { 07:22:23 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG [...] 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 1712816543, 07:22:23 DEBUG "1.0266927591705144" 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG } 07:22:23 INFO [http_cmd]: http status code OK 07:22:23 DEBUG --- status code --- 07:22:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:22:23 DEBUG --- http response --- 07:22:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:22:23 DEBUG ----- output ----- 07:22:23 DEBUG { 07:22:23 DEBUG "status": "success", 07:22:23 DEBUG "data": { 07:22:23 DEBUG "resultType": "vector", 07:22:23 DEBUG "result": [ 07:22:23 DEBUG { 07:22:23 DEBUG "metric": { 07:22:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG [...] 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 1712816543, 07:22:23 DEBUG "0" 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG } 07:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712816543" 07:22:23 INFO [http_cmd]: http status code OK 07:22:23 DEBUG --- status code --- 07:22:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:22:23 DEBUG --- http response --- 07:22:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:22:23 DEBUG ----- output ----- 07:22:23 DEBUG { 07:22:23 DEBUG "status": "success", 07:22:23 DEBUG "data": { 07:22:23 DEBUG "resultType": "vector", 07:22:23 DEBUG "result": [ 07:22:23 DEBUG { 07:22:23 DEBUG "metric": { 07:22:23 DEBUG "pod": "ds-idrepo-2" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG [...] 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG "pod": "ds-cts-2" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 1712816543, 07:22:23 DEBUG "0" 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG } 07:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816543" 07:22:23 INFO [http_cmd]: http status code OK 07:22:23 DEBUG --- status code --- 07:22:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:22:23 DEBUG --- http response --- 07:22:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:22:23 DEBUG ----- output ----- 07:22:23 DEBUG { 07:22:23 DEBUG "status": "success", 07:22:23 DEBUG "data": { 07:22:23 DEBUG "resultType": "vector", 07:22:23 DEBUG "result": [ 07:22:23 DEBUG { 07:22:23 DEBUG "metric": { 07:22:23 DEBUG "instance": "10.142.0.107:9100" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG [...] 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG "instance": "10.142.0.77:9100" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 1712816543, 07:22:23 DEBUG "0.0026333333333331175" 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG } 07:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816543" 07:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816543" 07:22:23 INFO [http_cmd]: http status code OK 07:22:23 DEBUG --- status code --- 07:22:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:22:23 DEBUG --- http response --- 07:22:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:22:23 DEBUG ----- output ----- 07:22:23 DEBUG { 07:22:23 DEBUG "status": "success", 07:22:23 DEBUG "data": { 07:22:23 DEBUG "resultType": "vector", 07:22:23 DEBUG "result": [ 07:22:23 DEBUG { 07:22:23 DEBUG "metric": { 07:22:23 DEBUG "pod": "ds-idrepo-2" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG [...] 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG "pod": "ds-idrepo-0" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 1712816543, 07:22:23 DEBUG "0" 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG } 07:22:23 INFO [http_cmd]: http status code OK 07:22:23 DEBUG --- status code --- 07:22:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:22:23 DEBUG --- http response --- 07:22:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:22:23 DEBUG ----- output ----- 07:22:23 DEBUG { 07:22:23 DEBUG "status": "success", 07:22:23 DEBUG "data": { 07:22:23 DEBUG "resultType": "vector", 07:22:23 DEBUG "result": [ 07:22:23 DEBUG { 07:22:23 DEBUG "metric": { 07:22:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG [...] 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG "pod": "ds-cts-0" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 1712816543, 07:22:23 DEBUG "0" 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG } 07:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816543" 07:22:23 INFO [http_cmd]: http status code OK 07:22:23 DEBUG --- status code --- 07:22:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:22:23 DEBUG --- http response --- 07:22:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:22:23 DEBUG ----- output ----- 07:22:23 DEBUG { 07:22:23 DEBUG "status": "success", 07:22:23 DEBUG "data": { 07:22:23 DEBUG "resultType": "vector", 07:22:23 DEBUG "result": [ 07:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816543" 07:22:23 DEBUG { 07:22:23 DEBUG "metric": { 07:22:23 DEBUG "instance": "10.142.0.107:9100" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG [...] 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG "instance": "10.142.0.77:9100" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 1712816543, 07:22:23 DEBUG "53006736384" 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG } 07:22:23 INFO [http_cmd]: http status code OK 07:22:23 DEBUG --- status code --- 07:22:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:22:23 DEBUG --- http response --- 07:22:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:22:23 DEBUG ----- output ----- 07:22:23 DEBUG { 07:22:23 DEBUG "status": "success", 07:22:23 DEBUG "data": { 07:22:23 DEBUG "resultType": "vector", 07:22:23 DEBUG "result": [ 07:22:23 DEBUG { 07:22:23 DEBUG "metric": { 07:22:23 DEBUG "pod": "ds-idrepo-2" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG [...] 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG "pod": "ds-cts-2" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 1712816543, 07:22:23 DEBUG "29.4" 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG } 07:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816543" 07:22:23 INFO [http_cmd]: http status code OK 07:22:23 DEBUG --- status code --- 07:22:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:22:23 DEBUG --- http response --- 07:22:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:22:23 DEBUG ----- output ----- 07:22:23 DEBUG { 07:22:23 DEBUG "status": "success", 07:22:23 DEBUG "data": { 07:22:23 DEBUG "resultType": "vector", 07:22:23 DEBUG "result": [ 07:22:23 DEBUG { 07:22:23 DEBUG "metric": { 07:22:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG [...] 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG "pod": "ds-cts-0" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 1712816543, 07:22:23 DEBUG "16113.533585793597" 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG } 07:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816543" 07:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816543" 07:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816543" 07:22:23 INFO [http_cmd]: http status code OK 07:22:23 INFO [http_cmd]: http status code OK 07:22:23 DEBUG --- status code --- 07:22:23 INFO [http_cmd]: http status code OK 07:22:23 DEBUG --- status code --- 07:22:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:22:23 DEBUG --- status code --- 07:22:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:22:23 DEBUG --- http response --- 07:22:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:22:23 DEBUG --- http response --- 07:22:23 DEBUG --- http response --- 07:22:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:22:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:22:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:22:23 DEBUG ----- output ----- 07:22:23 DEBUG ----- output ----- 07:22:23 DEBUG ----- output ----- 07:22:23 DEBUG { 07:22:23 DEBUG { 07:22:23 DEBUG { 07:22:23 DEBUG "status": "success", 07:22:23 DEBUG "status": "success", 07:22:23 DEBUG "status": "success", 07:22:23 DEBUG "data": { 07:22:23 DEBUG "data": { 07:22:23 DEBUG "data": { 07:22:23 DEBUG "resultType": "vector", 07:22:23 DEBUG "resultType": "vector", 07:22:23 DEBUG "resultType": "vector", 07:22:23 DEBUG "result": [ 07:22:23 DEBUG "result": [ 07:22:23 DEBUG "result": [ 07:22:23 DEBUG { 07:22:23 DEBUG { 07:22:23 DEBUG { 07:22:23 DEBUG "metric": { 07:22:23 DEBUG "metric": { 07:22:23 DEBUG "metric": { 07:22:23 DEBUG "pod": "ds-idrepo-2" 07:22:23 DEBUG "pod": "ds-idrepo-2" 07:22:23 DEBUG "instance": "10.142.0.107:9100" 07:22:23 DEBUG }, 07:22:23 DEBUG }, 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG "value": [ 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG [...] 07:22:23 DEBUG [...] 07:22:23 DEBUG [...] 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG "pod": "ds-idrepo-0" 07:22:23 DEBUG "pod": "ds-cts-2" 07:22:23 DEBUG "instance": "10.142.0.77:9100" 07:22:23 DEBUG }, 07:22:23 DEBUG }, 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG "value": [ 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 1712816543, 07:22:23 DEBUG 1712816543, 07:22:23 DEBUG 1712816543, 07:22:23 DEBUG "0" 07:22:23 DEBUG "0" 07:22:23 DEBUG "35114.33333333333" 07:22:23 DEBUG ] 07:22:23 DEBUG ] 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG } 07:22:23 DEBUG } 07:22:23 DEBUG ] 07:22:23 DEBUG ] 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG } 07:22:23 DEBUG } 07:22:23 DEBUG } 07:22:23 DEBUG } 07:22:23 DEBUG } 07:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816543" 07:22:23 INFO [http_cmd]: http status code OK 07:22:23 DEBUG --- status code --- 07:22:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:22:23 DEBUG --- http response --- 07:22:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:22:23 DEBUG ----- output ----- 07:22:23 DEBUG { 07:22:23 DEBUG "status": "success", 07:22:23 DEBUG "data": { 07:22:23 DEBUG "resultType": "vector", 07:22:23 DEBUG "result": [ 07:22:23 DEBUG { 07:22:23 DEBUG "metric": { 07:22:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG [...] 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 1712816543, 07:22:23 DEBUG "0" 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG } 07:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816543" 07:22:23 INFO [http_cmd]: http status code OK 07:22:23 DEBUG --- status code --- 07:22:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:22:23 DEBUG --- http response --- 07:22:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:22:23 DEBUG ----- output ----- 07:22:23 DEBUG { 07:22:23 DEBUG "status": "success", 07:22:23 DEBUG "data": { 07:22:23 DEBUG "resultType": "vector", 07:22:23 DEBUG "result": [ 07:22:23 DEBUG { 07:22:23 DEBUG "metric": { 07:22:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG [...] 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 1712816543, 07:22:23 DEBUG "0" 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG } 07:22:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816543" 07:22:23 INFO [http_cmd]: http status code OK 07:22:23 DEBUG --- status code --- 07:22:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:22:23 DEBUG --- http response --- 07:22:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:22:23 DEBUG ----- output ----- 07:22:23 DEBUG { 07:22:23 DEBUG "status": "success", 07:22:23 DEBUG "data": { 07:22:23 DEBUG "resultType": "vector", 07:22:23 DEBUG "result": [ 07:22:23 DEBUG { 07:22:23 DEBUG "metric": { 07:22:23 DEBUG "pod": "ds-idrepo-2" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG [...] 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG 07:22:23 DEBUG "pod": "ds-cts-2" 07:22:23 DEBUG }, 07:22:23 DEBUG "value": [ 07:22:23 DEBUG 1712816543, 07:22:23 DEBUG "0" 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG ] 07:22:23 DEBUG } 07:22:23 DEBUG } 07:22:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816544" 07:22:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816544" 07:22:24 INFO [http_cmd]: http status code OK 07:22:24 DEBUG --- status code --- 07:22:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:22:24 DEBUG --- http response --- 07:22:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:22:24 DEBUG ----- output ----- 07:22:24 DEBUG { 07:22:24 DEBUG "status": "success", 07:22:24 DEBUG "data": { 07:22:24 DEBUG "resultType": "vector", 07:22:24 DEBUG "result": [ 07:22:24 DEBUG { 07:22:24 DEBUG "metric": { 07:22:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:22:24 DEBUG }, 07:22:24 DEBUG "value": [ 07:22:24 DEBUG 07:22:24 DEBUG 07:22:24 DEBUG 07:22:24 DEBUG [...] 07:22:24 DEBUG 07:22:24 DEBUG 07:22:24 DEBUG 07:22:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:22:24 DEBUG }, 07:22:24 DEBUG "value": [ 07:22:24 DEBUG 1712816544, 07:22:24 DEBUG "0" 07:22:24 DEBUG ] 07:22:24 DEBUG } 07:22:24 DEBUG ] 07:22:24 DEBUG } 07:22:24 DEBUG } 07:22:24 INFO [http_cmd]: http status code OK 07:22:24 DEBUG --- status code --- 07:22:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:22:24 DEBUG --- http response --- 07:22:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:22:24 DEBUG ----- output ----- 07:22:24 DEBUG { 07:22:24 DEBUG "status": "success", 07:22:24 DEBUG "data": { 07:22:24 DEBUG "resultType": "vector", 07:22:24 DEBUG "result": [ 07:22:24 DEBUG { 07:22:24 DEBUG "metric": { 07:22:24 DEBUG "instance": "10.142.0.107:9100" 07:22:24 DEBUG }, 07:22:24 DEBUG "value": [ 07:22:24 DEBUG 07:22:24 DEBUG 07:22:24 DEBUG 07:22:24 DEBUG [...] 07:22:24 DEBUG 07:22:24 DEBUG 07:22:24 DEBUG 07:22:24 DEBUG "instance": "10.142.0.77:9100" 07:22:24 DEBUG }, 07:22:24 DEBUG "value": [ 07:22:24 DEBUG 1712816544, 07:22:24 DEBUG "0.000333333333333267" 07:22:24 DEBUG ] 07:22:24 DEBUG } 07:22:24 DEBUG ] 07:22:24 DEBUG } 07:22:24 DEBUG } 07:22:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816544" 07:22:24 INFO [http_cmd]: http status code OK 07:22:24 DEBUG --- status code --- 07:22:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:22:24 DEBUG --- http response --- 07:22:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:22:24 DEBUG ----- output ----- 07:22:24 DEBUG { 07:22:24 DEBUG "status": "success", 07:22:24 DEBUG "data": { 07:22:24 DEBUG "resultType": "vector", 07:22:24 DEBUG "result": [ 07:22:24 DEBUG { 07:22:24 DEBUG "metric": { 07:22:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:22:24 DEBUG }, 07:22:24 DEBUG "value": [ 07:22:24 DEBUG 07:22:24 DEBUG 07:22:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712816544" 07:22:24 DEBUG 07:22:24 DEBUG [...] 07:22:24 DEBUG 07:22:24 DEBUG 07:22:24 DEBUG 07:22:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:22:24 DEBUG }, 07:22:24 DEBUG "value": [ 07:22:24 DEBUG 1712816544, 07:22:24 DEBUG "0" 07:22:24 DEBUG ] 07:22:24 DEBUG } 07:22:24 DEBUG ] 07:22:24 DEBUG } 07:22:24 DEBUG } 07:22:24 INFO [http_cmd]: http status code OK 07:22:24 DEBUG --- status code --- 07:22:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:22:24 DEBUG --- http response --- 07:22:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:22:24 DEBUG ----- output ----- 07:22:24 DEBUG { 07:22:24 DEBUG "status": "success", 07:22:24 DEBUG "data": { 07:22:24 DEBUG "resultType": "vector", 07:22:24 DEBUG "result": [ 07:22:24 DEBUG { 07:22:24 DEBUG "metric": { 07:22:24 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:22:24 DEBUG }, 07:22:24 DEBUG "value": [ 07:22:24 DEBUG 07:22:24 DEBUG 07:22:24 DEBUG 07:22:24 DEBUG [...] 07:22:24 DEBUG 07:22:24 DEBUG 07:22:24 DEBUG 07:22:24 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 07:22:24 DEBUG }, 07:22:24 DEBUG "value": [ 07:22:24 DEBUG 1712816544, 07:22:24 DEBUG "12929097728" 07:22:24 DEBUG ] 07:22:24 DEBUG } 07:22:24 DEBUG ] 07:22:24 DEBUG } 07:22:24 DEBUG } 07:22:36 INFO 07:22:36 INFO [loop_until]: kubectl --namespace=xlou top pods 07:22:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:22:36 INFO [loop_until]: OK (rc = 0) 07:22:36 DEBUG --- stdout --- 07:22:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 9m 8166Mi am-5b5bd8567b-8h2w4 7m 8661Mi am-5b5bd8567b-qlkw6 7m 8416Mi ds-cts-0 15m 12242Mi ds-cts-1 21m 13308Mi ds-cts-2 20m 13437Mi ds-idrepo-0 13m 5526Mi ds-idrepo-1 17m 5483Mi ds-idrepo-2 15m 5784Mi end-user-ui-58878d7668-gjq6d 1m 5Mi idm-668d898cfd-2vqdm 3m 1010Mi idm-668d898cfd-st2t7 4m 1136Mi lodemon-ccc964465-p7gdm 6m 68Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 1010m 703Mi 07:22:36 DEBUG --- stderr --- 07:22:36 DEBUG 07:22:40 INFO 07:22:40 INFO [loop_until]: kubectl --namespace=xlou top node 07:22:40 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:22:40 INFO [loop_until]: OK (rc = 0) 07:22:40 DEBUG --- stdout --- 07:22:40 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 2733Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 63m 0% 2774Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 54m 0% 10118Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 117m 0% 3061Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 65m 0% 9828Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 140m 0% 3851Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 67m 0% 10332Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 65m 0% 6473Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 64m 0% 6235Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 80m 0% 14101Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 74m 0% 12926Mi 22% gke-xlou-cdm-ds-32e4dcb1-j2hk 74m 0% 14000Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 69m 0% 6176Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 1079m 6% 4525Mi 7% 07:22:40 DEBUG --- stderr --- 07:22:40 DEBUG 07:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816603" 07:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816603" 07:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816603" 07:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712816603" 07:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712816603" 07:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816603" 07:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816603" 07:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816603" 07:23:23 ERROR [http_cmd]: ERROR 07:23:23 INFO [http_cmd]: http status code OK 07:23:23 INFO [http_cmd]: http status code OK 07:23:23 ERROR --- status code --- 07:23:23 INFO [http_cmd]: http status code OK 07:23:23 DEBUG --- status code --- 07:23:23 DEBUG --- status code --- 07:23:23 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:23:23 DEBUG --- status code --- 07:23:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:23:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:23:23 ERROR --- http response --- 07:23:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:23:23 DEBUG --- http response --- 07:23:23 DEBUG --- http response --- 07:23:23 DEBUG --- http response --- 07:23:23 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:23:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:23:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:23:23 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:23:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:23:23 DEBUG ----- output ----- 07:23:23 DEBUG ----- output ----- 07:23:23 DEBUG ----- output ----- 07:23:23 DEBUG { 07:23:23 DEBUG { 07:23:23 DEBUG { 07:23:23 DEBUG "status": "success", 07:23:23 DEBUG "status": "success", 07:23:23 DEBUG "status": "success", 07:23:23 DEBUG "data": { 07:23:23 DEBUG "data": { 07:23:23 DEBUG "data": { 07:23:23 DEBUG "resultType": "vector", 07:23:23 DEBUG "resultType": "vector", 07:23:23 DEBUG "resultType": "vector", 07:23:23 DEBUG "result": [ 07:23:23 DEBUG "result": [ 07:23:23 DEBUG "result": [ 07:23:23 DEBUG { 07:23:23 DEBUG { 07:23:23 DEBUG { 07:23:23 DEBUG "metric": { 07:23:23 DEBUG "metric": { 07:23:23 DEBUG "metric": { 07:23:23 DEBUG "pod": "ds-idrepo-2" 07:23:23 DEBUG "instance": "10.142.0.107:9100" 07:23:23 DEBUG "instance": "10.142.0.107:9100" 07:23:23 DEBUG }, 07:23:23 DEBUG }, 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG "value": [ 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG [...] 07:23:23 DEBUG [...] 07:23:23 DEBUG [...] 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG "pod": "ds-idrepo-0" 07:23:23 DEBUG "instance": "10.142.0.77:9100" 07:23:23 DEBUG "instance": "10.142.0.77:9100" 07:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816603" 07:23:23 DEBUG }, 07:23:23 DEBUG }, 07:23:23 INFO [http_cmd]: http status code OK 07:23:23 INFO [http_cmd]: http status code OK 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG "value": [ 07:23:23 DEBUG --- status code --- 07:23:23 DEBUG --- status code --- 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 1712816603, 07:23:23 DEBUG 1712816603, 07:23:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:23:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:23:23 DEBUG 1712816603, 07:23:23 DEBUG "0" 07:23:23 DEBUG "0.002633333333333828" 07:23:23 DEBUG --- http response --- 07:23:23 DEBUG --- http response --- 07:23:23 DEBUG "34581.933333333334" 07:23:23 DEBUG ] 07:23:23 DEBUG ] 07:23:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:23:23 DEBUG ] 07:23:23 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:23:23 DEBUG } 07:23:23 DEBUG } 07:23:23 DEBUG ----- output ----- 07:23:23 DEBUG } 07:23:23 DEBUG ] 07:23:23 DEBUG ] 07:23:23 DEBUG { 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG } 07:23:23 DEBUG "status": "success", 07:23:23 DEBUG } 07:23:23 DEBUG } 07:23:23 DEBUG } 07:23:23 DEBUG "data": { 07:23:23 DEBUG } 07:23:23 DEBUG "resultType": "vector", 07:23:23 DEBUG "result": [ 07:23:23 DEBUG { 07:23:23 DEBUG "metric": { 07:23:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG [...] 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 1712816603, 07:23:23 DEBUG "0.026" 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG } 07:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712816603" 07:23:23 INFO [http_cmd]: http status code OK 07:23:23 INFO [http_cmd]: http status code OK 07:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816603" 07:23:23 DEBUG --- status code --- 07:23:23 DEBUG --- status code --- 07:23:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:23:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:23:23 DEBUG --- http response --- 07:23:23 DEBUG --- http response --- 07:23:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:23:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:23:23 DEBUG ----- output ----- 07:23:23 DEBUG ----- output ----- 07:23:23 DEBUG { 07:23:23 DEBUG { 07:23:23 DEBUG "status": "success", 07:23:23 DEBUG "status": "success", 07:23:23 DEBUG "data": { 07:23:23 DEBUG "data": { 07:23:23 DEBUG "resultType": "vector", 07:23:23 DEBUG "resultType": "vector", 07:23:23 DEBUG "result": [ 07:23:23 DEBUG "result": [ 07:23:23 DEBUG { 07:23:23 DEBUG { 07:23:23 DEBUG "metric": { 07:23:23 DEBUG "metric": { 07:23:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:23:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:23:23 DEBUG }, 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG [...] 07:23:23 DEBUG [...] 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG "pod": "ds-cts-1" 07:23:23 DEBUG "pod": "ds-cts-1" 07:23:23 DEBUG }, 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 1712816603, 07:23:23 DEBUG 1712816603, 07:23:23 DEBUG "0" 07:23:23 DEBUG "0" 07:23:23 DEBUG ] 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG } 07:23:23 DEBUG ] 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG } 07:23:23 DEBUG } 07:23:23 DEBUG } 07:23:23 INFO [http_cmd]: http status code OK 07:23:23 DEBUG --- status code --- 07:23:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:23:23 DEBUG --- http response --- 07:23:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:23:23 DEBUG ----- output ----- 07:23:23 DEBUG { 07:23:23 DEBUG "status": "success", 07:23:23 DEBUG "data": { 07:23:23 DEBUG "resultType": "vector", 07:23:23 DEBUG "result": [ 07:23:23 DEBUG { 07:23:23 DEBUG "metric": { 07:23:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG [...] 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 1712816603, 07:23:23 DEBUG "0" 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG } 07:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712816603" 07:23:23 INFO [http_cmd]: http status code OK 07:23:23 INFO [http_cmd]: http status code OK 07:23:23 DEBUG --- status code --- 07:23:23 DEBUG --- status code --- 07:23:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:23:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:23:23 DEBUG --- http response --- 07:23:23 DEBUG --- http response --- 07:23:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:23:23 DEBUG ----- output ----- 07:23:23 DEBUG { 07:23:23 DEBUG "status": "success", 07:23:23 DEBUG "data": { 07:23:23 DEBUG "resultType": "vector", 07:23:23 DEBUG "result": [ 07:23:23 DEBUG { 07:23:23 DEBUG "metric": { 07:23:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG [...] 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 1712816603, 07:23:23 DEBUG "0" 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG } 07:23:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:23:23 DEBUG ----- output ----- 07:23:23 DEBUG { 07:23:23 DEBUG "status": "success", 07:23:23 DEBUG "data": { 07:23:23 DEBUG "resultType": "vector", 07:23:23 DEBUG "result": [ 07:23:23 DEBUG { 07:23:23 DEBUG "metric": { 07:23:23 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG [...] 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 1712816603, 07:23:23 DEBUG "1.0340473421694867" 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG } 07:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816603" 07:23:23 INFO [http_cmd]: http status code OK 07:23:23 DEBUG --- status code --- 07:23:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:23:23 DEBUG --- http response --- 07:23:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:23:23 DEBUG ----- output ----- 07:23:23 DEBUG { 07:23:23 DEBUG "status": "success", 07:23:23 DEBUG "data": { 07:23:23 DEBUG "resultType": "vector", 07:23:23 DEBUG "result": [ 07:23:23 DEBUG { 07:23:23 DEBUG "metric": { 07:23:23 DEBUG "pod": "ds-idrepo-2" 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG [...] 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG "pod": "ds-cts-2" 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 1712816603, 07:23:23 DEBUG "0" 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG } 07:23:23 INFO [http_cmd]: http status code OK 07:23:23 DEBUG --- status code --- 07:23:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:23:23 DEBUG --- http response --- 07:23:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:23:23 DEBUG ----- output ----- 07:23:23 DEBUG { 07:23:23 DEBUG "status": "success", 07:23:23 DEBUG "data": { 07:23:23 DEBUG "resultType": "vector", 07:23:23 DEBUG "result": [ 07:23:23 DEBUG { 07:23:23 DEBUG "metric": { 07:23:23 DEBUG "pod": "ds-idrepo-2" 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG [...] 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG "pod": "ds-idrepo-0" 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 1712816603, 07:23:23 DEBUG "0" 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG } 07:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816603" 07:23:23 INFO [http_cmd]: http status code OK 07:23:23 DEBUG --- status code --- 07:23:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:23:23 DEBUG --- http response --- 07:23:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:23:23 DEBUG ----- output ----- 07:23:23 DEBUG { 07:23:23 DEBUG "status": "success", 07:23:23 DEBUG "data": { 07:23:23 DEBUG "resultType": "vector", 07:23:23 DEBUG "result": [ 07:23:23 DEBUG { 07:23:23 DEBUG "metric": { 07:23:23 DEBUG "instance": "10.142.0.107:9100" 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG [...] 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG "instance": "10.142.0.77:9100" 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 1712816603, 07:23:23 DEBUG "53008303104" 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG } 07:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816603" 07:23:23 INFO [http_cmd]: http status code OK 07:23:23 DEBUG --- status code --- 07:23:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:23:23 DEBUG --- http response --- 07:23:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:23:23 DEBUG ----- output ----- 07:23:23 DEBUG { 07:23:23 DEBUG "status": "success", 07:23:23 DEBUG "data": { 07:23:23 DEBUG "resultType": "vector", 07:23:23 DEBUG "result": [ 07:23:23 DEBUG { 07:23:23 DEBUG "metric": { 07:23:23 DEBUG "pod": "ds-idrepo-2" 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG [...] 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG "pod": "ds-cts-2" 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 1712816603, 07:23:23 DEBUG "29.766666666666666" 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG } 07:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816603" 07:23:23 INFO [http_cmd]: http status code OK 07:23:23 DEBUG --- status code --- 07:23:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:23:23 DEBUG --- http response --- 07:23:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 07:23:23 DEBUG ----- output ----- 07:23:23 DEBUG { 07:23:23 DEBUG "status": "success", 07:23:23 DEBUG "data": { 07:23:23 DEBUG "resultType": "vector", 07:23:23 DEBUG "result": [ 07:23:23 DEBUG { 07:23:23 DEBUG "metric": { 07:23:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG [...] 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 1712816603, 07:23:23 DEBUG "0" 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG } 07:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816603" 07:23:23 INFO [http_cmd]: http status code OK 07:23:23 DEBUG --- status code --- 07:23:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:23:23 DEBUG --- http response --- 07:23:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 07:23:23 DEBUG ----- output ----- 07:23:23 DEBUG { 07:23:23 DEBUG "status": "success", 07:23:23 DEBUG "data": { 07:23:23 DEBUG "resultType": "vector", 07:23:23 DEBUG "result": [ 07:23:23 DEBUG { 07:23:23 DEBUG "metric": { 07:23:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG [...] 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 1712816603, 07:23:23 DEBUG "0" 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG } 07:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816603" 07:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816603" 07:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816603" 07:23:23 INFO [http_cmd]: http status code OK 07:23:23 INFO [http_cmd]: http status code OK 07:23:23 INFO [http_cmd]: http status code OK 07:23:23 DEBUG --- status code --- 07:23:23 DEBUG --- status code --- 07:23:23 DEBUG --- status code --- 07:23:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:23:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:23:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:23:23 DEBUG --- http response --- 07:23:23 DEBUG --- http response --- 07:23:23 DEBUG --- http response --- 07:23:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:23:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:23:23 DEBUG ----- output ----- 07:23:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:23:23 DEBUG ----- output ----- 07:23:23 DEBUG { 07:23:23 DEBUG ----- output ----- 07:23:23 DEBUG { 07:23:23 DEBUG "status": "success", 07:23:23 DEBUG { 07:23:23 DEBUG "status": "success", 07:23:23 DEBUG "data": { 07:23:23 DEBUG "status": "success", 07:23:23 DEBUG "data": { 07:23:23 DEBUG "resultType": "vector", 07:23:23 DEBUG "data": { 07:23:23 DEBUG "resultType": "vector", 07:23:23 DEBUG "result": [ 07:23:23 DEBUG "resultType": "vector", 07:23:23 DEBUG "result": [ 07:23:23 DEBUG { 07:23:23 DEBUG "result": [ 07:23:23 DEBUG { 07:23:23 DEBUG "metric": { 07:23:23 DEBUG { 07:23:23 DEBUG "metric": { 07:23:23 DEBUG "pod": "ds-idrepo-2" 07:23:23 DEBUG "metric": { 07:23:23 DEBUG "pod": "ds-idrepo-2" 07:23:23 DEBUG }, 07:23:23 DEBUG "instance": "10.142.0.107:9100" 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG [...] 07:23:23 DEBUG 07:23:23 DEBUG [...] 07:23:23 DEBUG 07:23:23 DEBUG [...] 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG "pod": "ds-idrepo-0" 07:23:23 DEBUG 07:23:23 DEBUG "pod": "ds-cts-2" 07:23:23 DEBUG }, 07:23:23 DEBUG "instance": "10.142.0.77:9100" 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 1712816603, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 1712816603, 07:23:23 DEBUG "0" 07:23:23 DEBUG 1712816603, 07:23:23 DEBUG "0" 07:23:23 DEBUG ] 07:23:23 DEBUG "35272.6" 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG } 07:23:23 DEBUG } 07:23:23 DEBUG } 07:23:23 DEBUG } 07:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816603" 07:23:23 INFO [http_cmd]: http status code OK 07:23:23 DEBUG --- status code --- 07:23:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:23:23 DEBUG --- http response --- 07:23:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:23:23 DEBUG ----- output ----- 07:23:23 DEBUG { 07:23:23 DEBUG "status": "success", 07:23:23 DEBUG "data": { 07:23:23 DEBUG "resultType": "vector", 07:23:23 DEBUG "result": [ 07:23:23 DEBUG { 07:23:23 DEBUG "metric": { 07:23:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG [...] 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 1712816603, 07:23:23 DEBUG "0" 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG } 07:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816603" 07:23:23 INFO [http_cmd]: http status code OK 07:23:23 DEBUG --- status code --- 07:23:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:23:23 DEBUG --- http response --- 07:23:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:23:23 DEBUG ----- output ----- 07:23:23 DEBUG { 07:23:23 DEBUG "status": "success", 07:23:23 DEBUG "data": { 07:23:23 DEBUG "resultType": "vector", 07:23:23 DEBUG "result": [ 07:23:23 DEBUG { 07:23:23 DEBUG "metric": { 07:23:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG [...] 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 1712816603, 07:23:23 DEBUG "0" 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG } 07:23:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816603" 07:23:23 INFO [http_cmd]: http status code OK 07:23:23 DEBUG --- status code --- 07:23:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:23:23 DEBUG --- http response --- 07:23:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:23:23 DEBUG ----- output ----- 07:23:23 DEBUG { 07:23:23 DEBUG "status": "success", 07:23:23 DEBUG "data": { 07:23:23 DEBUG "resultType": "vector", 07:23:23 DEBUG "result": [ 07:23:23 DEBUG { 07:23:23 DEBUG "metric": { 07:23:23 DEBUG "pod": "ds-idrepo-2" 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG [...] 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG 07:23:23 DEBUG "pod": "ds-cts-2" 07:23:23 DEBUG }, 07:23:23 DEBUG "value": [ 07:23:23 DEBUG 1712816603, 07:23:23 DEBUG "0" 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG ] 07:23:23 DEBUG } 07:23:23 DEBUG } 07:23:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816604" 07:23:24 INFO [http_cmd]: http status code OK 07:23:24 DEBUG --- status code --- 07:23:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:23:24 DEBUG --- http response --- 07:23:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:23:24 DEBUG ----- output ----- 07:23:24 DEBUG { 07:23:24 DEBUG "status": "success", 07:23:24 DEBUG "data": { 07:23:24 DEBUG "resultType": "vector", 07:23:24 DEBUG "result": [ 07:23:24 DEBUG { 07:23:24 DEBUG "metric": { 07:23:24 DEBUG "instance": "10.142.0.107:9100" 07:23:24 DEBUG }, 07:23:24 DEBUG "value": [ 07:23:24 DEBUG 07:23:24 DEBUG 07:23:24 DEBUG 07:23:24 DEBUG [...] 07:23:24 DEBUG 07:23:24 DEBUG 07:23:24 DEBUG 07:23:24 DEBUG "instance": "10.142.0.77:9100" 07:23:24 DEBUG }, 07:23:24 DEBUG "value": [ 07:23:24 DEBUG 1712816604, 07:23:24 DEBUG "0.0006666666666670077" 07:23:24 DEBUG ] 07:23:24 DEBUG } 07:23:24 DEBUG ] 07:23:24 DEBUG } 07:23:24 DEBUG } 07:23:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816604" 07:23:24 INFO [http_cmd]: http status code OK 07:23:24 DEBUG --- status code --- 07:23:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:23:24 DEBUG --- http response --- 07:23:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:23:24 DEBUG ----- output ----- 07:23:24 DEBUG { 07:23:24 DEBUG "status": "success", 07:23:24 DEBUG "data": { 07:23:24 DEBUG "resultType": "vector", 07:23:24 DEBUG "result": [ 07:23:24 DEBUG { 07:23:24 DEBUG "metric": { 07:23:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:23:24 DEBUG }, 07:23:24 DEBUG "value": [ 07:23:24 DEBUG 07:23:24 DEBUG 07:23:24 DEBUG 07:23:24 DEBUG [...] 07:23:24 DEBUG 07:23:24 DEBUG 07:23:24 DEBUG 07:23:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:23:24 DEBUG }, 07:23:24 DEBUG "value": [ 07:23:24 DEBUG 1712816604, 07:23:24 DEBUG "0" 07:23:24 DEBUG ] 07:23:24 DEBUG } 07:23:24 DEBUG ] 07:23:24 DEBUG } 07:23:24 DEBUG } 07:23:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816604" 07:23:24 INFO [http_cmd]: http status code OK 07:23:24 DEBUG --- status code --- 07:23:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:23:24 DEBUG --- http response --- 07:23:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:23:24 DEBUG ----- output ----- 07:23:24 DEBUG { 07:23:24 DEBUG "status": "success", 07:23:24 DEBUG "data": { 07:23:24 DEBUG "resultType": "vector", 07:23:24 DEBUG "result": [ 07:23:24 DEBUG { 07:23:24 DEBUG "metric": { 07:23:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:23:24 DEBUG }, 07:23:24 DEBUG "value": [ 07:23:24 DEBUG 07:23:24 DEBUG 07:23:24 DEBUG 07:23:24 DEBUG [...] 07:23:24 DEBUG 07:23:24 DEBUG 07:23:24 DEBUG 07:23:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:23:24 DEBUG }, 07:23:24 DEBUG "value": [ 07:23:24 DEBUG 1712816604, 07:23:24 DEBUG "0" 07:23:24 DEBUG ] 07:23:24 DEBUG } 07:23:24 DEBUG ] 07:23:24 DEBUG } 07:23:24 DEBUG } 07:23:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712816604" 07:23:24 INFO [http_cmd]: http status code OK 07:23:24 DEBUG --- status code --- 07:23:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:23:24 DEBUG --- http response --- 07:23:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:23:24 DEBUG ----- output ----- 07:23:24 DEBUG { 07:23:24 DEBUG "status": "success", 07:23:24 DEBUG "data": { 07:23:24 DEBUG "resultType": "vector", 07:23:24 DEBUG "result": [ 07:23:24 DEBUG { 07:23:24 DEBUG "metric": { 07:23:24 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:23:24 DEBUG }, 07:23:24 DEBUG "value": [ 07:23:24 DEBUG 07:23:24 DEBUG 07:23:24 DEBUG 07:23:24 DEBUG [...] 07:23:24 DEBUG 07:23:24 DEBUG 07:23:24 DEBUG 07:23:24 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 07:23:24 DEBUG }, 07:23:24 DEBUG "value": [ 07:23:24 DEBUG 1712816604, 07:23:24 DEBUG "12927950848" 07:23:24 DEBUG ] 07:23:24 DEBUG } 07:23:24 DEBUG ] 07:23:24 DEBUG } 07:23:24 DEBUG } 07:23:36 INFO 07:23:36 INFO [loop_until]: kubectl --namespace=xlou top pods 07:23:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:23:37 INFO [loop_until]: OK (rc = 0) 07:23:37 DEBUG --- stdout --- 07:23:37 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 7m 8165Mi am-5b5bd8567b-8h2w4 6m 8662Mi am-5b5bd8567b-qlkw6 6m 8416Mi ds-cts-0 18m 12242Mi ds-cts-1 17m 13308Mi ds-cts-2 19m 13437Mi ds-idrepo-0 15m 5526Mi ds-idrepo-1 11m 5483Mi ds-idrepo-2 11m 5784Mi end-user-ui-58878d7668-gjq6d 1m 5Mi idm-668d898cfd-2vqdm 4m 1010Mi idm-668d898cfd-st2t7 5m 1136Mi lodemon-ccc964465-p7gdm 4m 68Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 1009m 1084Mi 07:23:37 DEBUG --- stderr --- 07:23:37 DEBUG 07:23:41 INFO 07:23:41 INFO [loop_until]: kubectl --namespace=xlou top node 07:23:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:23:41 INFO [loop_until]: OK (rc = 0) 07:23:41 DEBUG --- stdout --- 07:23:41 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 2744Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 69m 0% 2776Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 55m 0% 10121Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 116m 0% 3047Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 59m 0% 9814Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 140m 0% 3840Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 62m 0% 10326Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 65m 0% 6474Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 68m 0% 6236Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 77m 0% 14099Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 71m 0% 12929Mi 22% gke-xlou-cdm-ds-32e4dcb1-j2hk 70m 0% 14001Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 69m 0% 6174Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 1078m 6% 4892Mi 8% 07:23:41 DEBUG --- stderr --- 07:23:41 DEBUG 07:24:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816663" 07:24:23 INFO [http_cmd]: http status code OK 07:24:23 DEBUG --- status code --- 07:24:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:24:23 DEBUG --- http response --- 07:24:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712816663" 07:24:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:24:23 DEBUG ----- output ----- 07:24:23 DEBUG { 07:24:23 DEBUG "status": "success", 07:24:23 DEBUG "data": { 07:24:23 DEBUG "resultType": "vector", 07:24:23 DEBUG "result": [ 07:24:23 DEBUG { 07:24:23 DEBUG "metric": { 07:24:23 DEBUG "pod": "ds-idrepo-2" 07:24:23 DEBUG }, 07:24:23 DEBUG "value": [ 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG [...] 07:24:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816663" 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG "pod": "ds-idrepo-0" 07:24:23 DEBUG }, 07:24:23 DEBUG "value": [ 07:24:23 DEBUG 1712816663, 07:24:23 DEBUG "0" 07:24:23 DEBUG ] 07:24:23 DEBUG } 07:24:23 DEBUG ] 07:24:23 DEBUG } 07:24:23 DEBUG } 07:24:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816663" 07:24:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816663" 07:24:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816663" 07:24:23 INFO [http_cmd]: http status code OK 07:24:23 DEBUG --- status code --- 07:24:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:24:23 DEBUG --- http response --- 07:24:23 INFO [http_cmd]: http status code OK 07:24:23 DEBUG --- status code --- 07:24:23 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:24:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:24:23 INFO [http_cmd]: http status code OK 07:24:23 DEBUG --- http response --- 07:24:23 DEBUG --- status code --- 07:24:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:24:23 DEBUG --- http response --- 07:24:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:24:23 DEBUG ----- output ----- 07:24:23 DEBUG { 07:24:23 DEBUG "status": "success", 07:24:23 DEBUG "data": { 07:24:23 DEBUG "resultType": "vector", 07:24:23 DEBUG "result": [ 07:24:23 DEBUG { 07:24:23 DEBUG "metric": { 07:24:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:24:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:24:23 DEBUG ----- output ----- 07:24:23 DEBUG }, 07:24:23 DEBUG { 07:24:23 DEBUG "value": [ 07:24:23 DEBUG "status": "success", 07:24:23 DEBUG 07:24:23 DEBUG "data": { 07:24:23 DEBUG 07:24:23 DEBUG "resultType": "vector", 07:24:23 DEBUG 07:24:23 DEBUG "result": [ 07:24:23 DEBUG [...] 07:24:23 DEBUG { 07:24:23 DEBUG 07:24:23 DEBUG "metric": { 07:24:23 DEBUG 07:24:23 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:24:23 DEBUG 07:24:23 DEBUG }, 07:24:23 DEBUG "pod": "ds-idrepo-1" 07:24:23 DEBUG "value": [ 07:24:23 DEBUG }, 07:24:23 DEBUG 07:24:23 DEBUG "value": [ 07:24:23 DEBUG 07:24:23 DEBUG 1712816663, 07:24:23 DEBUG 07:24:23 DEBUG "336.7575943918337" 07:24:23 DEBUG [...] 07:24:23 DEBUG ] 07:24:23 DEBUG 07:24:23 DEBUG } 07:24:23 DEBUG 07:24:23 DEBUG ] 07:24:23 DEBUG 07:24:23 DEBUG } 07:24:23 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:24:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712816663" 07:24:23 DEBUG } 07:24:23 DEBUG }, 07:24:23 DEBUG "value": [ 07:24:23 DEBUG 1712816663, 07:24:23 DEBUG "0.4564087219274805" 07:24:23 DEBUG ] 07:24:23 DEBUG } 07:24:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712816663" 07:24:23 DEBUG ] 07:24:23 DEBUG } 07:24:23 DEBUG } 07:24:23 INFO [http_cmd]: http status code OK 07:24:23 DEBUG --- status code --- 07:24:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:24:23 DEBUG --- http response --- 07:24:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:24:23 DEBUG ----- output ----- 07:24:23 DEBUG { 07:24:23 DEBUG "status": "success", 07:24:23 DEBUG "data": { 07:24:23 DEBUG "resultType": "vector", 07:24:23 DEBUG "result": [ 07:24:23 DEBUG { 07:24:23 DEBUG "metric": { 07:24:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:24:23 DEBUG }, 07:24:23 DEBUG "value": [ 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG [...] 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:24:23 DEBUG }, 07:24:23 DEBUG "value": [ 07:24:23 DEBUG 1712816663, 07:24:23 DEBUG "0" 07:24:23 DEBUG ] 07:24:23 DEBUG } 07:24:23 DEBUG ] 07:24:23 DEBUG } 07:24:23 DEBUG } 07:24:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816663" 07:24:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712816663" 07:24:23 INFO [http_cmd]: http status code OK 07:24:23 INFO [http_cmd]: http status code OK 07:24:23 ERROR [http_cmd]: ERROR 07:24:23 DEBUG --- status code --- 07:24:23 DEBUG --- status code --- 07:24:23 ERROR --- status code --- 07:24:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:24:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:24:23 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:24:23 DEBUG --- http response --- 07:24:23 DEBUG --- http response --- 07:24:23 ERROR --- http response --- 07:24:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:24:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:24:23 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:24:23 DEBUG ----- output ----- 07:24:23 DEBUG ----- output ----- 07:24:23 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:24:23 DEBUG { 07:24:23 DEBUG { 07:24:23 DEBUG "status": "success", 07:24:23 DEBUG "status": "success", 07:24:23 DEBUG "data": { 07:24:23 DEBUG "resultType": "vector", 07:24:23 DEBUG "result": [ 07:24:23 DEBUG "data": { 07:24:23 DEBUG { 07:24:23 DEBUG "resultType": "vector", 07:24:23 DEBUG "metric": { 07:24:23 DEBUG "result": [ 07:24:23 DEBUG "instance": "10.142.0.107:9100" 07:24:23 DEBUG { 07:24:23 DEBUG }, 07:24:23 DEBUG "metric": { 07:24:23 DEBUG "value": [ 07:24:23 DEBUG "pod": "ds-idrepo-2" 07:24:23 DEBUG 07:24:23 DEBUG }, 07:24:23 DEBUG 07:24:23 DEBUG "value": [ 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG [...] 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG [...] 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG "instance": "10.142.0.77:9100" 07:24:23 DEBUG 07:24:23 DEBUG }, 07:24:23 DEBUG 07:24:23 DEBUG "value": [ 07:24:23 DEBUG "pod": "ds-cts-2" 07:24:23 DEBUG 1712816663, 07:24:23 DEBUG }, 07:24:23 DEBUG "34584.666666666664" 07:24:23 DEBUG "value": [ 07:24:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816663" 07:24:23 DEBUG ] 07:24:23 DEBUG 1712816663, 07:24:23 DEBUG } 07:24:23 DEBUG "0" 07:24:23 DEBUG ] 07:24:23 DEBUG ] 07:24:23 DEBUG } 07:24:23 DEBUG } 07:24:23 DEBUG } 07:24:23 DEBUG ] 07:24:23 DEBUG } 07:24:23 DEBUG } 07:24:23 INFO [http_cmd]: http status code OK 07:24:23 DEBUG --- status code --- 07:24:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:24:23 DEBUG --- http response --- 07:24:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:24:23 DEBUG ----- output ----- 07:24:23 DEBUG { 07:24:23 DEBUG "status": "success", 07:24:23 DEBUG "data": { 07:24:23 DEBUG "resultType": "vector", 07:24:23 DEBUG "result": [ 07:24:23 DEBUG { 07:24:23 DEBUG "metric": { 07:24:23 DEBUG "instance": "10.142.0.107:9100" 07:24:23 DEBUG }, 07:24:23 DEBUG "value": [ 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG [...] 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG "instance": "10.142.0.77:9100" 07:24:23 DEBUG }, 07:24:23 DEBUG "value": [ 07:24:23 DEBUG 1712816663, 07:24:23 DEBUG "0.002399999999999854" 07:24:23 DEBUG ] 07:24:23 DEBUG } 07:24:23 DEBUG ] 07:24:23 DEBUG } 07:24:23 DEBUG } 07:24:23 INFO [http_cmd]: http status code OK 07:24:23 DEBUG --- status code --- 07:24:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:24:23 DEBUG --- http response --- 07:24:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:24:23 DEBUG ----- output ----- 07:24:23 DEBUG { 07:24:23 DEBUG "status": "success", 07:24:23 DEBUG "data": { 07:24:23 DEBUG "resultType": "vector", 07:24:23 DEBUG "result": [ 07:24:23 DEBUG { 07:24:23 DEBUG "metric": { 07:24:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:24:23 DEBUG }, 07:24:23 DEBUG "value": [ 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG [...] 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:24:23 DEBUG }, 07:24:23 DEBUG "value": [ 07:24:23 DEBUG 1712816663, 07:24:23 DEBUG "0.026" 07:24:23 DEBUG ] 07:24:23 DEBUG } 07:24:23 DEBUG ] 07:24:23 DEBUG } 07:24:23 DEBUG } 07:24:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816663" 07:24:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816663" 07:24:23 INFO [http_cmd]: http status code OK 07:24:23 DEBUG --- status code --- 07:24:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:24:23 DEBUG --- http response --- 07:24:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:24:23 DEBUG ----- output ----- 07:24:23 DEBUG { 07:24:23 DEBUG "status": "success", 07:24:23 DEBUG "data": { 07:24:23 DEBUG "resultType": "vector", 07:24:23 DEBUG "result": [ 07:24:23 DEBUG { 07:24:23 DEBUG "metric": { 07:24:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:24:23 DEBUG }, 07:24:23 DEBUG "value": [ 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG [...] 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG "pod": "ds-idrepo-1" 07:24:23 DEBUG }, 07:24:23 DEBUG "value": [ 07:24:23 DEBUG 1712816663, 07:24:23 DEBUG "0.005073618583157749" 07:24:23 DEBUG ] 07:24:23 DEBUG } 07:24:23 DEBUG ] 07:24:23 DEBUG } 07:24:23 DEBUG } 07:24:23 INFO [http_cmd]: http status code OK 07:24:23 DEBUG --- status code --- 07:24:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:24:23 DEBUG --- http response --- 07:24:23 INFO [http_cmd]: http status code OK 07:24:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:24:23 DEBUG --- status code --- 07:24:23 DEBUG ----- output ----- 07:24:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:24:23 DEBUG { 07:24:23 DEBUG --- http response --- 07:24:23 DEBUG "status": "success", 07:24:23 DEBUG "data": { 07:24:23 DEBUG "resultType": "vector", 07:24:23 DEBUG "result": [ 07:24:23 DEBUG { 07:24:23 DEBUG "metric": { 07:24:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:24:23 DEBUG }, 07:24:23 DEBUG "value": [ 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG [...] 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:24:23 DEBUG }, 07:24:23 DEBUG "value": [ 07:24:23 DEBUG 1712816663, 07:24:23 DEBUG "0" 07:24:23 DEBUG ] 07:24:23 DEBUG } 07:24:23 DEBUG ] 07:24:23 DEBUG } 07:24:23 DEBUG } 07:24:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:24:23 DEBUG ----- output ----- 07:24:23 DEBUG { 07:24:23 DEBUG "status": "success", 07:24:23 DEBUG "data": { 07:24:23 DEBUG "resultType": "vector", 07:24:23 DEBUG "result": [ 07:24:23 DEBUG { 07:24:23 DEBUG "metric": { 07:24:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:24:23 DEBUG }, 07:24:23 DEBUG "value": [ 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG [...] 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG "pod": "ds-idrepo-1" 07:24:23 DEBUG }, 07:24:23 DEBUG "value": [ 07:24:23 DEBUG 1712816663, 07:24:23 DEBUG "0" 07:24:23 DEBUG ] 07:24:23 DEBUG } 07:24:23 DEBUG ] 07:24:23 DEBUG } 07:24:23 DEBUG } 07:24:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816663" 07:24:23 INFO [http_cmd]: http status code OK 07:24:23 DEBUG --- status code --- 07:24:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:24:23 DEBUG --- http response --- 07:24:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:24:23 DEBUG ----- output ----- 07:24:23 DEBUG { 07:24:23 DEBUG "status": "success", 07:24:23 DEBUG "data": { 07:24:23 DEBUG "resultType": "vector", 07:24:23 DEBUG "result": [ 07:24:23 DEBUG { 07:24:23 DEBUG "metric": { 07:24:23 DEBUG "pod": "ds-idrepo-2" 07:24:23 DEBUG }, 07:24:23 DEBUG "value": [ 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG [...] 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG "pod": "ds-idrepo-0" 07:24:23 DEBUG }, 07:24:23 DEBUG "value": [ 07:24:23 DEBUG 1712816663, 07:24:23 DEBUG "0" 07:24:23 DEBUG ] 07:24:23 DEBUG } 07:24:23 DEBUG ] 07:24:23 DEBUG } 07:24:23 DEBUG } 07:24:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816663" 07:24:23 INFO [http_cmd]: http status code OK 07:24:23 DEBUG --- status code --- 07:24:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:24:23 DEBUG --- http response --- 07:24:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:24:23 DEBUG ----- output ----- 07:24:23 DEBUG { 07:24:23 DEBUG "status": "success", 07:24:23 DEBUG "data": { 07:24:23 DEBUG "resultType": "vector", 07:24:23 DEBUG "result": [ 07:24:23 DEBUG { 07:24:23 DEBUG "metric": { 07:24:23 DEBUG "instance": "10.142.0.107:9100" 07:24:23 DEBUG }, 07:24:23 DEBUG "value": [ 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG [...] 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG "instance": "10.142.0.77:9100" 07:24:23 DEBUG }, 07:24:23 DEBUG "value": [ 07:24:23 DEBUG 1712816663, 07:24:23 DEBUG "53009787904" 07:24:23 DEBUG ] 07:24:23 DEBUG } 07:24:23 DEBUG ] 07:24:23 DEBUG } 07:24:23 DEBUG } 07:24:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816663" 07:24:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816663" 07:24:23 INFO [http_cmd]: http status code OK 07:24:23 DEBUG --- status code --- 07:24:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:24:23 DEBUG --- http response --- 07:24:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:24:23 DEBUG ----- output ----- 07:24:23 DEBUG { 07:24:23 DEBUG "status": "success", 07:24:23 DEBUG "data": { 07:24:23 DEBUG "resultType": "vector", 07:24:23 DEBUG "result": [ 07:24:23 DEBUG { 07:24:23 DEBUG "metric": { 07:24:23 DEBUG "pod": "ds-idrepo-2" 07:24:23 DEBUG }, 07:24:23 DEBUG "value": [ 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG [...] 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG "pod": "ds-cts-2" 07:24:23 DEBUG }, 07:24:23 DEBUG "value": [ 07:24:23 DEBUG 1712816663, 07:24:23 DEBUG "29.03333333333333" 07:24:23 DEBUG ] 07:24:23 DEBUG } 07:24:23 DEBUG ] 07:24:23 DEBUG } 07:24:23 DEBUG } 07:24:23 INFO [http_cmd]: http status code OK 07:24:23 DEBUG --- status code --- 07:24:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:24:23 DEBUG --- http response --- 07:24:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:24:23 DEBUG ----- output ----- 07:24:23 DEBUG { 07:24:23 DEBUG "status": "success", 07:24:23 DEBUG "data": { 07:24:23 DEBUG "resultType": "vector", 07:24:23 DEBUG "result": [ 07:24:23 DEBUG { 07:24:23 DEBUG "metric": { 07:24:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:24:23 DEBUG }, 07:24:23 DEBUG "value": [ 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG [...] 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG "pod": "ds-idrepo-1" 07:24:23 DEBUG }, 07:24:23 DEBUG "value": [ 07:24:23 DEBUG 1712816663, 07:24:23 DEBUG "1347.0303775673349" 07:24:23 DEBUG ] 07:24:23 DEBUG } 07:24:23 DEBUG ] 07:24:23 DEBUG } 07:24:23 DEBUG } 07:24:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816663" 07:24:23 INFO [http_cmd]: http status code OK 07:24:23 DEBUG --- status code --- 07:24:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:24:23 DEBUG --- http response --- 07:24:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:24:23 DEBUG ----- output ----- 07:24:23 DEBUG { 07:24:23 DEBUG "status": "success", 07:24:23 DEBUG "data": { 07:24:23 DEBUG "resultType": "vector", 07:24:23 DEBUG "result": [ 07:24:23 DEBUG { 07:24:23 DEBUG "metric": { 07:24:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:24:23 DEBUG }, 07:24:23 DEBUG "value": [ 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG [...] 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG 07:24:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:24:23 DEBUG }, 07:24:23 DEBUG "value": [ 07:24:23 DEBUG 1712816663, 07:24:23 DEBUG "0" 07:24:23 DEBUG ] 07:24:23 DEBUG } 07:24:23 DEBUG ] 07:24:23 DEBUG } 07:24:23 DEBUG } 07:24:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816663" 07:24:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816663" 07:24:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816663" 07:24:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816663" 07:24:24 INFO [http_cmd]: http status code OK 07:24:24 DEBUG --- status code --- 07:24:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:24:24 DEBUG --- http response --- 07:24:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:24:24 DEBUG ----- output ----- 07:24:24 DEBUG { 07:24:24 DEBUG "status": "success", 07:24:24 DEBUG "data": { 07:24:24 DEBUG "resultType": "vector", 07:24:24 DEBUG "result": [ 07:24:24 DEBUG { 07:24:24 DEBUG "metric": { 07:24:24 DEBUG "pod": "ds-idrepo-2" 07:24:24 DEBUG }, 07:24:24 DEBUG "value": [ 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG [...] 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG "pod": "ds-idrepo-0" 07:24:24 DEBUG }, 07:24:24 DEBUG "value": [ 07:24:24 DEBUG 1712816663, 07:24:24 DEBUG "0" 07:24:24 DEBUG ] 07:24:24 DEBUG } 07:24:24 DEBUG ] 07:24:24 DEBUG } 07:24:24 DEBUG } 07:24:24 INFO [http_cmd]: http status code OK 07:24:24 INFO [http_cmd]: http status code OK 07:24:24 INFO [http_cmd]: http status code OK 07:24:24 DEBUG --- status code --- 07:24:24 DEBUG --- status code --- 07:24:24 DEBUG --- status code --- 07:24:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:24:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:24:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:24:24 DEBUG --- http response --- 07:24:24 DEBUG --- http response --- 07:24:24 DEBUG --- http response --- 07:24:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:24:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:24:24 DEBUG ----- output ----- 07:24:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:24:24 DEBUG ----- output ----- 07:24:24 DEBUG { 07:24:24 DEBUG ----- output ----- 07:24:24 DEBUG { 07:24:24 DEBUG "status": "success", 07:24:24 DEBUG { 07:24:24 DEBUG "status": "success", 07:24:24 DEBUG "data": { 07:24:24 DEBUG "status": "success", 07:24:24 DEBUG "data": { 07:24:24 DEBUG "resultType": "vector", 07:24:24 DEBUG "data": { 07:24:24 DEBUG "resultType": "vector", 07:24:24 DEBUG "result": [ 07:24:24 DEBUG "resultType": "vector", 07:24:24 DEBUG "result": [ 07:24:24 DEBUG { 07:24:24 DEBUG "result": [ 07:24:24 DEBUG { 07:24:24 DEBUG "metric": { 07:24:24 DEBUG { 07:24:24 DEBUG "metric": { 07:24:24 DEBUG "pod": "ds-idrepo-2" 07:24:24 DEBUG "metric": { 07:24:24 DEBUG "pod": "ds-idrepo-2" 07:24:24 DEBUG }, 07:24:24 DEBUG "instance": "10.142.0.107:9100" 07:24:24 DEBUG }, 07:24:24 DEBUG "value": [ 07:24:24 DEBUG }, 07:24:24 DEBUG "value": [ 07:24:24 DEBUG 07:24:24 DEBUG "value": [ 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG [...] 07:24:24 DEBUG 07:24:24 DEBUG [...] 07:24:24 DEBUG 07:24:24 DEBUG [...] 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG "pod": "ds-cts-2" 07:24:24 DEBUG 07:24:24 DEBUG "pod": "ds-cts-2" 07:24:24 DEBUG }, 07:24:24 DEBUG "instance": "10.142.0.77:9100" 07:24:24 DEBUG }, 07:24:24 DEBUG "value": [ 07:24:24 DEBUG }, 07:24:24 DEBUG "value": [ 07:24:24 DEBUG 1712816663, 07:24:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816664" 07:24:24 DEBUG "value": [ 07:24:24 DEBUG 1712816663, 07:24:24 DEBUG "0" 07:24:24 DEBUG 1712816663, 07:24:24 DEBUG "0" 07:24:24 DEBUG ] 07:24:24 DEBUG "35456.5" 07:24:24 DEBUG ] 07:24:24 DEBUG } 07:24:24 DEBUG ] 07:24:24 DEBUG } 07:24:24 DEBUG ] 07:24:24 DEBUG } 07:24:24 DEBUG ] 07:24:24 DEBUG } 07:24:24 DEBUG ] 07:24:24 DEBUG } 07:24:24 DEBUG } 07:24:24 DEBUG } 07:24:24 DEBUG } 07:24:24 DEBUG } 07:24:24 INFO [http_cmd]: http status code OK 07:24:24 DEBUG --- status code --- 07:24:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:24:24 DEBUG --- http response --- 07:24:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:24:24 DEBUG ----- output ----- 07:24:24 DEBUG { 07:24:24 DEBUG "status": "success", 07:24:24 DEBUG "data": { 07:24:24 DEBUG "resultType": "vector", 07:24:24 DEBUG "result": [ 07:24:24 DEBUG { 07:24:24 DEBUG "metric": { 07:24:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:24:24 DEBUG }, 07:24:24 DEBUG "value": [ 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG [...] 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:24:24 DEBUG }, 07:24:24 DEBUG "value": [ 07:24:24 DEBUG 1712816664, 07:24:24 DEBUG "0" 07:24:24 DEBUG ] 07:24:24 DEBUG } 07:24:24 DEBUG ] 07:24:24 DEBUG } 07:24:24 DEBUG } 07:24:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712816664" 07:24:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816664" 07:24:24 INFO [http_cmd]: http status code OK 07:24:24 DEBUG --- status code --- 07:24:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:24:24 DEBUG --- http response --- 07:24:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:24:24 DEBUG ----- output ----- 07:24:24 DEBUG { 07:24:24 DEBUG "status": "success", 07:24:24 DEBUG "data": { 07:24:24 DEBUG "resultType": "vector", 07:24:24 DEBUG "result": [ 07:24:24 DEBUG { 07:24:24 DEBUG "metric": { 07:24:24 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:24:24 DEBUG }, 07:24:24 DEBUG "value": [ 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG [...] 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 07:24:24 DEBUG }, 07:24:24 DEBUG "value": [ 07:24:24 DEBUG 1712816664, 07:24:24 DEBUG "12929396736" 07:24:24 DEBUG ] 07:24:24 DEBUG } 07:24:24 DEBUG ] 07:24:24 DEBUG } 07:24:24 DEBUG } 07:24:24 INFO [http_cmd]: http status code OK 07:24:24 DEBUG --- status code --- 07:24:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:24:24 DEBUG --- http response --- 07:24:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:24:24 DEBUG ----- output ----- 07:24:24 DEBUG { 07:24:24 DEBUG "status": "success", 07:24:24 DEBUG "data": { 07:24:24 DEBUG "resultType": "vector", 07:24:24 DEBUG "result": [ 07:24:24 DEBUG { 07:24:24 DEBUG "metric": { 07:24:24 DEBUG "instance": "10.142.0.107:9100" 07:24:24 DEBUG }, 07:24:24 DEBUG "value": [ 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG [...] 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG "instance": "10.142.0.77:9100" 07:24:24 DEBUG }, 07:24:24 DEBUG "value": [ 07:24:24 DEBUG 1712816664, 07:24:24 DEBUG "0" 07:24:24 DEBUG ] 07:24:24 DEBUG } 07:24:24 DEBUG ] 07:24:24 DEBUG } 07:24:24 DEBUG } 07:24:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816664" 07:24:24 INFO [http_cmd]: http status code OK 07:24:24 DEBUG --- status code --- 07:24:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:24:24 DEBUG --- http response --- 07:24:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:24:24 DEBUG ----- output ----- 07:24:24 DEBUG { 07:24:24 DEBUG "status": "success", 07:24:24 DEBUG "data": { 07:24:24 DEBUG "resultType": "vector", 07:24:24 DEBUG "result": [ 07:24:24 DEBUG { 07:24:24 DEBUG "metric": { 07:24:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:24:24 DEBUG }, 07:24:24 DEBUG "value": [ 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG [...] 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:24:24 DEBUG }, 07:24:24 DEBUG "value": [ 07:24:24 DEBUG 1712816664, 07:24:24 DEBUG "0" 07:24:24 DEBUG ] 07:24:24 DEBUG } 07:24:24 DEBUG ] 07:24:24 DEBUG } 07:24:24 DEBUG } 07:24:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816664" 07:24:24 INFO [http_cmd]: http status code OK 07:24:24 DEBUG --- status code --- 07:24:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:24:24 DEBUG --- http response --- 07:24:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:24:24 DEBUG ----- output ----- 07:24:24 DEBUG { 07:24:24 DEBUG "status": "success", 07:24:24 DEBUG "data": { 07:24:24 DEBUG "resultType": "vector", 07:24:24 DEBUG "result": [ 07:24:24 DEBUG { 07:24:24 DEBUG "metric": { 07:24:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:24:24 DEBUG }, 07:24:24 DEBUG "value": [ 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG [...] 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG 07:24:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:24:24 DEBUG }, 07:24:24 DEBUG "value": [ 07:24:24 DEBUG 1712816664, 07:24:24 DEBUG "0" 07:24:24 DEBUG ] 07:24:24 DEBUG } 07:24:24 DEBUG ] 07:24:24 DEBUG } 07:24:24 DEBUG } 07:24:37 INFO 07:24:37 INFO [loop_until]: kubectl --namespace=xlou top pods 07:24:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:24:37 INFO [loop_until]: OK (rc = 0) 07:24:37 DEBUG --- stdout --- 07:24:37 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 7m 8165Mi am-5b5bd8567b-8h2w4 6m 8661Mi am-5b5bd8567b-qlkw6 6m 8417Mi ds-cts-0 16m 12242Mi ds-cts-1 19m 13308Mi ds-cts-2 18m 13437Mi ds-idrepo-0 13m 5525Mi ds-idrepo-1 12m 5483Mi ds-idrepo-2 13m 5784Mi end-user-ui-58878d7668-gjq6d 1m 5Mi idm-668d898cfd-2vqdm 3m 1011Mi idm-668d898cfd-st2t7 4m 1136Mi lodemon-ccc964465-p7gdm 2m 68Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 529m 1193Mi 07:24:37 DEBUG --- stderr --- 07:24:37 DEBUG 07:24:41 INFO 07:24:41 INFO [loop_until]: kubectl --namespace=xlou top node 07:24:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:24:41 INFO [loop_until]: OK (rc = 0) 07:24:41 DEBUG --- stdout --- 07:24:41 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 2737Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 64m 0% 2785Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 54m 0% 10111Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 119m 0% 3047Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 58m 0% 9801Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 128m 0% 3850Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 66m 0% 10319Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 67m 0% 6473Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 66m 0% 6236Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 78m 0% 14094Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 73m 0% 12931Mi 22% gke-xlou-cdm-ds-32e4dcb1-j2hk 73m 0% 14002Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 67m 0% 6174Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 494m 3% 5039Mi 8% 07:24:41 DEBUG --- stderr --- 07:24:41 DEBUG 07:25:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816723" 07:25:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816723" 07:25:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816723" 07:25:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712816723" 07:25:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816723" 07:25:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712816723" 07:25:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816723" 07:25:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816723" 07:25:23 INFO [http_cmd]: http status code OK 07:25:23 DEBUG --- status code --- 07:25:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:25:23 DEBUG --- http response --- 07:25:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:25:23 DEBUG ----- output ----- 07:25:23 DEBUG { 07:25:23 DEBUG "status": "success", 07:25:23 DEBUG "data": { 07:25:23 DEBUG "resultType": "vector", 07:25:23 DEBUG "result": [ 07:25:23 DEBUG { 07:25:23 DEBUG "metric": { 07:25:23 DEBUG "pod": "ds-idrepo-2" 07:25:23 DEBUG }, 07:25:23 DEBUG "value": [ 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG [...] 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG "pod": "ds-idrepo-0" 07:25:23 DEBUG }, 07:25:23 DEBUG "value": [ 07:25:23 DEBUG 1712816723, 07:25:23 DEBUG "0" 07:25:23 DEBUG ] 07:25:23 DEBUG } 07:25:23 DEBUG ] 07:25:23 DEBUG } 07:25:23 DEBUG } 07:25:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816723" 07:25:23 ERROR [http_cmd]: ERROR 07:25:23 INFO [http_cmd]: http status code OK 07:25:23 ERROR --- status code --- 07:25:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712816723" 07:25:23 DEBUG --- status code --- 07:25:23 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:25:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:25:23 INFO [http_cmd]: http status code OK 07:25:23 INFO [http_cmd]: http status code OK 07:25:23 ERROR --- http response --- 07:25:23 DEBUG --- http response --- 07:25:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816723" 07:25:23 INFO [http_cmd]: http status code OK 07:25:23 DEBUG --- status code --- 07:25:23 DEBUG --- status code --- 07:25:23 INFO [http_cmd]: http status code OK 07:25:23 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:25:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:25:23 DEBUG --- status code --- 07:25:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712816723" 07:25:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:25:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:25:23 DEBUG --- status code --- 07:25:23 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:25:23 DEBUG ----- output ----- 07:25:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:25:23 DEBUG --- http response --- 07:25:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816723" 07:25:23 DEBUG --- http response --- 07:25:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:25:23 DEBUG { 07:25:23 DEBUG --- http response --- 07:25:23 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:25:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:25:23 DEBUG --- http response --- 07:25:23 INFO [http_cmd]: http status code OK 07:25:23 DEBUG "status": "success", 07:25:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:25:23 DEBUG ----- output ----- 07:25:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:25:23 DEBUG --- status code --- 07:25:23 DEBUG "data": { 07:25:23 DEBUG ----- output ----- 07:25:23 DEBUG { 07:25:23 DEBUG ----- output ----- 07:25:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:25:23 DEBUG "resultType": "vector", 07:25:23 DEBUG { 07:25:23 DEBUG "status": "success", 07:25:23 DEBUG { 07:25:23 DEBUG --- http response --- 07:25:23 INFO [http_cmd]: http status code OK 07:25:23 DEBUG "result": [ 07:25:23 DEBUG "status": "success", 07:25:23 DEBUG "data": { 07:25:23 DEBUG "status": "success", 07:25:23 DEBUG --- status code --- 07:25:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:25:23 DEBUG { 07:25:23 DEBUG "data": { 07:25:23 DEBUG "resultType": "vector", 07:25:23 DEBUG "data": { 07:25:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:25:23 DEBUG ----- output ----- 07:25:23 INFO [http_cmd]: http status code OK 07:25:23 DEBUG "metric": { 07:25:23 DEBUG "resultType": "vector", 07:25:23 DEBUG "result": [ 07:25:23 DEBUG "resultType": "vector", 07:25:23 DEBUG --- http response --- 07:25:23 DEBUG { 07:25:23 DEBUG --- status code --- 07:25:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:25:23 DEBUG "result": [ 07:25:23 DEBUG { 07:25:23 DEBUG "result": [ 07:25:23 DEBUG "status": "success", 07:25:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:25:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:25:23 DEBUG }, 07:25:23 DEBUG { 07:25:23 DEBUG "metric": { 07:25:23 DEBUG { 07:25:23 DEBUG "data": { 07:25:23 DEBUG ----- output ----- 07:25:23 DEBUG --- http response --- 07:25:23 DEBUG "value": [ 07:25:23 DEBUG "metric": { 07:25:23 DEBUG "pod": "ds-idrepo-2" 07:25:23 DEBUG "metric": { 07:25:23 INFO [http_cmd]: http status code OK 07:25:23 INFO [http_cmd]: http status code OK 07:25:23 INFO [http_cmd]: http status code OK 07:25:23 DEBUG "resultType": "vector", 07:25:23 DEBUG { 07:25:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:25:23 DEBUG 07:25:23 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:25:23 DEBUG }, 07:25:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:25:23 DEBUG --- status code --- 07:25:23 DEBUG --- status code --- 07:25:23 DEBUG --- status code --- 07:25:23 DEBUG "result": [ 07:25:23 DEBUG "status": "success", 07:25:23 DEBUG ----- output ----- 07:25:23 DEBUG 07:25:23 DEBUG }, 07:25:23 DEBUG "value": [ 07:25:23 DEBUG }, 07:25:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:25:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:25:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:25:23 DEBUG { 07:25:23 DEBUG "data": { 07:25:23 DEBUG { 07:25:23 DEBUG 07:25:23 DEBUG "value": [ 07:25:23 DEBUG 07:25:23 DEBUG "value": [ 07:25:23 DEBUG --- http response --- 07:25:23 DEBUG --- http response --- 07:25:23 DEBUG --- http response --- 07:25:23 DEBUG "metric": { 07:25:23 DEBUG "resultType": "vector", 07:25:23 DEBUG "status": "success", 07:25:23 DEBUG [...] 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:25:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:25:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:25:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:25:23 DEBUG "result": [ 07:25:23 DEBUG "data": { 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG ----- output ----- 07:25:23 DEBUG ----- output ----- 07:25:23 DEBUG }, 07:25:23 DEBUG ----- output ----- 07:25:23 DEBUG { 07:25:23 DEBUG "resultType": "vector", 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG [...] 07:25:23 DEBUG 07:25:23 DEBUG { 07:25:23 DEBUG { 07:25:23 DEBUG "value": [ 07:25:23 DEBUG { 07:25:23 DEBUG "metric": { 07:25:23 DEBUG "result": [ 07:25:23 DEBUG 07:25:23 DEBUG [...] 07:25:23 DEBUG 07:25:23 DEBUG [...] 07:25:23 DEBUG "status": "success", 07:25:23 DEBUG "status": "success", 07:25:23 DEBUG 07:25:23 DEBUG "status": "success", 07:25:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:25:23 DEBUG { 07:25:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG "data": { 07:25:23 DEBUG "data": { 07:25:23 DEBUG 07:25:23 DEBUG "data": { 07:25:23 DEBUG }, 07:25:23 DEBUG "metric": { 07:25:23 DEBUG }, 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG "resultType": "vector", 07:25:23 DEBUG "resultType": "vector", 07:25:23 DEBUG 07:25:23 DEBUG "resultType": "vector", 07:25:23 DEBUG "value": [ 07:25:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:25:23 DEBUG "value": [ 07:25:23 DEBUG 07:25:23 DEBUG "pod": "ds-idrepo-0" 07:25:23 DEBUG 07:25:23 DEBUG "result": [ 07:25:23 DEBUG "result": [ 07:25:23 DEBUG [...] 07:25:23 DEBUG "result": [ 07:25:23 DEBUG 07:25:23 DEBUG }, 07:25:23 DEBUG 1712816723, 07:25:23 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:25:23 DEBUG }, 07:25:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:25:23 DEBUG { 07:25:23 DEBUG { 07:25:23 DEBUG 07:25:23 DEBUG { 07:25:23 DEBUG 07:25:23 DEBUG "value": [ 07:25:23 DEBUG "0" 07:25:23 DEBUG }, 07:25:23 DEBUG "value": [ 07:25:23 DEBUG }, 07:25:23 DEBUG "metric": { 07:25:23 DEBUG "metric": { 07:25:23 DEBUG 07:25:23 DEBUG "metric": { 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG ] 07:25:23 DEBUG "value": [ 07:25:23 DEBUG 1712816723, 07:25:23 DEBUG "value": [ 07:25:23 DEBUG "pod": "ds-idrepo-2" 07:25:23 DEBUG "instance": "10.142.0.107:9100" 07:25:23 DEBUG 07:25:23 DEBUG "instance": "10.142.0.107:9100" 07:25:23 DEBUG [...] 07:25:23 DEBUG 07:25:23 DEBUG } 07:25:23 DEBUG 1712816723, 07:25:23 DEBUG "0" 07:25:23 DEBUG 1712816723, 07:25:23 DEBUG }, 07:25:23 DEBUG }, 07:25:23 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:25:23 DEBUG }, 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG ] 07:25:23 DEBUG "1.0336681697990162" 07:25:23 DEBUG ] 07:25:23 DEBUG "0" 07:25:23 DEBUG "value": [ 07:25:23 DEBUG "value": [ 07:25:23 DEBUG }, 07:25:23 DEBUG "value": [ 07:25:23 DEBUG 07:25:23 DEBUG [...] 07:25:23 DEBUG } 07:25:23 DEBUG ] 07:25:23 DEBUG } 07:25:23 DEBUG ] 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG "value": [ 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG } 07:25:23 DEBUG } 07:25:23 DEBUG ] 07:25:23 DEBUG } 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG 1712816723, 07:25:23 DEBUG 07:25:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:25:23 DEBUG 07:25:23 DEBUG ] 07:25:23 DEBUG } 07:25:23 DEBUG ] 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG "0" 07:25:23 DEBUG 07:25:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816723" 07:25:23 DEBUG }, 07:25:23 DEBUG 07:25:23 DEBUG } 07:25:23 DEBUG } 07:25:23 DEBUG } 07:25:23 DEBUG [...] 07:25:23 DEBUG [...] 07:25:23 DEBUG ] 07:25:23 DEBUG [...] 07:25:23 DEBUG "value": [ 07:25:23 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:25:23 DEBUG } 07:25:23 DEBUG } 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG } 07:25:23 DEBUG 07:25:23 DEBUG 1712816723, 07:25:23 DEBUG }, 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG ] 07:25:23 DEBUG "0" 07:25:23 DEBUG "value": [ 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG "instance": "10.142.0.77:9100" 07:25:23 DEBUG } 07:25:23 DEBUG ] 07:25:23 DEBUG 1712816723, 07:25:23 DEBUG "pod": "ds-cts-2" 07:25:23 DEBUG 07:25:23 DEBUG }, 07:25:23 DEBUG } 07:25:23 DEBUG } 07:25:23 DEBUG "0.026" 07:25:23 DEBUG }, 07:25:23 DEBUG "instance": "10.142.0.77:9100" 07:25:23 DEBUG "value": [ 07:25:23 DEBUG ] 07:25:23 DEBUG ] 07:25:23 DEBUG "value": [ 07:25:23 DEBUG }, 07:25:23 DEBUG 1712816723, 07:25:23 DEBUG } 07:25:23 DEBUG } 07:25:23 DEBUG 1712816723, 07:25:23 DEBUG "value": [ 07:25:23 DEBUG "0.0024666666666663655" 07:25:23 DEBUG } 07:25:23 DEBUG ] 07:25:23 DEBUG "0" 07:25:23 DEBUG 1712816723, 07:25:23 DEBUG ] 07:25:23 DEBUG } 07:25:23 DEBUG ] 07:25:23 DEBUG "36441.166666666664" 07:25:23 DEBUG } 07:25:23 DEBUG } 07:25:23 DEBUG } 07:25:23 DEBUG ] 07:25:23 DEBUG ] 07:25:23 DEBUG ] 07:25:23 DEBUG } 07:25:23 DEBUG } 07:25:23 DEBUG } 07:25:23 DEBUG ] 07:25:23 DEBUG } 07:25:23 DEBUG } 07:25:23 DEBUG } 07:25:23 DEBUG } 07:25:23 INFO [http_cmd]: http status code OK 07:25:23 DEBUG --- status code --- 07:25:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:25:23 DEBUG --- http response --- 07:25:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 07:25:23 DEBUG ----- output ----- 07:25:23 DEBUG { 07:25:23 DEBUG "status": "success", 07:25:23 DEBUG "data": { 07:25:23 DEBUG "resultType": "vector", 07:25:23 DEBUG "result": [ 07:25:23 DEBUG { 07:25:23 DEBUG "metric": { 07:25:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:25:23 DEBUG }, 07:25:23 DEBUG "value": [ 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG [...] 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:25:23 DEBUG }, 07:25:23 DEBUG "value": [ 07:25:23 DEBUG 1712816723, 07:25:23 DEBUG "0" 07:25:23 DEBUG ] 07:25:23 DEBUG } 07:25:23 DEBUG ] 07:25:23 DEBUG } 07:25:23 DEBUG } 07:25:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816723" 07:25:23 INFO [http_cmd]: http status code OK 07:25:23 DEBUG --- status code --- 07:25:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:25:23 DEBUG --- http response --- 07:25:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:25:23 DEBUG ----- output ----- 07:25:23 DEBUG { 07:25:23 DEBUG "status": "success", 07:25:23 DEBUG "data": { 07:25:23 DEBUG "resultType": "vector", 07:25:23 DEBUG "result": [ 07:25:23 DEBUG { 07:25:23 DEBUG "metric": { 07:25:23 DEBUG "instance": "10.142.0.107:9100" 07:25:23 DEBUG }, 07:25:23 DEBUG "value": [ 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG [...] 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG "instance": "10.142.0.77:9100" 07:25:23 DEBUG }, 07:25:23 DEBUG "value": [ 07:25:23 DEBUG 1712816723, 07:25:23 DEBUG "53011317760" 07:25:23 DEBUG ] 07:25:23 DEBUG } 07:25:23 DEBUG ] 07:25:23 DEBUG } 07:25:23 DEBUG } 07:25:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816723" 07:25:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816723" 07:25:23 INFO [http_cmd]: http status code OK 07:25:23 DEBUG --- status code --- 07:25:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:25:23 DEBUG --- http response --- 07:25:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:25:23 DEBUG ----- output ----- 07:25:23 DEBUG { 07:25:23 DEBUG "status": "success", 07:25:23 DEBUG "data": { 07:25:23 DEBUG "resultType": "vector", 07:25:23 DEBUG "result": [ 07:25:23 DEBUG { 07:25:23 DEBUG "metric": { 07:25:23 DEBUG "pod": "ds-idrepo-2" 07:25:23 DEBUG }, 07:25:23 DEBUG "value": [ 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG [...] 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG "pod": "ds-cts-2" 07:25:23 DEBUG }, 07:25:23 DEBUG "value": [ 07:25:23 DEBUG 1712816723, 07:25:23 DEBUG "29.8" 07:25:23 DEBUG ] 07:25:23 DEBUG } 07:25:23 DEBUG ] 07:25:23 DEBUG } 07:25:23 DEBUG } 07:25:23 INFO [http_cmd]: http status code OK 07:25:23 DEBUG --- status code --- 07:25:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:25:23 DEBUG --- http response --- 07:25:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 07:25:23 DEBUG ----- output ----- 07:25:23 DEBUG { 07:25:23 DEBUG "status": "success", 07:25:23 DEBUG "data": { 07:25:23 DEBUG "resultType": "vector", 07:25:23 DEBUG "result": [ 07:25:23 DEBUG { 07:25:23 DEBUG "metric": { 07:25:23 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:25:23 DEBUG }, 07:25:23 DEBUG "value": [ 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG [...] 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG 07:25:23 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:25:23 DEBUG }, 07:25:23 DEBUG "value": [ 07:25:23 DEBUG 1712816723, 07:25:23 DEBUG "0" 07:25:23 DEBUG ] 07:25:23 DEBUG } 07:25:23 DEBUG ] 07:25:23 DEBUG } 07:25:23 DEBUG } 07:25:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816724" 07:25:24 INFO [http_cmd]: http status code OK 07:25:24 DEBUG --- status code --- 07:25:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:25:24 DEBUG --- http response --- 07:25:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:25:24 DEBUG ----- output ----- 07:25:24 DEBUG { 07:25:24 DEBUG "status": "success", 07:25:24 DEBUG "data": { 07:25:24 DEBUG "resultType": "vector", 07:25:24 DEBUG "result": [ 07:25:24 DEBUG { 07:25:24 DEBUG "metric": { 07:25:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:25:24 DEBUG }, 07:25:24 DEBUG "value": [ 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG [...] 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:25:24 DEBUG }, 07:25:24 DEBUG "value": [ 07:25:24 DEBUG 1712816724, 07:25:24 DEBUG "0" 07:25:24 DEBUG ] 07:25:24 DEBUG } 07:25:24 DEBUG ] 07:25:24 DEBUG } 07:25:24 DEBUG } 07:25:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816724" 07:25:24 INFO [http_cmd]: http status code OK 07:25:24 DEBUG --- status code --- 07:25:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:25:24 DEBUG --- http response --- 07:25:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:25:24 DEBUG ----- output ----- 07:25:24 DEBUG { 07:25:24 DEBUG "status": "success", 07:25:24 DEBUG "data": { 07:25:24 DEBUG "resultType": "vector", 07:25:24 DEBUG "result": [ 07:25:24 DEBUG { 07:25:24 DEBUG "metric": { 07:25:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:25:24 DEBUG }, 07:25:24 DEBUG "value": [ 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG [...] 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:25:24 DEBUG }, 07:25:24 DEBUG "value": [ 07:25:24 DEBUG 1712816724, 07:25:24 DEBUG "0" 07:25:24 DEBUG ] 07:25:24 DEBUG } 07:25:24 DEBUG ] 07:25:24 DEBUG } 07:25:24 DEBUG } 07:25:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816724" 07:25:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816724" 07:25:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816724" 07:25:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816724" 07:25:24 INFO [http_cmd]: http status code OK 07:25:24 DEBUG --- status code --- 07:25:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:25:24 DEBUG --- http response --- 07:25:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:25:24 DEBUG ----- output ----- 07:25:24 DEBUG { 07:25:24 DEBUG "status": "success", 07:25:24 DEBUG "data": { 07:25:24 DEBUG "resultType": "vector", 07:25:24 DEBUG "result": [ 07:25:24 DEBUG { 07:25:24 DEBUG "metric": { 07:25:24 DEBUG "pod": "ds-idrepo-2" 07:25:24 DEBUG }, 07:25:24 DEBUG "value": [ 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG [...] 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG "pod": "ds-idrepo-0" 07:25:24 DEBUG }, 07:25:24 DEBUG "value": [ 07:25:24 DEBUG 1712816724, 07:25:24 DEBUG "0" 07:25:24 DEBUG ] 07:25:24 DEBUG } 07:25:24 DEBUG ] 07:25:24 DEBUG } 07:25:24 DEBUG } 07:25:24 INFO [http_cmd]: http status code OK 07:25:24 DEBUG --- status code --- 07:25:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:25:24 DEBUG --- http response --- 07:25:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:25:24 DEBUG ----- output ----- 07:25:24 DEBUG { 07:25:24 DEBUG "status": "success", 07:25:24 DEBUG "data": { 07:25:24 DEBUG "resultType": "vector", 07:25:24 DEBUG "result": [ 07:25:24 DEBUG { 07:25:24 DEBUG "metric": { 07:25:24 DEBUG "pod": "ds-idrepo-2" 07:25:24 DEBUG }, 07:25:24 DEBUG "value": [ 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG [...] 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG "pod": "ds-cts-2" 07:25:24 DEBUG }, 07:25:24 DEBUG "value": [ 07:25:24 DEBUG 1712816724, 07:25:24 DEBUG "0" 07:25:24 DEBUG ] 07:25:24 DEBUG } 07:25:24 DEBUG ] 07:25:24 DEBUG } 07:25:24 DEBUG } 07:25:24 INFO [http_cmd]: http status code OK 07:25:24 DEBUG --- status code --- 07:25:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:25:24 DEBUG --- http response --- 07:25:24 INFO [http_cmd]: http status code OK 07:25:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:25:24 DEBUG --- status code --- 07:25:24 DEBUG ----- output ----- 07:25:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:25:24 DEBUG { 07:25:24 DEBUG --- http response --- 07:25:24 DEBUG "status": "success", 07:25:24 DEBUG "data": { 07:25:24 DEBUG "resultType": "vector", 07:25:24 DEBUG "result": [ 07:25:24 DEBUG { 07:25:24 DEBUG "metric": { 07:25:24 DEBUG "pod": "ds-idrepo-2" 07:25:24 DEBUG }, 07:25:24 DEBUG "value": [ 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG [...] 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG "pod": "ds-cts-2" 07:25:24 DEBUG }, 07:25:24 DEBUG "value": [ 07:25:24 DEBUG 1712816724, 07:25:24 DEBUG "0" 07:25:24 DEBUG ] 07:25:24 DEBUG } 07:25:24 DEBUG ] 07:25:24 DEBUG } 07:25:24 DEBUG } 07:25:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:25:24 DEBUG ----- output ----- 07:25:24 DEBUG { 07:25:24 DEBUG "status": "success", 07:25:24 DEBUG "data": { 07:25:24 DEBUG "resultType": "vector", 07:25:24 DEBUG "result": [ 07:25:24 DEBUG { 07:25:24 DEBUG "metric": { 07:25:24 DEBUG "instance": "10.142.0.107:9100" 07:25:24 DEBUG }, 07:25:24 DEBUG "value": [ 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG [...] 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG "instance": "10.142.0.77:9100" 07:25:24 DEBUG }, 07:25:24 DEBUG "value": [ 07:25:24 DEBUG 1712816724, 07:25:24 DEBUG "36480.566666666666" 07:25:24 DEBUG ] 07:25:24 DEBUG } 07:25:24 DEBUG ] 07:25:24 DEBUG } 07:25:24 DEBUG } 07:25:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816724" 07:25:24 INFO [http_cmd]: http status code OK 07:25:24 DEBUG --- status code --- 07:25:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:25:24 DEBUG --- http response --- 07:25:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:25:24 DEBUG ----- output ----- 07:25:24 DEBUG { 07:25:24 DEBUG "status": "success", 07:25:24 DEBUG "data": { 07:25:24 DEBUG "resultType": "vector", 07:25:24 DEBUG "result": [ 07:25:24 DEBUG { 07:25:24 DEBUG "metric": { 07:25:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:25:24 DEBUG }, 07:25:24 DEBUG "value": [ 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG [...] 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:25:24 DEBUG }, 07:25:24 DEBUG "value": [ 07:25:24 DEBUG 1712816724, 07:25:24 DEBUG "0" 07:25:24 DEBUG ] 07:25:24 DEBUG } 07:25:24 DEBUG ] 07:25:24 DEBUG } 07:25:24 DEBUG } 07:25:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712816724" 07:25:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816724" 07:25:24 INFO [http_cmd]: http status code OK 07:25:24 DEBUG --- status code --- 07:25:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:25:24 DEBUG --- http response --- 07:25:24 INFO [http_cmd]: http status code OK 07:25:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:25:24 DEBUG --- status code --- 07:25:24 DEBUG ----- output ----- 07:25:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:25:24 DEBUG { 07:25:24 DEBUG --- http response --- 07:25:24 DEBUG "status": "success", 07:25:24 DEBUG "data": { 07:25:24 DEBUG "resultType": "vector", 07:25:24 DEBUG "result": [ 07:25:24 DEBUG { 07:25:24 DEBUG "metric": { 07:25:24 DEBUG "instance": "10.142.0.107:9100" 07:25:24 DEBUG }, 07:25:24 DEBUG "value": [ 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG [...] 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG "instance": "10.142.0.77:9100" 07:25:24 DEBUG }, 07:25:24 DEBUG "value": [ 07:25:24 DEBUG 1712816724, 07:25:24 DEBUG "0" 07:25:24 DEBUG ] 07:25:24 DEBUG } 07:25:24 DEBUG ] 07:25:24 DEBUG } 07:25:24 DEBUG } 07:25:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:25:24 DEBUG ----- output ----- 07:25:24 DEBUG { 07:25:24 DEBUG "status": "success", 07:25:24 DEBUG "data": { 07:25:24 DEBUG "resultType": "vector", 07:25:24 DEBUG "result": [ 07:25:24 DEBUG { 07:25:24 DEBUG "metric": { 07:25:24 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:25:24 DEBUG }, 07:25:24 DEBUG "value": [ 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG [...] 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 07:25:24 DEBUG }, 07:25:24 DEBUG "value": [ 07:25:24 DEBUG 1712816724, 07:25:24 DEBUG "12927877120" 07:25:24 DEBUG ] 07:25:24 DEBUG } 07:25:24 DEBUG ] 07:25:24 DEBUG } 07:25:24 DEBUG } 07:25:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816724" 07:25:24 INFO [http_cmd]: http status code OK 07:25:24 DEBUG --- status code --- 07:25:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:25:24 DEBUG --- http response --- 07:25:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:25:24 DEBUG ----- output ----- 07:25:24 DEBUG { 07:25:24 DEBUG "status": "success", 07:25:24 DEBUG "data": { 07:25:24 DEBUG "resultType": "vector", 07:25:24 DEBUG "result": [ 07:25:24 DEBUG { 07:25:24 DEBUG "metric": { 07:25:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:25:24 DEBUG }, 07:25:24 DEBUG "value": [ 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG [...] 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG 07:25:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:25:24 DEBUG }, 07:25:24 DEBUG "value": [ 07:25:24 DEBUG 1712816724, 07:25:24 DEBUG "0" 07:25:24 DEBUG ] 07:25:24 DEBUG } 07:25:24 DEBUG ] 07:25:24 DEBUG } 07:25:24 DEBUG } 07:25:37 INFO 07:25:37 INFO [loop_until]: kubectl --namespace=xlou top pods 07:25:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:25:37 INFO [loop_until]: OK (rc = 0) 07:25:37 DEBUG --- stdout --- 07:25:37 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 7m 8165Mi am-5b5bd8567b-8h2w4 6m 8661Mi am-5b5bd8567b-qlkw6 6m 8416Mi ds-cts-0 17m 12242Mi ds-cts-1 18m 13308Mi ds-cts-2 19m 13437Mi ds-idrepo-0 13m 5525Mi ds-idrepo-1 11m 5483Mi ds-idrepo-2 13m 5784Mi end-user-ui-58878d7668-gjq6d 1m 5Mi idm-668d898cfd-2vqdm 4m 1010Mi idm-668d898cfd-st2t7 4m 1136Mi lodemon-ccc964465-p7gdm 2m 68Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 1007m 1688Mi 07:25:37 DEBUG --- stderr --- 07:25:37 DEBUG 07:25:41 INFO 07:25:41 INFO [loop_until]: kubectl --namespace=xlou top node 07:25:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:25:41 INFO [loop_until]: OK (rc = 0) 07:25:41 DEBUG --- stdout --- 07:25:41 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 2743Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 65m 0% 2786Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 54m 0% 10108Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 124m 0% 3048Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 60m 0% 9815Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 147m 0% 3854Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 65m 0% 10317Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 67m 0% 6478Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 75m 0% 6240Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 79m 0% 14096Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 74m 0% 12929Mi 22% gke-xlou-cdm-ds-32e4dcb1-j2hk 72m 0% 14002Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 66m 0% 6177Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 1085m 6% 5562Mi 9% 07:25:41 DEBUG --- stderr --- 07:25:41 DEBUG 07:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816783" 07:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816783" 07:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816783" 07:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816783" 07:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712816783" 07:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816783" 07:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816783" 07:26:23 INFO [http_cmd]: http status code OK 07:26:23 INFO [http_cmd]: http status code OK 07:26:23 INFO [http_cmd]: http status code OK 07:26:23 DEBUG --- status code --- 07:26:23 INFO [http_cmd]: http status code OK 07:26:23 DEBUG --- status code --- 07:26:23 DEBUG --- status code --- 07:26:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:26:23 DEBUG --- status code --- 07:26:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:26:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:26:23 DEBUG --- http response --- 07:26:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:26:23 DEBUG --- http response --- 07:26:23 DEBUG --- http response --- 07:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:26:23 DEBUG --- http response --- 07:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:26:23 DEBUG ----- output ----- 07:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:26:23 DEBUG ----- output ----- 07:26:23 DEBUG ----- output ----- 07:26:23 DEBUG { 07:26:23 DEBUG ----- output ----- 07:26:23 DEBUG { 07:26:23 DEBUG { 07:26:23 DEBUG "status": "success", 07:26:23 DEBUG { 07:26:23 DEBUG "status": "success", 07:26:23 DEBUG "status": "success", 07:26:23 DEBUG "data": { 07:26:23 DEBUG "status": "success", 07:26:23 DEBUG "data": { 07:26:23 DEBUG "data": { 07:26:23 DEBUG "resultType": "vector", 07:26:23 DEBUG "data": { 07:26:23 DEBUG "resultType": "vector", 07:26:23 DEBUG "resultType": "vector", 07:26:23 DEBUG "result": [ 07:26:23 DEBUG "resultType": "vector", 07:26:23 DEBUG "result": [ 07:26:23 DEBUG "result": [ 07:26:23 DEBUG { 07:26:23 DEBUG "result": [ 07:26:23 DEBUG { 07:26:23 DEBUG { 07:26:23 DEBUG "metric": { 07:26:23 DEBUG { 07:26:23 DEBUG "metric": { 07:26:23 DEBUG "metric": { 07:26:23 DEBUG "pod": "ds-idrepo-2" 07:26:23 DEBUG "metric": { 07:26:23 DEBUG "pod": "ds-idrepo-2" 07:26:23 DEBUG "instance": "10.142.0.107:9100" 07:26:23 DEBUG }, 07:26:23 DEBUG "instance": "10.142.0.107:9100" 07:26:23 DEBUG }, 07:26:23 DEBUG }, 07:26:23 DEBUG "value": [ 07:26:23 DEBUG }, 07:26:23 DEBUG "value": [ 07:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712816783" 07:26:23 DEBUG "value": [ 07:26:23 DEBUG 07:26:23 DEBUG "value": [ 07:26:23 DEBUG 07:26:23 DEBUG 07:26:23 DEBUG 07:26:23 DEBUG 07:26:23 DEBUG 07:26:23 DEBUG 07:26:23 DEBUG 07:26:23 DEBUG 07:26:23 DEBUG 07:26:23 DEBUG 07:26:23 DEBUG [...] 07:26:23 DEBUG 07:26:23 DEBUG [...] 07:26:23 DEBUG [...] 07:26:23 DEBUG 07:26:23 DEBUG [...] 07:26:23 DEBUG 07:26:23 DEBUG 07:26:23 DEBUG 07:26:23 DEBUG 07:26:23 DEBUG 07:26:23 DEBUG 07:26:23 DEBUG 07:26:23 DEBUG 07:26:23 DEBUG 07:26:23 DEBUG 07:26:23 DEBUG "pod": "ds-idrepo-0" 07:26:23 DEBUG 07:26:23 DEBUG "pod": "ds-idrepo-0" 07:26:23 DEBUG "instance": "10.142.0.77:9100" 07:26:23 DEBUG }, 07:26:23 DEBUG "instance": "10.142.0.77:9100" 07:26:23 DEBUG }, 07:26:23 DEBUG }, 07:26:23 DEBUG "value": [ 07:26:23 DEBUG }, 07:26:23 DEBUG "value": [ 07:26:23 DEBUG "value": [ 07:26:23 DEBUG 1712816783, 07:26:23 DEBUG 1712816783, 07:26:23 DEBUG 1712816783, 07:26:23 DEBUG "0" 07:26:23 DEBUG "value": [ 07:26:23 DEBUG "0.00290000000000011" 07:26:23 DEBUG "34598.26666666666" 07:26:23 DEBUG ] 07:26:23 DEBUG 1712816783, 07:26:23 DEBUG ] 07:26:23 DEBUG ] 07:26:23 DEBUG } 07:26:23 DEBUG "0" 07:26:23 DEBUG } 07:26:23 DEBUG } 07:26:23 DEBUG ] 07:26:23 DEBUG ] 07:26:23 DEBUG ] 07:26:23 DEBUG ] 07:26:23 DEBUG } 07:26:23 DEBUG } 07:26:23 DEBUG } 07:26:23 DEBUG } 07:26:23 DEBUG } 07:26:23 DEBUG ] 07:26:23 DEBUG } 07:26:23 DEBUG } 07:26:23 DEBUG } 07:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816783" 07:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712816783" 07:26:23 INFO [http_cmd]: http status code OK 07:26:23 ERROR [http_cmd]: ERROR 07:26:23 DEBUG } 07:26:23 INFO [http_cmd]: http status code OK 07:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712816783" 07:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816783" 07:26:23 DEBUG --- status code --- 07:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816783" 07:26:23 ERROR --- status code --- 07:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816783" 07:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816783" 07:26:23 DEBUG --- status code --- 07:26:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:26:23 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816783" 07:26:23 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816783" 07:26:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:26:23 DEBUG --- http response --- 07:26:23 ERROR --- http response --- 07:26:23 INFO [http_cmd]: http status code OK 07:26:23 DEBUG --- http response --- 07:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:26:23 DEBUG ----- output ----- 07:26:23 DEBUG --- status code --- 07:26:23 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:26:23 DEBUG { 07:26:23 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:26:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:26:23 DEBUG --- http response --- 07:26:23 DEBUG "status": "success", 07:26:23 INFO [http_cmd]: http status code OK 07:26:23 INFO [http_cmd]: http status code OK 07:26:23 INFO [http_cmd]: http status code OK 07:26:23 INFO [http_cmd]: http status code OK 07:26:23 INFO [http_cmd]: http status code OK 07:26:23 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:26:23 DEBUG "data": { 07:26:23 DEBUG --- status code --- 07:26:23 DEBUG --- status code --- 07:26:23 DEBUG --- status code --- 07:26:23 DEBUG --- status code --- 07:26:23 DEBUG --- status code --- 07:26:23 DEBUG ----- output ----- 07:26:23 DEBUG "resultType": "vector", 07:26:23 INFO [http_cmd]: http status code OK 07:26:23 INFO [http_cmd]: http status code OK 07:26:23 INFO [http_cmd]: http status code OK 07:26:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:26:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:26:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:26:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:26:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:26:23 DEBUG { 07:26:23 DEBUG "result": [ 07:26:23 DEBUG --- status code --- 07:26:23 DEBUG --- status code --- 07:26:23 DEBUG --- status code --- 07:26:23 DEBUG --- http response --- 07:26:23 DEBUG --- http response --- 07:26:23 DEBUG --- http response --- 07:26:23 DEBUG --- http response --- 07:26:23 DEBUG --- http response --- 07:26:23 DEBUG "status": "success", 07:26:23 DEBUG { 07:26:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:26:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:26:23 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:26:23 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:26:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:26:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 07:26:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:26:24 DEBUG "data": { 07:26:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:26:24 DEBUG "metric": { 07:26:24 DEBUG --- http response --- 07:26:24 DEBUG --- http response --- 07:26:24 DEBUG --- http response --- 07:26:24 DEBUG ----- output ----- 07:26:24 DEBUG ----- output ----- 07:26:24 DEBUG ----- output ----- 07:26:24 DEBUG ----- output ----- 07:26:24 DEBUG "resultType": "vector", 07:26:24 DEBUG ----- output ----- 07:26:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:26:24 INFO [http_cmd]: http status code OK 07:26:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:26:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:26:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:26:24 DEBUG { 07:26:24 DEBUG { 07:26:24 DEBUG { 07:26:24 DEBUG { 07:26:24 DEBUG "result": [ 07:26:24 DEBUG { 07:26:24 DEBUG }, 07:26:24 DEBUG --- status code --- 07:26:24 DEBUG ----- output ----- 07:26:24 DEBUG ----- output ----- 07:26:24 DEBUG ----- output ----- 07:26:24 DEBUG "status": "success", 07:26:24 DEBUG "status": "success", 07:26:24 DEBUG "status": "success", 07:26:24 DEBUG "status": "success", 07:26:24 DEBUG { 07:26:24 DEBUG "status": "success", 07:26:24 DEBUG "value": [ 07:26:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:26:24 DEBUG { 07:26:24 DEBUG { 07:26:24 DEBUG { 07:26:24 DEBUG "data": { 07:26:24 DEBUG "data": { 07:26:24 DEBUG "data": { 07:26:24 DEBUG "data": { 07:26:24 DEBUG "metric": { 07:26:24 DEBUG "data": { 07:26:24 DEBUG 07:26:24 DEBUG --- http response --- 07:26:24 DEBUG "status": "success", 07:26:24 DEBUG "status": "success", 07:26:24 DEBUG "status": "success", 07:26:24 DEBUG "resultType": "vector", 07:26:24 DEBUG "resultType": "vector", 07:26:24 DEBUG "resultType": "vector", 07:26:24 DEBUG "resultType": "vector", 07:26:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:26:24 DEBUG "resultType": "vector", 07:26:24 DEBUG 07:26:24 DEBUG "data": { 07:26:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 07:26:24 DEBUG "data": { 07:26:24 DEBUG "data": { 07:26:24 DEBUG "result": [ 07:26:24 DEBUG "result": [ 07:26:24 DEBUG "result": [ 07:26:24 DEBUG "result": [ 07:26:24 DEBUG }, 07:26:24 DEBUG "result": [ 07:26:24 DEBUG 07:26:24 DEBUG "resultType": "vector", 07:26:24 DEBUG ----- output ----- 07:26:24 DEBUG "resultType": "vector", 07:26:24 DEBUG "resultType": "vector", 07:26:24 DEBUG { 07:26:24 DEBUG { 07:26:24 DEBUG { 07:26:24 DEBUG { 07:26:24 DEBUG "value": [ 07:26:24 DEBUG { 07:26:24 DEBUG [...] 07:26:24 DEBUG "result": [ 07:26:24 DEBUG { 07:26:24 DEBUG "result": [ 07:26:24 DEBUG "result": [ 07:26:24 DEBUG "metric": { 07:26:24 DEBUG "metric": { 07:26:24 DEBUG "metric": { 07:26:24 DEBUG "metric": { 07:26:24 DEBUG 07:26:24 DEBUG "metric": { 07:26:24 DEBUG 07:26:24 DEBUG { 07:26:24 DEBUG "status": "success", 07:26:24 DEBUG { 07:26:24 DEBUG { 07:26:24 DEBUG "pod": "ds-idrepo-2" 07:26:24 DEBUG "pod": "ds-idrepo-2" 07:26:24 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:26:24 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:26:24 DEBUG 07:26:24 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:26:24 DEBUG 07:26:24 DEBUG "metric": { 07:26:24 DEBUG "data": { 07:26:24 DEBUG "metric": { 07:26:24 DEBUG "metric": { 07:26:24 DEBUG }, 07:26:24 DEBUG }, 07:26:24 DEBUG }, 07:26:24 DEBUG }, 07:26:24 DEBUG 07:26:24 DEBUG }, 07:26:24 DEBUG 07:26:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:26:24 DEBUG "resultType": "vector", 07:26:24 DEBUG "instance": "10.142.0.107:9100" 07:26:24 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:26:24 DEBUG "value": [ 07:26:24 DEBUG "value": [ 07:26:24 DEBUG "value": [ 07:26:24 DEBUG "value": [ 07:26:24 DEBUG [...] 07:26:24 DEBUG "value": [ 07:26:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:26:24 DEBUG }, 07:26:24 DEBUG "result": [ 07:26:24 DEBUG }, 07:26:24 DEBUG }, 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG }, 07:26:24 DEBUG "value": [ 07:26:24 DEBUG { 07:26:24 DEBUG "value": [ 07:26:24 DEBUG "value": [ 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG "value": [ 07:26:24 DEBUG 07:26:24 DEBUG "metric": { 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 1712816783, 07:26:24 DEBUG 07:26:24 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG [...] 07:26:24 DEBUG [...] 07:26:24 DEBUG [...] 07:26:24 DEBUG [...] 07:26:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:26:24 DEBUG [...] 07:26:24 DEBUG "0" 07:26:24 DEBUG 07:26:24 DEBUG }, 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG }, 07:26:24 DEBUG 07:26:24 DEBUG ] 07:26:24 DEBUG [...] 07:26:24 DEBUG "value": [ 07:26:24 DEBUG [...] 07:26:24 DEBUG [...] 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG "value": [ 07:26:24 DEBUG 07:26:24 DEBUG } 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 1712816783, 07:26:24 DEBUG 07:26:24 DEBUG ] 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG "pod": "ds-cts-2" 07:26:24 DEBUG "pod": "ds-cts-2" 07:26:24 DEBUG "pod": "ds-idrepo-1" 07:26:24 DEBUG "pod": "ds-idrepo-1" 07:26:24 DEBUG "0.026" 07:26:24 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:26:24 DEBUG } 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG }, 07:26:24 DEBUG }, 07:26:24 DEBUG }, 07:26:24 DEBUG }, 07:26:24 DEBUG ] 07:26:24 DEBUG }, 07:26:24 DEBUG } 07:26:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:26:24 DEBUG [...] 07:26:24 DEBUG "instance": "10.142.0.77:9100" 07:26:24 DEBUG "pod": "ds-idrepo-1" 07:26:24 DEBUG "value": [ 07:26:24 DEBUG "value": [ 07:26:24 DEBUG "value": [ 07:26:24 DEBUG "value": [ 07:26:24 DEBUG } 07:26:24 DEBUG "value": [ 07:26:24 DEBUG }, 07:26:24 DEBUG 07:26:24 DEBUG }, 07:26:24 DEBUG }, 07:26:24 DEBUG 1712816783, 07:26:24 DEBUG 1712816783, 07:26:24 DEBUG 1712816783, 07:26:24 DEBUG 1712816783, 07:26:24 DEBUG ] 07:26:24 DEBUG 1712816783, 07:26:24 DEBUG "value": [ 07:26:24 DEBUG 07:26:24 DEBUG "value": [ 07:26:24 DEBUG "value": [ 07:26:24 DEBUG "0" 07:26:24 DEBUG "29.233333333333334" 07:26:24 DEBUG "3116.5525351041015" 07:26:24 DEBUG "0.010442395241032245" 07:26:24 DEBUG } 07:26:24 DEBUG "1.0300880766387455" 07:26:24 DEBUG 1712816783, 07:26:24 DEBUG 07:26:24 DEBUG 1712816783, 07:26:24 DEBUG 1712816783, 07:26:24 DEBUG ] 07:26:24 DEBUG ] 07:26:24 DEBUG ] 07:26:24 DEBUG ] 07:26:24 DEBUG } 07:26:24 DEBUG ] 07:26:24 DEBUG "0" 07:26:24 DEBUG "pod": "ds-idrepo-1" 07:26:24 DEBUG "53013062656" 07:26:24 DEBUG "849.9688732102095" 07:26:24 DEBUG } 07:26:24 DEBUG } 07:26:24 DEBUG } 07:26:24 DEBUG } 07:26:24 DEBUG } 07:26:24 DEBUG ] 07:26:24 DEBUG }, 07:26:24 DEBUG ] 07:26:24 DEBUG ] 07:26:24 DEBUG ] 07:26:24 DEBUG ] 07:26:24 DEBUG ] 07:26:24 DEBUG ] 07:26:24 DEBUG ] 07:26:24 DEBUG } 07:26:24 DEBUG "value": [ 07:26:24 DEBUG } 07:26:24 DEBUG } 07:26:24 DEBUG } 07:26:24 DEBUG } 07:26:24 DEBUG } 07:26:24 DEBUG } 07:26:24 DEBUG } 07:26:24 DEBUG ] 07:26:24 DEBUG 1712816783, 07:26:24 DEBUG ] 07:26:24 DEBUG ] 07:26:24 DEBUG } 07:26:24 DEBUG } 07:26:24 DEBUG } 07:26:24 DEBUG } 07:26:24 DEBUG } 07:26:24 DEBUG } 07:26:24 DEBUG "0" 07:26:24 DEBUG } 07:26:24 DEBUG } 07:26:24 DEBUG } 07:26:24 DEBUG ] 07:26:24 DEBUG } 07:26:24 DEBUG } 07:26:24 DEBUG } 07:26:24 DEBUG ] 07:26:24 DEBUG } 07:26:24 DEBUG } 07:26:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816784" 07:26:24 INFO [http_cmd]: http status code OK 07:26:24 DEBUG --- status code --- 07:26:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:26:24 DEBUG --- http response --- 07:26:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:26:24 DEBUG ----- output ----- 07:26:24 DEBUG { 07:26:24 DEBUG "status": "success", 07:26:24 DEBUG "data": { 07:26:24 DEBUG "resultType": "vector", 07:26:24 DEBUG "result": [ 07:26:24 DEBUG { 07:26:24 DEBUG "metric": { 07:26:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:26:24 DEBUG }, 07:26:24 DEBUG "value": [ 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG [...] 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:26:24 DEBUG }, 07:26:24 DEBUG "value": [ 07:26:24 DEBUG 1712816784, 07:26:24 DEBUG "0" 07:26:24 DEBUG ] 07:26:24 DEBUG } 07:26:24 DEBUG ] 07:26:24 DEBUG } 07:26:24 DEBUG } 07:26:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816784" 07:26:24 INFO [http_cmd]: http status code OK 07:26:24 DEBUG --- status code --- 07:26:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:26:24 DEBUG --- http response --- 07:26:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:26:24 DEBUG ----- output ----- 07:26:24 DEBUG { 07:26:24 DEBUG "status": "success", 07:26:24 DEBUG "data": { 07:26:24 DEBUG "resultType": "vector", 07:26:24 DEBUG "result": [ 07:26:24 DEBUG { 07:26:24 DEBUG "metric": { 07:26:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:26:24 DEBUG }, 07:26:24 DEBUG "value": [ 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG [...] 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:26:24 DEBUG }, 07:26:24 DEBUG "value": [ 07:26:24 DEBUG 1712816784, 07:26:24 DEBUG "0" 07:26:24 DEBUG ] 07:26:24 DEBUG } 07:26:24 DEBUG ] 07:26:24 DEBUG } 07:26:24 DEBUG } 07:26:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816784" 07:26:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816784" 07:26:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816784" 07:26:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816784" 07:26:24 INFO [http_cmd]: http status code OK 07:26:24 DEBUG --- status code --- 07:26:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:26:24 DEBUG --- http response --- 07:26:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:26:24 DEBUG ----- output ----- 07:26:24 DEBUG { 07:26:24 DEBUG "status": "success", 07:26:24 DEBUG "data": { 07:26:24 DEBUG "resultType": "vector", 07:26:24 DEBUG "result": [ 07:26:24 DEBUG { 07:26:24 DEBUG "metric": { 07:26:24 DEBUG "pod": "ds-idrepo-2" 07:26:24 DEBUG }, 07:26:24 DEBUG "value": [ 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG [...] 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG "pod": "ds-idrepo-0" 07:26:24 DEBUG }, 07:26:24 DEBUG "value": [ 07:26:24 DEBUG 1712816784, 07:26:24 DEBUG "0" 07:26:24 DEBUG ] 07:26:24 DEBUG } 07:26:24 DEBUG ] 07:26:24 DEBUG } 07:26:24 DEBUG } 07:26:24 INFO [http_cmd]: http status code OK 07:26:24 DEBUG --- status code --- 07:26:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:26:24 DEBUG --- http response --- 07:26:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:26:24 DEBUG ----- output ----- 07:26:24 DEBUG { 07:26:24 DEBUG "status": "success", 07:26:24 DEBUG "data": { 07:26:24 DEBUG "resultType": "vector", 07:26:24 DEBUG "result": [ 07:26:24 DEBUG { 07:26:24 DEBUG "metric": { 07:26:24 DEBUG "pod": "ds-idrepo-2" 07:26:24 DEBUG }, 07:26:24 DEBUG "value": [ 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG [...] 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG "pod": "ds-cts-2" 07:26:24 DEBUG }, 07:26:24 DEBUG "value": [ 07:26:24 DEBUG 1712816784, 07:26:24 INFO [http_cmd]: http status code OK 07:26:24 DEBUG "0" 07:26:24 DEBUG --- status code --- 07:26:24 DEBUG ] 07:26:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:26:24 DEBUG } 07:26:24 DEBUG --- http response --- 07:26:24 DEBUG ] 07:26:24 DEBUG } 07:26:24 DEBUG } 07:26:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:26:24 DEBUG ----- output ----- 07:26:24 DEBUG { 07:26:24 DEBUG "status": "success", 07:26:24 DEBUG "data": { 07:26:24 DEBUG "resultType": "vector", 07:26:24 DEBUG "result": [ 07:26:24 DEBUG { 07:26:24 DEBUG "metric": { 07:26:24 DEBUG "pod": "ds-idrepo-2" 07:26:24 DEBUG }, 07:26:24 DEBUG "value": [ 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG [...] 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG "pod": "ds-cts-2" 07:26:24 DEBUG }, 07:26:24 DEBUG "value": [ 07:26:24 DEBUG 1712816784, 07:26:24 DEBUG "0" 07:26:24 DEBUG ] 07:26:24 DEBUG } 07:26:24 DEBUG ] 07:26:24 DEBUG } 07:26:24 DEBUG } 07:26:24 INFO [http_cmd]: http status code OK 07:26:24 DEBUG --- status code --- 07:26:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:26:24 DEBUG --- http response --- 07:26:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:26:24 DEBUG ----- output ----- 07:26:24 DEBUG { 07:26:24 DEBUG "status": "success", 07:26:24 DEBUG "data": { 07:26:24 DEBUG "resultType": "vector", 07:26:24 DEBUG "result": [ 07:26:24 DEBUG { 07:26:24 DEBUG "metric": { 07:26:24 DEBUG "instance": "10.142.0.107:9100" 07:26:24 DEBUG }, 07:26:24 DEBUG "value": [ 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG [...] 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG "instance": "10.142.0.77:9100" 07:26:24 DEBUG }, 07:26:24 DEBUG "value": [ 07:26:24 DEBUG 1712816784, 07:26:24 DEBUG "35357.36666666667" 07:26:24 DEBUG ] 07:26:24 DEBUG } 07:26:24 DEBUG ] 07:26:24 DEBUG } 07:26:24 DEBUG } 07:26:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816784" 07:26:24 INFO [http_cmd]: http status code OK 07:26:24 DEBUG --- status code --- 07:26:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:26:24 DEBUG --- http response --- 07:26:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:26:24 DEBUG ----- output ----- 07:26:24 DEBUG { 07:26:24 DEBUG "status": "success", 07:26:24 DEBUG "data": { 07:26:24 DEBUG "resultType": "vector", 07:26:24 DEBUG "result": [ 07:26:24 DEBUG { 07:26:24 DEBUG "metric": { 07:26:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:26:24 DEBUG }, 07:26:24 DEBUG "value": [ 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG [...] 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:26:24 DEBUG }, 07:26:24 DEBUG "value": [ 07:26:24 DEBUG 1712816784, 07:26:24 DEBUG "0" 07:26:24 DEBUG ] 07:26:24 DEBUG } 07:26:24 DEBUG ] 07:26:24 DEBUG } 07:26:24 DEBUG } 07:26:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816784" 07:26:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712816784" 07:26:24 INFO [http_cmd]: http status code OK 07:26:24 DEBUG --- status code --- 07:26:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:26:24 DEBUG --- http response --- 07:26:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:26:24 DEBUG ----- output ----- 07:26:24 DEBUG { 07:26:24 DEBUG "status": "success", 07:26:24 DEBUG "data": { 07:26:24 DEBUG "resultType": "vector", 07:26:24 DEBUG "result": [ 07:26:24 DEBUG { 07:26:24 DEBUG "metric": { 07:26:24 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:26:24 DEBUG }, 07:26:24 DEBUG "value": [ 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG [...] 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 07:26:24 DEBUG }, 07:26:24 DEBUG "value": [ 07:26:24 DEBUG 1712816784, 07:26:24 DEBUG "12928024576" 07:26:24 DEBUG ] 07:26:24 DEBUG } 07:26:24 DEBUG ] 07:26:24 INFO [http_cmd]: http status code OK 07:26:24 DEBUG } 07:26:24 DEBUG --- status code --- 07:26:24 DEBUG } 07:26:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:26:24 DEBUG --- http response --- 07:26:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:26:24 DEBUG ----- output ----- 07:26:24 DEBUG { 07:26:24 DEBUG "status": "success", 07:26:24 DEBUG "data": { 07:26:24 DEBUG "resultType": "vector", 07:26:24 DEBUG "result": [ 07:26:24 DEBUG { 07:26:24 DEBUG "metric": { 07:26:24 DEBUG "instance": "10.142.0.107:9100" 07:26:24 DEBUG }, 07:26:24 DEBUG "value": [ 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG [...] 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG "instance": "10.142.0.77:9100" 07:26:24 DEBUG }, 07:26:24 DEBUG "value": [ 07:26:24 DEBUG 1712816784, 07:26:24 DEBUG "0" 07:26:24 DEBUG ] 07:26:24 DEBUG } 07:26:24 DEBUG ] 07:26:24 DEBUG } 07:26:24 DEBUG } 07:26:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816784" 07:26:24 INFO [http_cmd]: http status code OK 07:26:24 DEBUG --- status code --- 07:26:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:26:24 DEBUG --- http response --- 07:26:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:26:24 DEBUG ----- output ----- 07:26:24 DEBUG { 07:26:24 DEBUG "status": "success", 07:26:24 DEBUG "data": { 07:26:24 DEBUG "resultType": "vector", 07:26:24 DEBUG "result": [ 07:26:24 DEBUG { 07:26:24 DEBUG "metric": { 07:26:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:26:24 DEBUG }, 07:26:24 DEBUG "value": [ 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG [...] 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG 07:26:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:26:24 DEBUG }, 07:26:24 DEBUG "value": [ 07:26:24 DEBUG 1712816784, 07:26:24 DEBUG "0" 07:26:24 DEBUG ] 07:26:24 DEBUG } 07:26:24 DEBUG ] 07:26:24 DEBUG } 07:26:24 DEBUG } 07:26:37 INFO 07:26:37 INFO [loop_until]: kubectl --namespace=xlou top pods 07:26:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:26:37 INFO [loop_until]: OK (rc = 0) 07:26:37 DEBUG --- stdout --- 07:26:37 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 5m 8165Mi am-5b5bd8567b-8h2w4 6m 8661Mi am-5b5bd8567b-qlkw6 16m 8416Mi ds-cts-0 15m 12242Mi ds-cts-1 17m 13308Mi ds-cts-2 17m 13437Mi ds-idrepo-0 13m 5525Mi ds-idrepo-1 10m 5483Mi ds-idrepo-2 10m 5783Mi end-user-ui-58878d7668-gjq6d 1m 5Mi idm-668d898cfd-2vqdm 4m 1011Mi idm-668d898cfd-st2t7 7m 1136Mi lodemon-ccc964465-p7gdm 6m 68Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 985m 2315Mi 07:26:37 DEBUG --- stderr --- 07:26:37 DEBUG 07:26:41 INFO 07:26:41 INFO [loop_until]: kubectl --namespace=xlou top node 07:26:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:26:41 INFO [loop_until]: OK (rc = 0) 07:26:41 DEBUG --- stdout --- 07:26:41 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 2739Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 70m 0% 2791Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 66m 0% 10107Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 118m 0% 3048Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 55m 0% 9803Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 131m 0% 3829Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 65m 0% 10319Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 63m 0% 6475Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 65m 0% 6239Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 78m 0% 14098Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 74m 0% 12928Mi 22% gke-xlou-cdm-ds-32e4dcb1-j2hk 76m 0% 13999Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 64m 0% 6171Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 1081m 6% 6135Mi 10% 07:26:41 DEBUG --- stderr --- 07:26:41 DEBUG 07:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816844" 07:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816844" 07:27:24 INFO [http_cmd]: http status code OK 07:27:24 DEBUG --- status code --- 07:27:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:27:24 DEBUG --- http response --- 07:27:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:27:24 DEBUG ----- output ----- 07:27:24 DEBUG { 07:27:24 DEBUG "status": "success", 07:27:24 DEBUG "data": { 07:27:24 DEBUG "resultType": "vector", 07:27:24 DEBUG "result": [ 07:27:24 DEBUG { 07:27:24 DEBUG "metric": { 07:27:24 DEBUG "pod": "ds-idrepo-2" 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG [...] 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG "pod": "ds-idrepo-0" 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 1712816844, 07:27:24 DEBUG "0" 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG } 07:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816844" 07:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816844" 07:27:24 INFO [http_cmd]: http status code OK 07:27:24 DEBUG --- status code --- 07:27:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:27:24 DEBUG --- http response --- 07:27:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:27:24 DEBUG ----- output ----- 07:27:24 DEBUG { 07:27:24 DEBUG "status": "success", 07:27:24 DEBUG "data": { 07:27:24 DEBUG "resultType": "vector", 07:27:24 DEBUG "result": [ 07:27:24 DEBUG { 07:27:24 DEBUG "metric": { 07:27:24 DEBUG "instance": "10.142.0.107:9100" 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG [...] 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG "instance": "10.142.0.77:9100" 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 1712816844, 07:27:24 DEBUG "34361.4" 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG } 07:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816844" 07:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816844" 07:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816844" 07:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816844" 07:27:24 INFO [http_cmd]: http status code OK 07:27:24 DEBUG --- status code --- 07:27:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:27:24 DEBUG --- http response --- 07:27:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:27:24 DEBUG ----- output ----- 07:27:24 DEBUG { 07:27:24 DEBUG "status": "success", 07:27:24 DEBUG "data": { 07:27:24 DEBUG "resultType": "vector", 07:27:24 DEBUG "result": [ 07:27:24 DEBUG { 07:27:24 DEBUG "metric": { 07:27:24 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG [...] 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 1712816844, 07:27:24 DEBUG "0.3415003789675564" 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG } 07:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712816844" 07:27:24 INFO [http_cmd]: http status code OK 07:27:24 INFO [http_cmd]: http status code OK 07:27:24 DEBUG --- status code --- 07:27:24 DEBUG --- status code --- 07:27:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:27:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:27:24 DEBUG --- http response --- 07:27:24 DEBUG --- http response --- 07:27:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:27:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:27:24 DEBUG ----- output ----- 07:27:24 DEBUG ----- output ----- 07:27:24 DEBUG { 07:27:24 DEBUG { 07:27:24 DEBUG "status": "success", 07:27:24 DEBUG "status": "success", 07:27:24 DEBUG "data": { 07:27:24 DEBUG "data": { 07:27:24 DEBUG "resultType": "vector", 07:27:24 DEBUG "resultType": "vector", 07:27:24 DEBUG "result": [ 07:27:24 DEBUG "result": [ 07:27:24 DEBUG { 07:27:24 DEBUG { 07:27:24 DEBUG "metric": { 07:27:24 DEBUG "metric": { 07:27:24 DEBUG "instance": "10.142.0.107:9100" 07:27:24 INFO [http_cmd]: http status code OK 07:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712816844" 07:27:24 DEBUG "pod": "ds-idrepo-2" 07:27:24 INFO [http_cmd]: http status code OK 07:27:24 DEBUG }, 07:27:24 DEBUG --- status code --- 07:27:24 DEBUG }, 07:27:24 DEBUG --- status code --- 07:27:24 DEBUG "value": [ 07:27:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:27:24 DEBUG "value": [ 07:27:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:27:24 DEBUG 07:27:24 DEBUG --- http response --- 07:27:24 DEBUG 07:27:24 DEBUG --- http response --- 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:27:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG ----- output ----- 07:27:24 DEBUG [...] 07:27:24 DEBUG [...] 07:27:24 DEBUG { 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG "status": "success", 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG "data": { 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG "resultType": "vector", 07:27:24 DEBUG "instance": "10.142.0.77:9100" 07:27:24 DEBUG "pod": "ds-idrepo-0" 07:27:24 DEBUG "result": [ 07:27:24 DEBUG }, 07:27:24 DEBUG }, 07:27:24 DEBUG { 07:27:24 DEBUG "value": [ 07:27:24 DEBUG "value": [ 07:27:24 DEBUG "metric": { 07:27:24 DEBUG 1712816844, 07:27:24 DEBUG 1712816844, 07:27:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:27:24 DEBUG "0.0024333333333331096" 07:27:24 DEBUG "0" 07:27:24 ERROR [http_cmd]: ERROR 07:27:24 DEBUG }, 07:27:24 DEBUG ] 07:27:24 DEBUG ] 07:27:24 ERROR --- status code --- 07:27:24 DEBUG "value": [ 07:27:24 DEBUG } 07:27:24 DEBUG } 07:27:24 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:27:24 DEBUG 07:27:24 DEBUG ] 07:27:24 DEBUG ] 07:27:24 ERROR --- http response --- 07:27:24 DEBUG 07:27:24 DEBUG } 07:27:24 DEBUG } 07:27:24 DEBUG 07:27:24 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:27:24 DEBUG } 07:27:24 DEBUG } 07:27:24 DEBUG [...] 07:27:24 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 1712816844, 07:27:24 DEBUG "0" 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG } 07:27:24 INFO [http_cmd]: http status code OK 07:27:24 DEBUG --- status code --- 07:27:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:27:24 DEBUG --- http response --- 07:27:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:27:24 DEBUG ----- output ----- 07:27:24 DEBUG { 07:27:24 DEBUG "status": "success", 07:27:24 DEBUG "data": { 07:27:24 DEBUG "resultType": "vector", 07:27:24 DEBUG "result": [ 07:27:24 DEBUG { 07:27:24 DEBUG "metric": { 07:27:24 DEBUG "instance": "10.142.0.107:9100" 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG [...] 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG "instance": "10.142.0.77:9100" 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 1712816844, 07:27:24 DEBUG "53014653952" 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG } 07:27:24 INFO [http_cmd]: http status code OK 07:27:24 DEBUG --- status code --- 07:27:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:27:24 DEBUG --- http response --- 07:27:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:27:24 DEBUG ----- output ----- 07:27:24 DEBUG { 07:27:24 DEBUG "status": "success", 07:27:24 DEBUG "data": { 07:27:24 DEBUG "resultType": "vector", 07:27:24 DEBUG "result": [ 07:27:24 DEBUG { 07:27:24 DEBUG "metric": { 07:27:24 DEBUG "pod": "ds-idrepo-2" 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG [...] 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG "pod": "ds-cts-2" 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 1712816844, 07:27:24 DEBUG "0" 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG } 07:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712816844" 07:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816844" 07:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816844" 07:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712816844" 07:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816844" 07:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816844" 07:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816844" 07:27:24 INFO [http_cmd]: http status code OK 07:27:24 INFO [http_cmd]: http status code OK 07:27:24 INFO [http_cmd]: http status code OK 07:27:24 DEBUG --- status code --- 07:27:24 DEBUG --- status code --- 07:27:24 DEBUG --- status code --- 07:27:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:27:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:27:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:27:24 DEBUG --- http response --- 07:27:24 DEBUG --- http response --- 07:27:24 DEBUG --- http response --- 07:27:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:27:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:27:24 DEBUG ----- output ----- 07:27:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:27:24 DEBUG ----- output ----- 07:27:24 DEBUG { 07:27:24 DEBUG ----- output ----- 07:27:24 DEBUG { 07:27:24 DEBUG "status": "success", 07:27:24 DEBUG { 07:27:24 DEBUG "status": "success", 07:27:24 DEBUG "data": { 07:27:24 DEBUG "status": "success", 07:27:24 DEBUG "data": { 07:27:24 DEBUG "resultType": "vector", 07:27:24 DEBUG "data": { 07:27:24 DEBUG "resultType": "vector", 07:27:24 DEBUG "result": [ 07:27:24 DEBUG "resultType": "vector", 07:27:24 DEBUG "result": [ 07:27:24 DEBUG { 07:27:24 DEBUG "result": [ 07:27:24 DEBUG { 07:27:24 DEBUG "metric": { 07:27:24 DEBUG { 07:27:24 DEBUG "metric": { 07:27:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:27:24 DEBUG "metric": { 07:27:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:27:24 DEBUG }, 07:27:24 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 INFO [http_cmd]: http status code OK 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 07:27:24 DEBUG --- status code --- 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG --- http response --- 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG [...] 07:27:24 INFO [http_cmd]: http status code OK 07:27:24 DEBUG 07:27:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:27:24 DEBUG [...] 07:27:24 INFO [http_cmd]: http status code OK 07:27:24 DEBUG 07:27:24 INFO [http_cmd]: http status code OK 07:27:24 DEBUG --- status code --- 07:27:24 DEBUG [...] 07:27:24 DEBUG ----- output ----- 07:27:24 DEBUG 07:27:24 DEBUG --- status code --- 07:27:24 DEBUG 07:27:24 DEBUG --- status code --- 07:27:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:27:24 DEBUG 07:27:24 DEBUG { 07:27:24 DEBUG 07:27:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:27:24 DEBUG 07:27:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:27:24 DEBUG --- http response --- 07:27:24 DEBUG 07:27:24 DEBUG "status": "success", 07:27:24 DEBUG 07:27:24 DEBUG --- http response --- 07:27:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:27:24 DEBUG --- http response --- 07:27:24 DEBUG 07:27:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:27:24 DEBUG "data": { 07:27:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:27:24 DEBUG }, 07:27:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:27:24 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:27:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:27:24 DEBUG ----- output ----- 07:27:24 DEBUG "resultType": "vector", 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG ----- output ----- 07:27:24 DEBUG }, 07:27:24 DEBUG ----- output ----- 07:27:24 DEBUG { 07:27:24 DEBUG "result": [ 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 1712816844, 07:27:24 DEBUG { 07:27:24 DEBUG "value": [ 07:27:24 DEBUG { 07:27:24 DEBUG "status": "success", 07:27:24 DEBUG { 07:27:24 DEBUG 1712816844, 07:27:24 DEBUG "0.026" 07:27:24 DEBUG "status": "success", 07:27:24 DEBUG 1712816844, 07:27:24 DEBUG "status": "success", 07:27:24 DEBUG "data": { 07:27:24 DEBUG "metric": { 07:27:24 DEBUG "0" 07:27:24 DEBUG ] 07:27:24 DEBUG "data": { 07:27:24 DEBUG "324.58595855964586" 07:27:24 DEBUG "data": { 07:27:24 DEBUG "resultType": "vector", 07:27:24 DEBUG "pod": "ds-idrepo-2" 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG "resultType": "vector", 07:27:24 DEBUG ] 07:27:24 DEBUG "resultType": "vector", 07:27:24 DEBUG "result": [ 07:27:24 DEBUG }, 07:27:24 DEBUG } 07:27:24 DEBUG ] 07:27:24 DEBUG "result": [ 07:27:24 DEBUG } 07:27:24 DEBUG "result": [ 07:27:24 DEBUG { 07:27:24 DEBUG "value": [ 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG { 07:27:24 DEBUG ] 07:27:24 DEBUG { 07:27:24 DEBUG "metric": { 07:27:24 DEBUG 07:27:24 DEBUG } 07:27:24 DEBUG } 07:27:24 DEBUG "metric": { 07:27:24 DEBUG } 07:27:24 DEBUG "metric": { 07:27:24 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:27:24 DEBUG 07:27:24 DEBUG } 07:27:24 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:27:24 DEBUG } 07:27:24 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:27:24 DEBUG }, 07:27:24 DEBUG 07:27:24 DEBUG }, 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG [...] 07:27:24 DEBUG "value": [ 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG [...] 07:27:24 DEBUG "pod": "ds-cts-2" 07:27:24 DEBUG [...] 07:27:24 DEBUG [...] 07:27:24 DEBUG 07:27:24 DEBUG }, 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 1712816844, 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:27:24 DEBUG "29.666666666666668" 07:27:24 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:27:24 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:27:24 DEBUG }, 07:27:24 DEBUG ] 07:27:24 DEBUG }, 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG } 07:27:24 DEBUG "value": [ 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 1712816844, 07:27:24 DEBUG ] 07:27:24 DEBUG 1712816844, 07:27:24 DEBUG 1712816844, 07:27:24 DEBUG "0.6008172123117691" 07:27:24 DEBUG } 07:27:24 DEBUG "61982826.45123046" 07:27:24 DEBUG "0" 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG ] 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG } 07:27:24 DEBUG } 07:27:24 DEBUG ] 07:27:24 DEBUG ] 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG } 07:27:24 DEBUG } 07:27:24 DEBUG } 07:27:24 DEBUG } 07:27:24 DEBUG } 07:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816844" 07:27:24 INFO [http_cmd]: http status code OK 07:27:24 DEBUG --- status code --- 07:27:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:27:24 DEBUG --- http response --- 07:27:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:27:24 DEBUG ----- output ----- 07:27:24 DEBUG { 07:27:24 DEBUG "status": "success", 07:27:24 DEBUG "data": { 07:27:24 DEBUG "resultType": "vector", 07:27:24 DEBUG "result": [ 07:27:24 DEBUG { 07:27:24 DEBUG "metric": { 07:27:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG [...] 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 1712816844, 07:27:24 DEBUG "0" 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG } 07:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816844" 07:27:24 INFO [http_cmd]: http status code OK 07:27:24 DEBUG --- status code --- 07:27:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:27:24 DEBUG --- http response --- 07:27:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:27:24 DEBUG ----- output ----- 07:27:24 DEBUG { 07:27:24 DEBUG "status": "success", 07:27:24 DEBUG "data": { 07:27:24 DEBUG "resultType": "vector", 07:27:24 DEBUG "result": [ 07:27:24 DEBUG { 07:27:24 DEBUG "metric": { 07:27:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG [...] 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 1712816844, 07:27:24 DEBUG "0" 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG } 07:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816844" 07:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816844" 07:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816844" 07:27:24 INFO [http_cmd]: http status code OK 07:27:24 DEBUG --- status code --- 07:27:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:27:24 DEBUG --- http response --- 07:27:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:27:24 DEBUG ----- output ----- 07:27:24 DEBUG { 07:27:24 DEBUG "status": "success", 07:27:24 DEBUG "data": { 07:27:24 DEBUG "resultType": "vector", 07:27:24 DEBUG "result": [ 07:27:24 DEBUG { 07:27:24 DEBUG "metric": { 07:27:24 DEBUG "pod": "ds-idrepo-2" 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG [...] 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG "pod": "ds-idrepo-0" 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 1712816844, 07:27:24 DEBUG "0" 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG } 07:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816844" 07:27:24 INFO [http_cmd]: http status code OK 07:27:24 INFO [http_cmd]: http status code OK 07:27:24 DEBUG --- status code --- 07:27:24 DEBUG --- status code --- 07:27:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:27:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:27:24 DEBUG --- http response --- 07:27:24 DEBUG --- http response --- 07:27:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:27:24 DEBUG ----- output ----- 07:27:24 DEBUG { 07:27:24 DEBUG "status": "success", 07:27:24 DEBUG "data": { 07:27:24 DEBUG "resultType": "vector", 07:27:24 DEBUG "result": [ 07:27:24 DEBUG { 07:27:24 DEBUG "metric": { 07:27:24 DEBUG "pod": "ds-idrepo-2" 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG [...] 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG "pod": "ds-cts-2" 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 1712816844, 07:27:24 DEBUG "0" 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG } 07:27:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:27:24 DEBUG ----- output ----- 07:27:24 DEBUG { 07:27:24 DEBUG "status": "success", 07:27:24 DEBUG "data": { 07:27:24 DEBUG "resultType": "vector", 07:27:24 DEBUG "result": [ 07:27:24 DEBUG { 07:27:24 DEBUG "metric": { 07:27:24 DEBUG "pod": "ds-idrepo-2" 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG [...] 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG "pod": "ds-cts-2" 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 1712816844, 07:27:24 DEBUG "0" 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG } 07:27:24 INFO [http_cmd]: http status code OK 07:27:24 DEBUG --- status code --- 07:27:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:27:24 DEBUG --- http response --- 07:27:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:27:24 DEBUG ----- output ----- 07:27:24 DEBUG { 07:27:24 DEBUG "status": "success", 07:27:24 DEBUG "data": { 07:27:24 DEBUG "resultType": "vector", 07:27:24 DEBUG "result": [ 07:27:24 DEBUG { 07:27:24 DEBUG "metric": { 07:27:24 DEBUG "instance": "10.142.0.107:9100" 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG [...] 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG "instance": "10.142.0.77:9100" 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 1712816844, 07:27:24 DEBUG "35051.6" 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG } 07:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816844" 07:27:24 INFO [http_cmd]: http status code OK 07:27:24 DEBUG --- status code --- 07:27:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:27:24 DEBUG --- http response --- 07:27:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:27:24 DEBUG ----- output ----- 07:27:24 DEBUG { 07:27:24 DEBUG "status": "success", 07:27:24 DEBUG "data": { 07:27:24 DEBUG "resultType": "vector", 07:27:24 DEBUG "result": [ 07:27:24 DEBUG { 07:27:24 DEBUG "metric": { 07:27:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG [...] 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 1712816844, 07:27:24 DEBUG "0" 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG } 07:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712816844" 07:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816844" 07:27:24 INFO [http_cmd]: http status code OK 07:27:24 DEBUG --- status code --- 07:27:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:27:24 DEBUG --- http response --- 07:27:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:27:24 DEBUG ----- output ----- 07:27:24 DEBUG { 07:27:24 DEBUG "status": "success", 07:27:24 DEBUG "data": { 07:27:24 DEBUG "resultType": "vector", 07:27:24 DEBUG "result": [ 07:27:24 DEBUG { 07:27:24 INFO [http_cmd]: http status code OK 07:27:24 DEBUG "metric": { 07:27:24 DEBUG --- status code --- 07:27:24 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:27:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:27:24 DEBUG }, 07:27:24 DEBUG --- http response --- 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG [...] 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:27:24 DEBUG 07:27:24 DEBUG ----- output ----- 07:27:24 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 07:27:24 DEBUG { 07:27:24 DEBUG }, 07:27:24 DEBUG "status": "success", 07:27:24 DEBUG "value": [ 07:27:24 DEBUG "data": { 07:27:24 DEBUG 1712816844, 07:27:24 DEBUG "resultType": "vector", 07:27:24 DEBUG "12927590400" 07:27:24 DEBUG "result": [ 07:27:24 DEBUG ] 07:27:24 DEBUG { 07:27:24 DEBUG } 07:27:24 DEBUG "metric": { 07:27:24 DEBUG ] 07:27:24 DEBUG "instance": "10.142.0.107:9100" 07:27:24 DEBUG } 07:27:24 DEBUG }, 07:27:24 DEBUG } 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG [...] 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG "instance": "10.142.0.77:9100" 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 1712816844, 07:27:24 DEBUG "0" 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG } 07:27:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816844" 07:27:24 INFO [http_cmd]: http status code OK 07:27:24 DEBUG --- status code --- 07:27:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:27:24 DEBUG --- http response --- 07:27:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:27:24 DEBUG ----- output ----- 07:27:24 DEBUG { 07:27:24 DEBUG "status": "success", 07:27:24 DEBUG "data": { 07:27:24 DEBUG "resultType": "vector", 07:27:24 DEBUG "result": [ 07:27:24 DEBUG { 07:27:24 DEBUG "metric": { 07:27:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG [...] 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG 07:27:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:27:24 DEBUG }, 07:27:24 DEBUG "value": [ 07:27:24 DEBUG 1712816844, 07:27:24 DEBUG "0" 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG ] 07:27:24 DEBUG } 07:27:24 DEBUG } 07:27:37 INFO 07:27:37 INFO [loop_until]: kubectl --namespace=xlou top pods 07:27:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:27:37 INFO [loop_until]: OK (rc = 0) 07:27:37 DEBUG --- stdout --- 07:27:37 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 7m 8165Mi am-5b5bd8567b-8h2w4 5m 8662Mi am-5b5bd8567b-qlkw6 6m 8416Mi ds-cts-0 19m 12242Mi ds-cts-1 16m 13308Mi ds-cts-2 16m 13437Mi ds-idrepo-0 14m 5526Mi ds-idrepo-1 11m 5483Mi ds-idrepo-2 10m 5784Mi end-user-ui-58878d7668-gjq6d 1m 5Mi idm-668d898cfd-2vqdm 4m 1011Mi idm-668d898cfd-st2t7 4m 1136Mi lodemon-ccc964465-p7gdm 7m 68Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 360m 2497Mi 07:27:37 DEBUG --- stderr --- 07:27:37 DEBUG 07:27:41 INFO 07:27:41 INFO [loop_until]: kubectl --namespace=xlou top node 07:27:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:27:41 INFO [loop_until]: OK (rc = 0) 07:27:41 DEBUG --- stdout --- 07:27:41 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 2729Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 62m 0% 2794Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 54m 0% 10105Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 115m 0% 3055Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 59m 0% 9772Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 144m 0% 3826Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 63m 0% 10307Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 63m 0% 6476Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 65m 0% 6239Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 77m 0% 14096Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 75m 0% 12929Mi 22% gke-xlou-cdm-ds-32e4dcb1-j2hk 70m 0% 13998Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 65m 0% 6176Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 607m 3% 6335Mi 10% 07:27:41 DEBUG --- stderr --- 07:27:41 DEBUG 07:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816904" 07:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816904" 07:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816904" 07:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816904" 07:28:24 INFO [http_cmd]: http status code OK 07:28:24 DEBUG --- status code --- 07:28:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:28:24 DEBUG --- http response --- 07:28:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:28:24 DEBUG ----- output ----- 07:28:24 DEBUG { 07:28:24 DEBUG "status": "success", 07:28:24 DEBUG "data": { 07:28:24 DEBUG "resultType": "vector", 07:28:24 DEBUG "result": [ 07:28:24 DEBUG { 07:28:24 DEBUG "metric": { 07:28:24 DEBUG "pod": "ds-idrepo-2" 07:28:24 DEBUG }, 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG [...] 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG "pod": "ds-idrepo-0" 07:28:24 DEBUG }, 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 1712816904, 07:28:24 DEBUG "0" 07:28:24 DEBUG ] 07:28:24 DEBUG } 07:28:24 DEBUG ] 07:28:24 DEBUG } 07:28:24 DEBUG } 07:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816904" 07:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816904" 07:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816904" 07:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816904" 07:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712816904" 07:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816904" 07:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816904" 07:28:24 INFO [http_cmd]: http status code OK 07:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712816904" 07:28:24 DEBUG --- status code --- 07:28:24 INFO [http_cmd]: http status code OK 07:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816904" 07:28:24 INFO [http_cmd]: http status code OK 07:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816904" 07:28:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:28:24 DEBUG --- http response --- 07:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816904" 07:28:24 DEBUG --- status code --- 07:28:24 DEBUG --- status code --- 07:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712816904" 07:28:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:28:24 INFO [http_cmd]: http status code OK 07:28:24 INFO [http_cmd]: http status code OK 07:28:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:28:24 INFO [http_cmd]: http status code OK 07:28:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:28:24 INFO [http_cmd]: http status code OK 07:28:24 INFO [http_cmd]: http status code OK 07:28:24 INFO [http_cmd]: http status code OK 07:28:24 DEBUG ----- output ----- 07:28:24 INFO [http_cmd]: http status code OK 07:28:24 DEBUG --- status code --- 07:28:24 DEBUG --- status code --- 07:28:24 DEBUG --- http response --- 07:28:24 DEBUG --- status code --- 07:28:24 DEBUG --- http response --- 07:28:24 DEBUG --- status code --- 07:28:24 ERROR [http_cmd]: ERROR 07:28:24 INFO [http_cmd]: http status code OK 07:28:24 INFO [http_cmd]: http status code OK 07:28:24 DEBUG --- status code --- 07:28:24 DEBUG --- status code --- 07:28:24 DEBUG { 07:28:24 DEBUG --- status code --- 07:28:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:28:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:28:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:28:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:28:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:28:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:28:24 ERROR --- status code --- 07:28:24 DEBUG --- status code --- 07:28:24 DEBUG --- status code --- 07:28:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:28:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:28:24 DEBUG "status": "success", 07:28:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:28:24 DEBUG --- http response --- 07:28:24 DEBUG --- http response --- 07:28:24 DEBUG ----- output ----- 07:28:24 DEBUG --- http response --- 07:28:24 DEBUG ----- output ----- 07:28:24 DEBUG --- http response --- 07:28:24 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:28:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:28:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:28:24 DEBUG --- http response --- 07:28:24 DEBUG --- http response --- 07:28:24 DEBUG "data": { 07:28:24 DEBUG --- http response --- 07:28:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:28:24 INFO [http_cmd]: http status code OK 07:28:24 DEBUG { 07:28:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:28:24 DEBUG { 07:28:24 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:28:24 ERROR --- http response --- 07:28:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 07:28:24 DEBUG --- http response --- 07:28:24 DEBUG --- http response --- 07:28:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:28:24 DEBUG "resultType": "vector", 07:28:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:28:24 DEBUG ----- output ----- 07:28:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:28:24 DEBUG --- status code --- 07:28:24 DEBUG "status": "success", 07:28:24 DEBUG ----- output ----- 07:28:24 DEBUG "status": "success", 07:28:24 DEBUG ----- output ----- 07:28:24 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:28:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:28:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:28:24 DEBUG ----- output ----- 07:28:24 DEBUG "result": [ 07:28:24 DEBUG ----- output ----- 07:28:24 DEBUG { 07:28:24 DEBUG ----- output ----- 07:28:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:28:24 DEBUG "data": { 07:28:24 DEBUG { 07:28:24 DEBUG "data": { 07:28:24 DEBUG { 07:28:24 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:28:24 DEBUG ----- output ----- 07:28:24 DEBUG ----- output ----- 07:28:24 DEBUG { 07:28:24 INFO [http_cmd]: http status code OK 07:28:24 DEBUG { 07:28:24 DEBUG { 07:28:24 DEBUG "status": "success", 07:28:24 DEBUG { 07:28:24 DEBUG --- http response --- 07:28:24 DEBUG "resultType": "vector", 07:28:24 DEBUG "status": "success", 07:28:24 DEBUG "resultType": "vector", 07:28:24 DEBUG "status": "success", 07:28:24 DEBUG { 07:28:24 DEBUG { 07:28:24 DEBUG "status": "success", 07:28:24 DEBUG --- status code --- 07:28:24 DEBUG "metric": { 07:28:24 DEBUG "status": "success", 07:28:24 DEBUG "data": { 07:28:24 DEBUG "status": "success", 07:28:24 DEBUG "result": [ 07:28:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 07:28:24 DEBUG "data": { 07:28:24 DEBUG "result": [ 07:28:24 DEBUG "data": { 07:28:24 DEBUG "status": "success", 07:28:24 DEBUG "status": "success", 07:28:24 DEBUG "data": { 07:28:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:28:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:28:24 DEBUG "data": { 07:28:24 DEBUG "resultType": "vector", 07:28:24 DEBUG "data": { 07:28:24 DEBUG { 07:28:24 DEBUG ----- output ----- 07:28:24 DEBUG "resultType": "vector", 07:28:24 DEBUG { 07:28:24 DEBUG "resultType": "vector", 07:28:24 DEBUG "data": { 07:28:24 DEBUG "data": { 07:28:24 DEBUG "resultType": "vector", 07:28:24 DEBUG --- http response --- 07:28:24 DEBUG }, 07:28:24 DEBUG "resultType": "vector", 07:28:24 DEBUG "result": [ 07:28:24 DEBUG "resultType": "vector", 07:28:24 DEBUG "metric": { 07:28:24 DEBUG { 07:28:24 DEBUG "result": [ 07:28:24 DEBUG "metric": { 07:28:24 DEBUG "result": [ 07:28:24 DEBUG "resultType": "vector", 07:28:24 DEBUG "resultType": "vector", 07:28:24 DEBUG "result": [ 07:28:24 DEBUG "value": [ 07:28:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:28:24 DEBUG "result": [ 07:28:24 DEBUG { 07:28:24 DEBUG "result": [ 07:28:24 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:28:24 DEBUG "status": "success", 07:28:24 DEBUG { 07:28:24 DEBUG "instance": "10.142.0.107:9100" 07:28:24 DEBUG { 07:28:24 DEBUG "result": [ 07:28:24 DEBUG "result": [ 07:28:24 DEBUG { 07:28:24 DEBUG 07:28:24 DEBUG ----- output ----- 07:28:24 DEBUG { 07:28:24 DEBUG "metric": { 07:28:24 DEBUG { 07:28:24 DEBUG }, 07:28:24 DEBUG "data": { 07:28:24 DEBUG "metric": { 07:28:24 DEBUG }, 07:28:24 DEBUG "metric": { 07:28:24 DEBUG { 07:28:24 DEBUG { 07:28:24 DEBUG "metric": { 07:28:24 DEBUG 07:28:24 DEBUG { 07:28:24 DEBUG "metric": { 07:28:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:28:24 DEBUG "metric": { 07:28:24 DEBUG "value": [ 07:28:24 DEBUG "resultType": "vector", 07:28:24 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:28:24 DEBUG "value": [ 07:28:24 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:28:24 DEBUG "metric": { 07:28:24 DEBUG "metric": { 07:28:24 DEBUG "instance": "10.142.0.107:9100" 07:28:24 DEBUG 07:28:24 DEBUG "status": "success", 07:28:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:28:24 DEBUG }, 07:28:24 DEBUG "instance": "10.142.0.107:9100" 07:28:24 DEBUG 07:28:24 DEBUG "result": [ 07:28:24 DEBUG }, 07:28:24 DEBUG 07:28:24 DEBUG }, 07:28:24 DEBUG "pod": "ds-idrepo-2" 07:28:24 DEBUG "pod": "ds-idrepo-2" 07:28:24 DEBUG }, 07:28:24 DEBUG [...] 07:28:24 DEBUG "data": { 07:28:24 DEBUG }, 07:28:24 DEBUG "value": [ 07:28:24 DEBUG }, 07:28:24 DEBUG 07:28:24 DEBUG { 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 07:28:24 DEBUG "value": [ 07:28:24 DEBUG }, 07:28:24 DEBUG }, 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 07:28:24 DEBUG "resultType": "vector", 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 07:28:24 DEBUG "metric": { 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG "value": [ 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG "result": [ 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG [...] 07:28:24 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:28:24 DEBUG 07:28:24 DEBUG [...] 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG { 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG [...] 07:28:24 DEBUG 07:28:24 DEBUG }, 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712816904" 07:28:24 DEBUG "metric": { 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG [...] 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG [...] 07:28:24 DEBUG }, 07:28:24 DEBUG "pod": "ds-idrepo-2" 07:28:24 DEBUG [...] 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG [...] 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG [...] 07:28:24 DEBUG [...] 07:28:24 DEBUG 07:28:24 DEBUG "value": [ 07:28:24 DEBUG }, 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG [...] 07:28:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG "instance": "10.142.0.77:9100" 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 1712816904, 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 07:28:24 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:28:24 DEBUG 07:28:24 DEBUG }, 07:28:24 DEBUG 07:28:24 DEBUG [...] 07:28:24 DEBUG }, 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG "0" 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG }, 07:28:24 DEBUG 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG "value": [ 07:28:24 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG "instance": "10.142.0.77:9100" 07:28:24 DEBUG ] 07:28:24 DEBUG 07:28:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 07:28:24 DEBUG 1712816904, 07:28:24 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:28:24 DEBUG 07:28:24 DEBUG 1712816904, 07:28:24 DEBUG }, 07:28:24 DEBUG "pod": "ds-cts-2" 07:28:24 DEBUG "pod": "ds-idrepo-0" 07:28:24 DEBUG }, 07:28:24 DEBUG } 07:28:24 DEBUG 07:28:24 DEBUG }, 07:28:24 DEBUG 1712816904, 07:28:24 DEBUG "instance": "10.142.0.77:9100" 07:28:24 DEBUG "0.026" 07:28:24 DEBUG }, 07:28:24 DEBUG 07:28:24 DEBUG "34463.666666666664" 07:28:24 DEBUG "value": [ 07:28:24 DEBUG }, 07:28:24 DEBUG }, 07:28:24 DEBUG "value": [ 07:28:24 DEBUG ] 07:28:24 DEBUG [...] 07:28:24 DEBUG "value": [ 07:28:24 DEBUG "0.5483677603286108" 07:28:24 DEBUG }, 07:28:24 DEBUG ] 07:28:24 DEBUG "value": [ 07:28:24 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:28:24 DEBUG ] 07:28:24 DEBUG 1712816904, 07:28:24 DEBUG "value": [ 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 1712816904, 07:28:24 DEBUG } 07:28:24 DEBUG 07:28:24 DEBUG 1712816904, 07:28:24 DEBUG ] 07:28:24 DEBUG "value": [ 07:28:24 DEBUG } 07:28:24 DEBUG 1712816904, 07:28:24 DEBUG "8907335.539342199" 07:28:24 DEBUG } 07:28:24 DEBUG "0" 07:28:24 DEBUG 1712816904, 07:28:24 DEBUG 1712816904, 07:28:24 DEBUG "53016148992" 07:28:24 DEBUG } 07:28:24 DEBUG 07:28:24 DEBUG "0" 07:28:24 DEBUG } 07:28:24 DEBUG 1712816904, 07:28:24 DEBUG ] 07:28:24 DEBUG }, 07:28:24 DEBUG ] 07:28:24 DEBUG ] 07:28:24 DEBUG ] 07:28:24 DEBUG "29.133333333333333" 07:28:24 DEBUG "0" 07:28:24 DEBUG ] 07:28:24 DEBUG 07:28:24 DEBUG ] 07:28:24 DEBUG ] 07:28:24 DEBUG "0.0026333333333328802" 07:28:24 DEBUG } 07:28:24 DEBUG "value": [ 07:28:24 DEBUG } 07:28:24 DEBUG } 07:28:24 DEBUG } 07:28:24 DEBUG } 07:28:24 DEBUG ] 07:28:24 DEBUG } 07:28:24 DEBUG "pod": "ds-cts-2" 07:28:24 DEBUG } 07:28:24 DEBUG } 07:28:24 DEBUG } 07:28:24 DEBUG } 07:28:24 DEBUG 1712816904, 07:28:24 DEBUG ] 07:28:24 DEBUG ] 07:28:24 DEBUG ] 07:28:24 DEBUG } 07:28:24 DEBUG ] 07:28:24 DEBUG }, 07:28:24 DEBUG ] 07:28:24 DEBUG ] 07:28:24 DEBUG "0" 07:28:24 DEBUG } 07:28:24 DEBUG } 07:28:24 DEBUG } 07:28:24 DEBUG ] 07:28:24 DEBUG } 07:28:24 DEBUG "value": [ 07:28:24 DEBUG } 07:28:24 DEBUG } 07:28:24 DEBUG ] 07:28:24 DEBUG } 07:28:24 DEBUG ] 07:28:24 DEBUG } 07:28:24 DEBUG } 07:28:24 DEBUG } 07:28:24 DEBUG 1712816904, 07:28:24 DEBUG } 07:28:24 DEBUG ] 07:28:24 DEBUG } 07:28:24 DEBUG } 07:28:24 DEBUG } 07:28:24 DEBUG "0" 07:28:24 INFO [http_cmd]: http status code OK 07:28:24 DEBUG } 07:28:24 DEBUG ] 07:28:24 DEBUG } 07:28:24 DEBUG ] 07:28:24 DEBUG --- status code --- 07:28:24 DEBUG } 07:28:24 DEBUG } 07:28:24 DEBUG } 07:28:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:28:24 DEBUG } 07:28:24 DEBUG ] 07:28:24 DEBUG --- http response --- 07:28:24 DEBUG } 07:28:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:28:24 DEBUG } 07:28:24 DEBUG ----- output ----- 07:28:24 DEBUG { 07:28:24 DEBUG "status": "success", 07:28:24 DEBUG "data": { 07:28:24 DEBUG "resultType": "vector", 07:28:24 DEBUG "result": [ 07:28:24 DEBUG { 07:28:24 DEBUG "metric": { 07:28:24 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:28:24 DEBUG }, 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG [...] 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:28:24 DEBUG }, 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 1712816904, 07:28:24 DEBUG "1.0506907300311443" 07:28:24 DEBUG ] 07:28:24 DEBUG } 07:28:24 DEBUG ] 07:28:24 DEBUG } 07:28:24 DEBUG } 07:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816904" 07:28:24 INFO [http_cmd]: http status code OK 07:28:24 DEBUG --- status code --- 07:28:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:28:24 DEBUG --- http response --- 07:28:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:28:24 DEBUG ----- output ----- 07:28:24 DEBUG { 07:28:24 DEBUG "status": "success", 07:28:24 DEBUG "data": { 07:28:24 DEBUG "resultType": "vector", 07:28:24 DEBUG "result": [ 07:28:24 DEBUG { 07:28:24 DEBUG "metric": { 07:28:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:28:24 DEBUG }, 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG [...] 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:28:24 DEBUG }, 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 1712816904, 07:28:24 DEBUG "0" 07:28:24 DEBUG ] 07:28:24 DEBUG } 07:28:24 DEBUG ] 07:28:24 DEBUG } 07:28:24 DEBUG } 07:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816904" 07:28:24 INFO [http_cmd]: http status code OK 07:28:24 DEBUG --- status code --- 07:28:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:28:24 DEBUG --- http response --- 07:28:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:28:24 DEBUG ----- output ----- 07:28:24 DEBUG { 07:28:24 DEBUG "status": "success", 07:28:24 DEBUG "data": { 07:28:24 DEBUG "resultType": "vector", 07:28:24 DEBUG "result": [ 07:28:24 DEBUG { 07:28:24 DEBUG "metric": { 07:28:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:28:24 DEBUG }, 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG [...] 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:28:24 DEBUG }, 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 1712816904, 07:28:24 DEBUG "0" 07:28:24 DEBUG ] 07:28:24 DEBUG } 07:28:24 DEBUG ] 07:28:24 DEBUG } 07:28:24 DEBUG } 07:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816904" 07:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816904" 07:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816904" 07:28:24 INFO [http_cmd]: http status code OK 07:28:24 DEBUG --- status code --- 07:28:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:28:24 DEBUG --- http response --- 07:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816904" 07:28:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:28:24 DEBUG ----- output ----- 07:28:24 DEBUG { 07:28:24 DEBUG "status": "success", 07:28:24 DEBUG "data": { 07:28:24 DEBUG "resultType": "vector", 07:28:24 DEBUG "result": [ 07:28:24 DEBUG { 07:28:24 DEBUG "metric": { 07:28:24 DEBUG "pod": "ds-idrepo-2" 07:28:24 DEBUG }, 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG [...] 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG "pod": "ds-idrepo-0" 07:28:24 DEBUG }, 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 1712816904, 07:28:24 DEBUG "0" 07:28:24 DEBUG ] 07:28:24 DEBUG } 07:28:24 DEBUG ] 07:28:24 DEBUG } 07:28:24 DEBUG } 07:28:24 INFO [http_cmd]: http status code OK 07:28:24 DEBUG --- status code --- 07:28:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:28:24 DEBUG --- http response --- 07:28:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:28:24 DEBUG ----- output ----- 07:28:24 DEBUG { 07:28:24 DEBUG "status": "success", 07:28:24 DEBUG "data": { 07:28:24 DEBUG "resultType": "vector", 07:28:24 DEBUG "result": [ 07:28:24 DEBUG { 07:28:24 DEBUG "metric": { 07:28:24 DEBUG "pod": "ds-idrepo-2" 07:28:24 DEBUG }, 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG [...] 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG "pod": "ds-cts-2" 07:28:24 DEBUG }, 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 1712816904, 07:28:24 DEBUG "0" 07:28:24 DEBUG ] 07:28:24 DEBUG } 07:28:24 DEBUG ] 07:28:24 DEBUG } 07:28:24 DEBUG } 07:28:24 INFO [http_cmd]: http status code OK 07:28:24 DEBUG --- status code --- 07:28:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:28:24 DEBUG --- http response --- 07:28:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:28:24 DEBUG ----- output ----- 07:28:24 DEBUG { 07:28:24 DEBUG "status": "success", 07:28:24 DEBUG "data": { 07:28:24 DEBUG "resultType": "vector", 07:28:24 DEBUG "result": [ 07:28:24 DEBUG { 07:28:24 DEBUG "metric": { 07:28:24 DEBUG "pod": "ds-idrepo-2" 07:28:24 DEBUG }, 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG [...] 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG "pod": "ds-cts-2" 07:28:24 DEBUG }, 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 1712816904, 07:28:24 DEBUG "0" 07:28:24 DEBUG ] 07:28:24 DEBUG } 07:28:24 DEBUG ] 07:28:24 DEBUG } 07:28:24 DEBUG } 07:28:24 INFO [http_cmd]: http status code OK 07:28:24 DEBUG --- status code --- 07:28:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:28:24 DEBUG --- http response --- 07:28:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:28:24 DEBUG ----- output ----- 07:28:24 DEBUG { 07:28:24 DEBUG "status": "success", 07:28:24 DEBUG "data": { 07:28:24 DEBUG "resultType": "vector", 07:28:24 DEBUG "result": [ 07:28:24 DEBUG { 07:28:24 DEBUG "metric": { 07:28:24 DEBUG "instance": "10.142.0.107:9100" 07:28:24 DEBUG }, 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG [...] 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG "instance": "10.142.0.77:9100" 07:28:24 DEBUG }, 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 1712816904, 07:28:24 DEBUG "35342.166666666664" 07:28:24 DEBUG ] 07:28:24 DEBUG } 07:28:24 DEBUG ] 07:28:24 DEBUG } 07:28:24 DEBUG } 07:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816904" 07:28:24 INFO [http_cmd]: http status code OK 07:28:24 DEBUG --- status code --- 07:28:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:28:24 DEBUG --- http response --- 07:28:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:28:24 DEBUG ----- output ----- 07:28:24 DEBUG { 07:28:24 DEBUG "status": "success", 07:28:24 DEBUG "data": { 07:28:24 DEBUG "resultType": "vector", 07:28:24 DEBUG "result": [ 07:28:24 DEBUG { 07:28:24 DEBUG "metric": { 07:28:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:28:24 DEBUG }, 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG [...] 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:28:24 DEBUG }, 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 1712816904, 07:28:24 DEBUG "0" 07:28:24 DEBUG ] 07:28:24 DEBUG } 07:28:24 DEBUG ] 07:28:24 DEBUG } 07:28:24 DEBUG } 07:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816904" 07:28:24 INFO [http_cmd]: http status code OK 07:28:24 DEBUG --- status code --- 07:28:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:28:24 DEBUG --- http response --- 07:28:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:28:24 DEBUG ----- output ----- 07:28:24 DEBUG { 07:28:24 DEBUG "status": "success", 07:28:24 DEBUG "data": { 07:28:24 DEBUG "resultType": "vector", 07:28:24 DEBUG "result": [ 07:28:24 DEBUG { 07:28:24 DEBUG "metric": { 07:28:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:28:24 DEBUG }, 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG [...] 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:28:24 DEBUG }, 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 1712816904, 07:28:24 DEBUG "0" 07:28:24 DEBUG ] 07:28:24 DEBUG } 07:28:24 DEBUG ] 07:28:24 DEBUG } 07:28:24 DEBUG } 07:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712816904" 07:28:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816904" 07:28:24 INFO [http_cmd]: http status code OK 07:28:24 DEBUG --- status code --- 07:28:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:28:24 DEBUG --- http response --- 07:28:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:28:24 DEBUG ----- output ----- 07:28:24 DEBUG { 07:28:24 DEBUG "status": "success", 07:28:24 DEBUG "data": { 07:28:24 DEBUG "resultType": "vector", 07:28:24 DEBUG "result": [ 07:28:24 DEBUG { 07:28:24 DEBUG "metric": { 07:28:24 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:28:24 DEBUG }, 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG [...] 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 07:28:24 DEBUG }, 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 1712816904, 07:28:24 INFO [http_cmd]: http status code OK 07:28:24 DEBUG "12929183744" 07:28:24 DEBUG --- status code --- 07:28:24 DEBUG ] 07:28:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:28:24 DEBUG } 07:28:24 DEBUG --- http response --- 07:28:24 DEBUG ] 07:28:24 DEBUG } 07:28:24 DEBUG } 07:28:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:28:24 DEBUG ----- output ----- 07:28:24 DEBUG { 07:28:24 DEBUG "status": "success", 07:28:24 DEBUG "data": { 07:28:24 DEBUG "resultType": "vector", 07:28:24 DEBUG "result": [ 07:28:24 DEBUG { 07:28:24 DEBUG "metric": { 07:28:24 DEBUG "instance": "10.142.0.107:9100" 07:28:24 DEBUG }, 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG [...] 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG 07:28:24 DEBUG "instance": "10.142.0.77:9100" 07:28:24 DEBUG }, 07:28:24 DEBUG "value": [ 07:28:24 DEBUG 1712816904, 07:28:24 DEBUG "0" 07:28:24 DEBUG ] 07:28:24 DEBUG } 07:28:24 DEBUG ] 07:28:24 DEBUG } 07:28:24 DEBUG } 07:28:37 INFO 07:28:37 INFO [loop_until]: kubectl --namespace=xlou top pods 07:28:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:28:37 INFO [loop_until]: OK (rc = 0) 07:28:37 DEBUG --- stdout --- 07:28:37 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 6m 8165Mi am-5b5bd8567b-8h2w4 6m 8662Mi am-5b5bd8567b-qlkw6 5m 8416Mi ds-cts-0 20m 12242Mi ds-cts-1 17m 13308Mi ds-cts-2 19m 13437Mi ds-idrepo-0 14m 5525Mi ds-idrepo-1 11m 5483Mi ds-idrepo-2 11m 5784Mi end-user-ui-58878d7668-gjq6d 1m 5Mi idm-668d898cfd-2vqdm 4m 1010Mi idm-668d898cfd-st2t7 5m 1136Mi lodemon-ccc964465-p7gdm 2m 68Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 1013m 3266Mi 07:28:37 DEBUG --- stderr --- 07:28:37 DEBUG 07:28:41 INFO 07:28:41 INFO [loop_until]: kubectl --namespace=xlou top node 07:28:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:28:41 INFO [loop_until]: OK (rc = 0) 07:28:41 DEBUG --- stdout --- 07:28:41 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 2728Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 66m 0% 2787Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 58m 0% 10105Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 118m 0% 3066Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 61m 0% 9771Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 140m 0% 3855Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 10306Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 64m 0% 6476Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 65m 0% 6247Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 78m 0% 14098Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 72m 0% 12930Mi 22% gke-xlou-cdm-ds-32e4dcb1-j2hk 72m 0% 14002Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 67m 0% 6172Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 1061m 6% 7134Mi 12% 07:28:41 DEBUG --- stderr --- 07:28:41 DEBUG 07:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816964" 07:29:24 INFO [http_cmd]: http status code OK 07:29:24 DEBUG --- status code --- 07:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:29:24 DEBUG --- http response --- 07:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:29:24 DEBUG ----- output ----- 07:29:24 DEBUG { 07:29:24 DEBUG "status": "success", 07:29:24 DEBUG "data": { 07:29:24 DEBUG "resultType": "vector", 07:29:24 DEBUG "result": [ 07:29:24 DEBUG { 07:29:24 DEBUG "metric": { 07:29:24 DEBUG "pod": "ds-idrepo-2" 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG [...] 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG "pod": "ds-idrepo-0" 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 1712816964, 07:29:24 DEBUG "0" 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816964" 07:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816964" 07:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816964" 07:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816964" 07:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816964" 07:29:24 INFO [http_cmd]: http status code OK 07:29:24 DEBUG --- status code --- 07:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:29:24 DEBUG --- http response --- 07:29:24 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:29:24 INFO [http_cmd]: http status code OK 07:29:24 INFO [http_cmd]: http status code OK 07:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816964" 07:29:24 DEBUG --- status code --- 07:29:24 DEBUG --- status code --- 07:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:29:24 DEBUG --- http response --- 07:29:24 DEBUG --- http response --- 07:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:29:24 DEBUG ----- output ----- 07:29:24 DEBUG ----- output ----- 07:29:24 DEBUG { 07:29:24 DEBUG { 07:29:24 DEBUG "status": "success", 07:29:24 DEBUG "status": "success", 07:29:24 DEBUG "data": { 07:29:24 DEBUG "data": { 07:29:24 DEBUG "resultType": "vector", 07:29:24 DEBUG "resultType": "vector", 07:29:24 DEBUG "result": [ 07:29:24 DEBUG "result": [ 07:29:24 DEBUG { 07:29:24 DEBUG { 07:29:24 DEBUG "metric": { 07:29:24 DEBUG "metric": { 07:29:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:29:24 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:29:24 DEBUG }, 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG [...] 07:29:24 DEBUG [...] 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:29:24 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:29:24 DEBUG }, 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 1712816964, 07:29:24 DEBUG 1712816964, 07:29:24 DEBUG "0" 07:29:24 DEBUG "0" 07:29:24 DEBUG ] 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 DEBUG ] 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712816964" 07:29:24 INFO [http_cmd]: http status code OK 07:29:24 DEBUG --- status code --- 07:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:29:24 DEBUG --- http response --- 07:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:29:24 DEBUG ----- output ----- 07:29:24 DEBUG { 07:29:24 DEBUG "status": "success", 07:29:24 DEBUG "data": { 07:29:24 DEBUG "resultType": "vector", 07:29:24 DEBUG "result": [ 07:29:24 DEBUG { 07:29:24 DEBUG "metric": { 07:29:24 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG [...] 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 1712816964, 07:29:24 DEBUG "0" 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 INFO [http_cmd]: http status code OK 07:29:24 INFO [http_cmd]: http status code OK 07:29:24 DEBUG --- status code --- 07:29:24 DEBUG --- status code --- 07:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:29:24 DEBUG --- http response --- 07:29:24 DEBUG --- http response --- 07:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:29:24 DEBUG ----- output ----- 07:29:24 DEBUG ----- output ----- 07:29:24 DEBUG { 07:29:24 DEBUG { 07:29:24 DEBUG "status": "success", 07:29:24 DEBUG "status": "success", 07:29:24 DEBUG "data": { 07:29:24 DEBUG "data": { 07:29:24 DEBUG "resultType": "vector", 07:29:24 DEBUG "resultType": "vector", 07:29:24 DEBUG "result": [ 07:29:24 DEBUG "result": [ 07:29:24 DEBUG { 07:29:24 DEBUG { 07:29:24 DEBUG "metric": { 07:29:24 DEBUG "metric": { 07:29:24 DEBUG "instance": "10.142.0.107:9100" 07:29:24 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:29:24 DEBUG }, 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG [...] 07:29:24 DEBUG [...] 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG "instance": "10.142.0.77:9100" 07:29:24 DEBUG "pod": "ds-cts-0" 07:29:24 DEBUG }, 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 1712816964, 07:29:24 DEBUG 1712816964, 07:29:24 DEBUG "53017635840" 07:29:24 DEBUG "0" 07:29:24 DEBUG ] 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 DEBUG ] 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712816964" 07:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816964" 07:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712816964" 07:29:24 INFO [http_cmd]: http status code OK 07:29:24 DEBUG --- status code --- 07:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:29:24 DEBUG --- http response --- 07:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:29:24 DEBUG ----- output ----- 07:29:24 DEBUG { 07:29:24 DEBUG "status": "success", 07:29:24 DEBUG "data": { 07:29:24 DEBUG "resultType": "vector", 07:29:24 DEBUG "result": [ 07:29:24 DEBUG { 07:29:24 DEBUG "metric": { 07:29:24 DEBUG "pod": "ds-idrepo-2" 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG [...] 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG "pod": "ds-cts-2" 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 1712816964, 07:29:24 DEBUG "0" 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 ERROR [http_cmd]: ERROR 07:29:24 ERROR --- status code --- 07:29:24 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:29:24 ERROR --- http response --- 07:29:24 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:29:24 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816964" 07:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816964" 07:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712816964" 07:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816964" 07:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816964" 07:29:24 INFO [http_cmd]: http status code OK 07:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816964" 07:29:24 DEBUG --- status code --- 07:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:29:24 DEBUG --- http response --- 07:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:29:24 DEBUG ----- output ----- 07:29:24 DEBUG { 07:29:24 DEBUG "status": "success", 07:29:24 DEBUG "data": { 07:29:24 DEBUG "resultType": "vector", 07:29:24 DEBUG "result": [ 07:29:24 DEBUG { 07:29:24 DEBUG "metric": { 07:29:24 DEBUG "instance": "10.142.0.107:9100" 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG [...] 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG "instance": "10.142.0.77:9100" 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 1712816964, 07:29:24 DEBUG "0.002733333333333121" 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 INFO [http_cmd]: http status code OK 07:29:24 DEBUG --- status code --- 07:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:29:24 DEBUG --- http response --- 07:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:29:24 DEBUG ----- output ----- 07:29:24 DEBUG { 07:29:24 DEBUG "status": "success", 07:29:24 DEBUG "data": { 07:29:24 DEBUG "resultType": "vector", 07:29:24 DEBUG "result": [ 07:29:24 DEBUG { 07:29:24 DEBUG "metric": { 07:29:24 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG [...] 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 1712816964, 07:29:24 DEBUG "1.0326862064151336" 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 INFO [http_cmd]: http status code OK 07:29:24 INFO [http_cmd]: http status code OK 07:29:24 INFO [http_cmd]: http status code OK 07:29:24 DEBUG --- status code --- 07:29:24 DEBUG --- status code --- 07:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:29:24 INFO [http_cmd]: http status code OK 07:29:24 DEBUG --- status code --- 07:29:24 INFO [http_cmd]: http status code OK 07:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:29:24 INFO [http_cmd]: http status code OK 07:29:24 DEBUG --- http response --- 07:29:24 DEBUG --- status code --- 07:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:29:24 DEBUG --- status code --- 07:29:24 DEBUG --- http response --- 07:29:24 DEBUG --- status code --- 07:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:29:24 DEBUG --- http response --- 07:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:29:24 DEBUG --- http response --- 07:29:24 DEBUG ----- output ----- 07:29:24 DEBUG --- http response --- 07:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:29:24 DEBUG --- http response --- 07:29:24 DEBUG ----- output ----- 07:29:24 DEBUG { 07:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 07:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:29:24 DEBUG ----- output ----- 07:29:24 DEBUG { 07:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:29:24 DEBUG "status": "success", 07:29:24 DEBUG ----- output ----- 07:29:24 DEBUG ----- output ----- 07:29:24 DEBUG { 07:29:24 DEBUG "status": "success", 07:29:24 DEBUG ----- output ----- 07:29:24 DEBUG "data": { 07:29:24 DEBUG { 07:29:24 DEBUG { 07:29:24 DEBUG "status": "success", 07:29:24 DEBUG "data": { 07:29:24 DEBUG { 07:29:24 DEBUG "resultType": "vector", 07:29:24 DEBUG "status": "success", 07:29:24 DEBUG "status": "success", 07:29:24 DEBUG "data": { 07:29:24 DEBUG "resultType": "vector", 07:29:24 DEBUG "status": "success", 07:29:24 DEBUG "result": [ 07:29:24 DEBUG "data": { 07:29:24 DEBUG "data": { 07:29:24 DEBUG "resultType": "vector", 07:29:24 DEBUG "result": [ 07:29:24 DEBUG "data": { 07:29:24 DEBUG { 07:29:24 DEBUG "resultType": "vector", 07:29:24 DEBUG "resultType": "vector", 07:29:24 DEBUG "result": [ 07:29:24 DEBUG { 07:29:24 DEBUG "resultType": "vector", 07:29:24 DEBUG "metric": { 07:29:24 DEBUG "result": [ 07:29:24 DEBUG "result": [ 07:29:24 DEBUG { 07:29:24 DEBUG "metric": { 07:29:24 DEBUG "result": [ 07:29:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:29:24 DEBUG { 07:29:24 DEBUG { 07:29:24 DEBUG "metric": { 07:29:24 DEBUG "pod": "ds-idrepo-2" 07:29:24 DEBUG { 07:29:24 DEBUG }, 07:29:24 DEBUG "metric": { 07:29:24 DEBUG "metric": { 07:29:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:29:24 DEBUG }, 07:29:24 DEBUG "metric": { 07:29:24 DEBUG "value": [ 07:29:24 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:29:24 DEBUG "pod": "ds-idrepo-2" 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG "instance": "10.142.0.107:9100" 07:29:24 DEBUG 07:29:24 DEBUG }, 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 07:29:24 DEBUG }, 07:29:24 DEBUG 07:29:24 DEBUG "value": [ 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG [...] 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG [...] 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG [...] 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG [...] 07:29:24 DEBUG [...] 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG [...] 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG "pod": "ds-idrepo-0" 07:29:24 DEBUG 07:29:24 DEBUG }, 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:29:24 DEBUG }, 07:29:24 DEBUG 07:29:24 DEBUG "value": [ 07:29:24 DEBUG "pod": "ds-cts-0" 07:29:24 DEBUG "pod": "ds-cts-2" 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG "instance": "10.142.0.77:9100" 07:29:24 DEBUG 1712816964, 07:29:24 DEBUG }, 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 1712816964, 07:29:24 DEBUG }, 07:29:24 DEBUG "0" 07:29:24 DEBUG "value": [ 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 1712816964, 07:29:24 DEBUG "0" 07:29:24 DEBUG "value": [ 07:29:24 DEBUG ] 07:29:24 DEBUG 1712816964, 07:29:24 DEBUG 1712816964, 07:29:24 DEBUG "0.026" 07:29:24 DEBUG ] 07:29:24 DEBUG 1712816964, 07:29:24 DEBUG } 07:29:24 DEBUG "15446.006170723344" 07:29:24 DEBUG "29.266666666666666" 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG "34403.33333333333" 07:29:24 DEBUG ] 07:29:24 DEBUG ] 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG ] 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 DEBUG ] 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816964" 07:29:24 INFO [http_cmd]: http status code OK 07:29:24 DEBUG --- status code --- 07:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:29:24 DEBUG --- http response --- 07:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:29:24 DEBUG ----- output ----- 07:29:24 DEBUG { 07:29:24 DEBUG "status": "success", 07:29:24 DEBUG "data": { 07:29:24 DEBUG "resultType": "vector", 07:29:24 DEBUG "result": [ 07:29:24 DEBUG { 07:29:24 DEBUG "metric": { 07:29:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG [...] 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 1712816964, 07:29:24 DEBUG "0" 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816964" 07:29:24 INFO [http_cmd]: http status code OK 07:29:24 DEBUG --- status code --- 07:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:29:24 DEBUG --- http response --- 07:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:29:24 DEBUG ----- output ----- 07:29:24 DEBUG { 07:29:24 DEBUG "status": "success", 07:29:24 DEBUG "data": { 07:29:24 DEBUG "resultType": "vector", 07:29:24 DEBUG "result": [ 07:29:24 DEBUG { 07:29:24 DEBUG "metric": { 07:29:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG [...] 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 1712816964, 07:29:24 DEBUG "0" 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816964" 07:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816964" 07:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816964" 07:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816964" 07:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816964" 07:29:24 INFO [http_cmd]: http status code OK 07:29:24 DEBUG --- status code --- 07:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:29:24 DEBUG --- http response --- 07:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:29:24 DEBUG ----- output ----- 07:29:24 DEBUG { 07:29:24 DEBUG "status": "success", 07:29:24 DEBUG "data": { 07:29:24 DEBUG "resultType": "vector", 07:29:24 DEBUG "result": [ 07:29:24 DEBUG { 07:29:24 DEBUG "metric": { 07:29:24 DEBUG "pod": "ds-idrepo-2" 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG [...] 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG "pod": "ds-idrepo-0" 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 1712816964, 07:29:24 DEBUG "0" 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 INFO [http_cmd]: http status code OK 07:29:24 INFO [http_cmd]: http status code OK 07:29:24 INFO [http_cmd]: http status code OK 07:29:24 DEBUG --- status code --- 07:29:24 DEBUG --- status code --- 07:29:24 DEBUG --- status code --- 07:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:29:24 DEBUG --- http response --- 07:29:24 DEBUG --- http response --- 07:29:24 DEBUG --- http response --- 07:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:29:24 DEBUG ----- output ----- 07:29:24 DEBUG ----- output ----- 07:29:24 DEBUG ----- output ----- 07:29:24 DEBUG { 07:29:24 DEBUG { 07:29:24 DEBUG { 07:29:24 DEBUG "status": "success", 07:29:24 DEBUG "status": "success", 07:29:24 DEBUG "status": "success", 07:29:24 DEBUG "data": { 07:29:24 DEBUG "data": { 07:29:24 DEBUG "data": { 07:29:24 DEBUG "resultType": "vector", 07:29:24 DEBUG "resultType": "vector", 07:29:24 DEBUG "resultType": "vector", 07:29:24 DEBUG "result": [ 07:29:24 DEBUG "result": [ 07:29:24 DEBUG "result": [ 07:29:24 DEBUG { 07:29:24 DEBUG { 07:29:24 DEBUG { 07:29:24 DEBUG "metric": { 07:29:24 DEBUG "metric": { 07:29:24 DEBUG "metric": { 07:29:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:29:24 DEBUG "pod": "ds-idrepo-2" 07:29:24 DEBUG "pod": "ds-idrepo-2" 07:29:24 DEBUG }, 07:29:24 DEBUG }, 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG "value": [ 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG [...] 07:29:24 DEBUG [...] 07:29:24 DEBUG [...] 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:29:24 DEBUG "pod": "ds-cts-2" 07:29:24 DEBUG "pod": "ds-cts-2" 07:29:24 DEBUG }, 07:29:24 DEBUG }, 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG "value": [ 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 1712816964, 07:29:24 DEBUG 1712816964, 07:29:24 DEBUG 1712816964, 07:29:24 DEBUG "0" 07:29:24 DEBUG "0" 07:29:24 DEBUG "0" 07:29:24 DEBUG ] 07:29:24 DEBUG ] 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 DEBUG ] 07:29:24 DEBUG ] 07:29:24 INFO [http_cmd]: http status code OK 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 DEBUG --- status code --- 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:29:24 DEBUG } 07:29:24 DEBUG --- http response --- 07:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:29:24 DEBUG ----- output ----- 07:29:24 DEBUG { 07:29:24 DEBUG "status": "success", 07:29:24 DEBUG "data": { 07:29:24 DEBUG "resultType": "vector", 07:29:24 DEBUG "result": [ 07:29:24 DEBUG { 07:29:24 DEBUG "metric": { 07:29:24 DEBUG "instance": "10.142.0.107:9100" 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG [...] 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG "instance": "10.142.0.77:9100" 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 1712816964, 07:29:24 DEBUG "35075.799999999996" 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712816964" 07:29:24 INFO [http_cmd]: http status code OK 07:29:24 DEBUG --- status code --- 07:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:29:24 DEBUG --- http response --- 07:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:29:24 DEBUG ----- output ----- 07:29:24 DEBUG { 07:29:24 DEBUG "status": "success", 07:29:24 DEBUG "data": { 07:29:24 DEBUG "resultType": "vector", 07:29:24 DEBUG "result": [ 07:29:24 DEBUG { 07:29:24 DEBUG "metric": { 07:29:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG [...] 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 1712816964, 07:29:24 DEBUG "0" 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712816964" 07:29:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712816964" 07:29:24 INFO [http_cmd]: http status code OK 07:29:24 DEBUG --- status code --- 07:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:29:24 DEBUG --- http response --- 07:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:29:24 DEBUG ----- output ----- 07:29:24 DEBUG { 07:29:24 DEBUG "status": "success", 07:29:24 DEBUG "data": { 07:29:24 DEBUG "resultType": "vector", 07:29:24 DEBUG "result": [ 07:29:24 DEBUG { 07:29:24 DEBUG "metric": { 07:29:24 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG [...] 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 1712816964, 07:29:24 DEBUG "12928684032" 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:24 INFO [http_cmd]: http status code OK 07:29:24 DEBUG --- status code --- 07:29:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:29:24 DEBUG --- http response --- 07:29:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:29:24 DEBUG ----- output ----- 07:29:24 DEBUG { 07:29:24 DEBUG "status": "success", 07:29:24 DEBUG "data": { 07:29:24 DEBUG "resultType": "vector", 07:29:24 DEBUG "result": [ 07:29:24 DEBUG { 07:29:24 DEBUG "metric": { 07:29:24 DEBUG "instance": "10.142.0.107:9100" 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG [...] 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG 07:29:24 DEBUG "instance": "10.142.0.77:9100" 07:29:24 DEBUG }, 07:29:24 DEBUG "value": [ 07:29:24 DEBUG 1712816964, 07:29:24 DEBUG "0" 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG ] 07:29:24 DEBUG } 07:29:24 DEBUG } 07:29:37 INFO 07:29:37 INFO [loop_until]: kubectl --namespace=xlou top pods 07:29:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:29:37 INFO [loop_until]: OK (rc = 0) 07:29:37 DEBUG --- stdout --- 07:29:37 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-pvbnd 1m 8Mi am-5b5bd8567b-6lzwk 6m 8165Mi am-5b5bd8567b-8h2w4 6m 8662Mi am-5b5bd8567b-qlkw6 5m 8416Mi ds-cts-0 18m 12243Mi ds-cts-1 17m 13308Mi ds-cts-2 18m 13437Mi ds-idrepo-0 15m 5526Mi ds-idrepo-1 11m 5483Mi ds-idrepo-2 10m 5784Mi end-user-ui-58878d7668-gjq6d 1m 5Mi idm-668d898cfd-2vqdm 4m 1011Mi idm-668d898cfd-st2t7 4m 1136Mi lodemon-ccc964465-p7gdm 2m 68Mi login-ui-5fdf49fc66-s678w 1m 4Mi overseer-0-78845f4f8c-v2r2q 1003m 4178Mi 07:29:37 DEBUG --- stderr --- 07:29:37 DEBUG 07:29:41 INFO 07:29:41 INFO [loop_until]: kubectl --namespace=xlou top node 07:29:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 07:29:41 INFO [loop_until]: OK (rc = 0) 07:29:41 DEBUG --- stdout --- 07:29:41 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 2731Mi 2% gke-xlou-cdm-default-pool-1b0769c8-mpfw 67m 0% 2779Mi 2% gke-xlou-cdm-default-pool-1b0769c8-nhlf 58m 0% 10106Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 114m 0% 3053Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 68m 0% 9784Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 138m 0% 3847Mi 3% gke-xlou-cdm-default-pool-1b0769c8-xnmp 67m 0% 10315Mi 9% gke-xlou-cdm-ds-32e4dcb1-32mw 65m 0% 6478Mi 11% gke-xlou-cdm-ds-32e4dcb1-3pj9 70m 0% 6236Mi 10% gke-xlou-cdm-ds-32e4dcb1-4hls 77m 0% 14103Mi 24% gke-xlou-cdm-ds-32e4dcb1-gxzj 73m 0% 12927Mi 22% gke-xlou-cdm-ds-32e4dcb1-j2hk 73m 0% 14001Mi 23% gke-xlou-cdm-ds-32e4dcb1-rr7g 67m 0% 6174Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 1083m 6% 8127Mi 13% 07:29:41 DEBUG --- stderr --- 07:29:41 DEBUG 07:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712817024" 07:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712817024" 07:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712817024" 07:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712817024" 07:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712817024" 07:30:24 INFO [http_cmd]: http status code OK 07:30:24 INFO [http_cmd]: http status code OK 07:30:24 DEBUG --- status code --- 07:30:24 INFO [http_cmd]: http status code OK 07:30:24 DEBUG --- status code --- 07:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:30:24 DEBUG --- status code --- 07:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:30:24 DEBUG --- http response --- 07:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:30:24 DEBUG --- http response --- 07:30:24 DEBUG --- http response --- 07:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:30:24 DEBUG ----- output ----- 07:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:30:24 DEBUG ----- output ----- 07:30:24 DEBUG { 07:30:24 DEBUG ----- output ----- 07:30:24 DEBUG { 07:30:24 DEBUG "status": "success", 07:30:24 DEBUG { 07:30:24 DEBUG "status": "success", 07:30:24 DEBUG "data": { 07:30:24 DEBUG "status": "success", 07:30:24 DEBUG "data": { 07:30:24 DEBUG "resultType": "vector", 07:30:24 DEBUG "data": { 07:30:24 DEBUG "resultType": "vector", 07:30:24 DEBUG "result": [ 07:30:24 DEBUG "resultType": "vector", 07:30:24 DEBUG "result": [ 07:30:24 DEBUG { 07:30:24 DEBUG "result": [ 07:30:24 DEBUG { 07:30:24 DEBUG "metric": { 07:30:24 DEBUG { 07:30:24 DEBUG "metric": { 07:30:24 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:30:24 DEBUG "metric": { 07:30:24 DEBUG "pod": "ds-idrepo-2" 07:30:24 DEBUG }, 07:30:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG [...] 07:30:24 DEBUG 07:30:24 DEBUG [...] 07:30:24 DEBUG 07:30:24 DEBUG [...] 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG "pod": "ds-cts-1" 07:30:24 DEBUG 07:30:24 DEBUG "pod": "ds-idrepo-0" 07:30:24 DEBUG }, 07:30:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 1712817024, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 1712817024, 07:30:24 DEBUG "0" 07:30:24 DEBUG 1712817024, 07:30:24 DEBUG "0" 07:30:24 DEBUG ] 07:30:24 DEBUG "0" 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG } 07:30:24 DEBUG } 07:30:24 DEBUG } 07:30:24 DEBUG } 07:30:24 INFO [http_cmd]: http status code OK 07:30:24 DEBUG --- status code --- 07:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:30:24 DEBUG --- http response --- 07:30:24 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 07:30:24 INFO [http_cmd]: http status code OK 07:30:24 DEBUG --- status code --- 07:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:30:24 DEBUG --- http response --- 07:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 07:30:24 DEBUG ----- output ----- 07:30:24 DEBUG { 07:30:24 DEBUG "status": "success", 07:30:24 DEBUG "data": { 07:30:24 DEBUG "resultType": "vector", 07:30:24 DEBUG "result": [ 07:30:24 DEBUG { 07:30:24 DEBUG "metric": { 07:30:24 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG [...] 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 1712817024, 07:30:24 DEBUG "29458038.178225853" 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG } 07:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712817024" 07:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1712817024" 07:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712817024" 07:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1712817024" 07:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712817024" 07:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1712817024" 07:30:24 INFO [http_cmd]: http status code OK 07:30:24 INFO [http_cmd]: http status code OK 07:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712817024" 07:30:24 DEBUG --- status code --- 07:30:24 DEBUG --- status code --- 07:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:30:24 DEBUG --- http response --- 07:30:24 DEBUG --- http response --- 07:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:30:24 DEBUG ----- output ----- 07:30:24 DEBUG ----- output ----- 07:30:24 DEBUG { 07:30:24 INFO [http_cmd]: http status code OK 07:30:24 DEBUG { 07:30:24 DEBUG "status": "success", 07:30:24 DEBUG --- status code --- 07:30:24 DEBUG "status": "success", 07:30:24 DEBUG "data": { 07:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:30:24 DEBUG "data": { 07:30:24 DEBUG "resultType": "vector", 07:30:24 DEBUG --- http response --- 07:30:24 DEBUG "resultType": "vector", 07:30:24 DEBUG "result": [ 07:30:24 INFO [http_cmd]: http status code OK 07:30:24 DEBUG "result": [ 07:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:30:24 DEBUG { 07:30:24 DEBUG --- status code --- 07:30:24 DEBUG { 07:30:24 DEBUG ----- output ----- 07:30:24 DEBUG "metric": { 07:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:30:24 DEBUG "metric": { 07:30:24 DEBUG { 07:30:24 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:30:24 DEBUG --- http response --- 07:30:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:30:24 DEBUG "status": "success", 07:30:24 DEBUG }, 07:30:24 DEBUG }, 07:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:30:24 DEBUG "data": { 07:30:24 DEBUG "value": [ 07:30:24 DEBUG "value": [ 07:30:24 DEBUG ----- output ----- 07:30:24 DEBUG "resultType": "vector", 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG { 07:30:24 DEBUG "result": [ 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG "status": "success", 07:30:24 DEBUG { 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG "data": { 07:30:24 DEBUG "metric": { 07:30:24 INFO [http_cmd]: http status code OK 07:30:24 DEBUG [...] 07:30:24 DEBUG [...] 07:30:24 DEBUG "resultType": "vector", 07:30:24 DEBUG "instance": "10.142.0.107:9100" 07:30:24 DEBUG --- status code --- 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG "result": [ 07:30:24 DEBUG }, 07:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG { 07:30:24 INFO [http_cmd]: http status code OK 07:30:24 DEBUG "value": [ 07:30:24 ERROR [http_cmd]: ERROR 07:30:24 DEBUG --- http response --- 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG "metric": { 07:30:24 DEBUG --- status code --- 07:30:24 DEBUG 07:30:24 ERROR --- status code --- 07:30:24 DEBUG "pod": "ds-cts-1" 07:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:30:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:30:24 DEBUG "pod": "ds-idrepo-2" 07:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:30:24 DEBUG 07:30:24 ERROR http status code is 400 (expected [200, 401, 403, 404]) 07:30:24 DEBUG }, 07:30:24 DEBUG ----- output ----- 07:30:24 DEBUG }, 07:30:24 DEBUG }, 07:30:24 DEBUG --- http response --- 07:30:24 DEBUG 07:30:24 ERROR --- http response --- 07:30:24 DEBUG "value": [ 07:30:24 DEBUG { 07:30:24 DEBUG "value": [ 07:30:24 DEBUG "value": [ 07:30:24 DEBUG [...] 07:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:30:24 DEBUG 1712817024, 07:30:24 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 07:30:24 DEBUG "status": "success", 07:30:24 DEBUG 1712817024, 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG ----- output ----- 07:30:24 DEBUG "0" 07:30:24 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 07:30:24 DEBUG "data": { 07:30:24 DEBUG "0.026" 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG { 07:30:24 DEBUG ] 07:30:24 DEBUG "resultType": "vector", 07:30:24 DEBUG ] 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG "status": "success", 07:30:24 DEBUG } 07:30:24 DEBUG "result": [ 07:30:24 DEBUG } 07:30:24 DEBUG [...] 07:30:24 DEBUG "instance": "10.142.0.77:9100" 07:30:24 DEBUG "data": { 07:30:24 DEBUG ] 07:30:24 DEBUG { 07:30:24 DEBUG ] 07:30:24 DEBUG 07:30:24 DEBUG }, 07:30:24 DEBUG "resultType": "vector", 07:30:24 DEBUG } 07:30:24 DEBUG "metric": { 07:30:24 DEBUG } 07:30:24 DEBUG 07:30:24 DEBUG "value": [ 07:30:24 DEBUG "result": [ 07:30:24 DEBUG } 07:30:24 DEBUG "pod": "ds-idrepo-2" 07:30:24 DEBUG } 07:30:24 DEBUG 07:30:24 DEBUG 1712817024, 07:30:24 DEBUG { 07:30:24 DEBUG }, 07:30:24 DEBUG "pod": "ds-cts-2" 07:30:24 DEBUG "53019200512" 07:30:24 DEBUG "metric": { 07:30:24 DEBUG "value": [ 07:30:24 DEBUG }, 07:30:24 DEBUG ] 07:30:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1712817024" 07:30:24 DEBUG 07:30:24 DEBUG "value": [ 07:30:24 DEBUG } 07:30:24 DEBUG }, 07:30:24 DEBUG 07:30:24 DEBUG 1712817024, 07:30:24 DEBUG ] 07:30:24 DEBUG "value": [ 07:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1712817024" 07:30:24 DEBUG 07:30:24 DEBUG "0" 07:30:24 DEBUG } 07:30:24 DEBUG 07:30:24 DEBUG [...] 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG } 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG ] 07:30:24 DEBUG [...] 07:30:24 DEBUG 07:30:24 DEBUG } 07:30:24 DEBUG 07:30:24 DEBUG "pod": "ds-idrepo-0" 07:30:24 DEBUG } 07:30:24 DEBUG 07:30:24 DEBUG }, 07:30:24 DEBUG 07:30:24 DEBUG "value": [ 07:30:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:30:24 DEBUG 1712817024, 07:30:24 DEBUG }, 07:30:24 DEBUG "0" 07:30:24 DEBUG "value": [ 07:30:24 DEBUG ] 07:30:24 DEBUG 1712817024, 07:30:24 DEBUG } 07:30:24 DEBUG "0" 07:30:24 DEBUG ] 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG } 07:30:24 DEBUG } 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG } 07:30:24 INFO [http_cmd]: http status code OK 07:30:24 DEBUG --- status code --- 07:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:30:24 DEBUG --- http response --- 07:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:30:24 DEBUG ----- output ----- 07:30:24 DEBUG { 07:30:24 DEBUG "status": "success", 07:30:24 DEBUG "data": { 07:30:24 DEBUG "resultType": "vector", 07:30:24 DEBUG "result": [ 07:30:24 DEBUG { 07:30:24 DEBUG "metric": { 07:30:24 DEBUG "instance": "10.142.0.107:9100" 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG [...] 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG "instance": "10.142.0.77:9100" 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 1712817024, 07:30:24 DEBUG "0.002966666666666858" 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG } 07:30:24 INFO [http_cmd]: http status code OK 07:30:24 DEBUG --- status code --- 07:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:30:24 DEBUG --- http response --- 07:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:30:24 DEBUG ----- output ----- 07:30:24 DEBUG { 07:30:24 DEBUG "status": "success", 07:30:24 DEBUG "data": { 07:30:24 DEBUG "resultType": "vector", 07:30:24 DEBUG "result": [ 07:30:24 DEBUG { 07:30:24 DEBUG "metric": { 07:30:24 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG [...] 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 1712817024, 07:30:24 DEBUG "0.5693347166312149" 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG } 07:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712817024" 07:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1712817024" 07:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1712817024" 07:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712817024" 07:30:24 INFO [http_cmd]: http status code OK 07:30:24 DEBUG --- status code --- 07:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:30:24 DEBUG --- http response --- 07:30:24 INFO [http_cmd]: http status code OK 07:30:24 DEBUG --- status code --- 07:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:30:24 DEBUG --- http response --- 07:30:24 INFO [http_cmd]: http status code OK 07:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 07:30:24 DEBUG --- status code --- 07:30:24 DEBUG ----- output ----- 07:30:24 DEBUG ----- output ----- 07:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:30:24 DEBUG { 07:30:24 DEBUG { 07:30:24 DEBUG --- http response --- 07:30:24 DEBUG "status": "success", 07:30:24 DEBUG "status": "success", 07:30:24 DEBUG "data": { 07:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:30:24 DEBUG "data": { 07:30:24 DEBUG "resultType": "vector", 07:30:24 DEBUG ----- output ----- 07:30:24 DEBUG "resultType": "vector", 07:30:24 DEBUG "result": [ 07:30:24 DEBUG { 07:30:24 DEBUG "result": [ 07:30:24 DEBUG { 07:30:24 DEBUG "status": "success", 07:30:24 DEBUG { 07:30:24 DEBUG "metric": { 07:30:24 DEBUG "data": { 07:30:24 DEBUG "metric": { 07:30:24 DEBUG "pod": "ds-idrepo-2" 07:30:24 DEBUG "resultType": "vector", 07:30:24 DEBUG "pod": "admin-ui-c4c7bf584-pvbnd" 07:30:24 DEBUG }, 07:30:24 DEBUG "result": [ 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG { 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 07:30:24 DEBUG "metric": { 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG "instance": "10.142.0.107:9100" 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG }, 07:30:24 DEBUG 07:30:24 DEBUG [...] 07:30:24 DEBUG "value": [ 07:30:24 DEBUG [...] 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG "pod": "ds-cts-2" 07:30:24 DEBUG [...] 07:30:24 DEBUG "pod": "overseer-0-78845f4f8c-v2r2q" 07:30:24 DEBUG }, 07:30:24 DEBUG 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 1712817024, 07:30:24 DEBUG 07:30:24 DEBUG 1712817024, 07:30:24 DEBUG "28.633333333333333" 07:30:24 DEBUG "instance": "10.142.0.77:9100" 07:30:24 DEBUG "7232.454893992023" 07:30:24 DEBUG ] 07:30:24 DEBUG }, 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG "value": [ 07:30:24 DEBUG } 07:30:24 DEBUG ] 07:30:24 DEBUG 1712817024, 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG "34507.63333333333" 07:30:24 DEBUG } 07:30:24 DEBUG } 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG } 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG } 07:30:24 INFO [http_cmd]: http status code OK 07:30:24 DEBUG --- status code --- 07:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:30:24 DEBUG --- http response --- 07:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:30:24 DEBUG ----- output ----- 07:30:24 DEBUG { 07:30:24 DEBUG "status": "success", 07:30:24 DEBUG "data": { 07:30:24 DEBUG "resultType": "vector", 07:30:24 DEBUG "result": [ 07:30:24 DEBUG { 07:30:24 DEBUG "metric": { 07:30:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG [...] 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 1712817024, 07:30:24 DEBUG "0" 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG } 07:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712817024" 07:30:24 INFO [http_cmd]: http status code OK 07:30:24 DEBUG --- status code --- 07:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:30:24 DEBUG --- http response --- 07:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:30:24 DEBUG ----- output ----- 07:30:24 DEBUG { 07:30:24 DEBUG "status": "success", 07:30:24 DEBUG "data": { 07:30:24 DEBUG "resultType": "vector", 07:30:24 DEBUG "result": [ 07:30:24 DEBUG { 07:30:24 DEBUG "metric": { 07:30:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG [...] 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 1712817024, 07:30:24 DEBUG "0" 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG } 07:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712817024" 07:30:24 INFO [http_cmd]: http status code OK 07:30:24 DEBUG --- status code --- 07:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:30:24 DEBUG --- http response --- 07:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:30:24 DEBUG ----- output ----- 07:30:24 DEBUG { 07:30:24 DEBUG "status": "success", 07:30:24 DEBUG "data": { 07:30:24 DEBUG "resultType": "vector", 07:30:24 DEBUG "result": [ 07:30:24 DEBUG { 07:30:24 DEBUG "metric": { 07:30:24 DEBUG "pod": "ds-idrepo-2" 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG [...] 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG "pod": "ds-idrepo-0" 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 1712817024, 07:30:24 DEBUG "0" 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG } 07:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1712817024" 07:30:24 INFO [http_cmd]: http status code OK 07:30:24 DEBUG --- status code --- 07:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:30:24 DEBUG --- http response --- 07:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:30:24 DEBUG ----- output ----- 07:30:24 DEBUG { 07:30:24 DEBUG "status": "success", 07:30:24 DEBUG "data": { 07:30:24 DEBUG "resultType": "vector", 07:30:24 DEBUG "result": [ 07:30:24 DEBUG { 07:30:24 DEBUG "metric": { 07:30:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG [...] 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 1712817024, 07:30:24 DEBUG "0" 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG } 07:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1712817024" 07:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1712817024" 07:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1712817024" 07:30:24 INFO [http_cmd]: http status code OK 07:30:24 DEBUG --- status code --- 07:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:30:24 DEBUG --- http response --- 07:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 07:30:24 DEBUG ----- output ----- 07:30:24 DEBUG { 07:30:24 DEBUG "status": "success", 07:30:24 DEBUG "data": { 07:30:24 DEBUG "resultType": "vector", 07:30:24 DEBUG "result": [ 07:30:24 DEBUG { 07:30:24 DEBUG "metric": { 07:30:24 DEBUG "pod": "ds-idrepo-2" 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG [...] 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG "pod": "ds-cts-2" 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 1712817024, 07:30:24 DEBUG "0" 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG } 07:30:24 INFO [http_cmd]: http status code OK 07:30:24 DEBUG --- status code --- 07:30:24 INFO [http_cmd]: http status code OK 07:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:30:24 DEBUG --- status code --- 07:30:24 DEBUG --- http response --- 07:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:30:24 DEBUG --- http response --- 07:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 07:30:24 DEBUG ----- output ----- 07:30:24 DEBUG { 07:30:24 DEBUG "status": "success", 07:30:24 DEBUG "data": { 07:30:24 DEBUG "resultType": "vector", 07:30:24 DEBUG "result": [ 07:30:24 DEBUG { 07:30:24 DEBUG "metric": { 07:30:24 DEBUG "pod": "ds-idrepo-2" 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG [...] 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG "pod": "ds-cts-2" 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 1712817024, 07:30:24 DEBUG "0" 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG } 07:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:30:24 DEBUG ----- output ----- 07:30:24 DEBUG { 07:30:24 DEBUG "status": "success", 07:30:24 DEBUG "data": { 07:30:24 DEBUG "resultType": "vector", 07:30:24 DEBUG "result": [ 07:30:24 DEBUG { 07:30:24 DEBUG "metric": { 07:30:24 DEBUG "instance": "10.142.0.107:9100" 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG [...] 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG "instance": "10.142.0.77:9100" 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 1712817024, 07:30:24 DEBUG "35229.46666666667" 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG } 07:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1712817024" 07:30:24 INFO [http_cmd]: http status code OK 07:30:24 DEBUG --- status code --- 07:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:30:24 DEBUG --- http response --- 07:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 07:30:24 DEBUG ----- output ----- 07:30:24 DEBUG { 07:30:24 DEBUG "status": "success", 07:30:24 DEBUG "data": { 07:30:24 DEBUG "resultType": "vector", 07:30:24 DEBUG "result": [ 07:30:24 DEBUG { 07:30:24 DEBUG "metric": { 07:30:24 DEBUG "pod": "am-5b5bd8567b-6lzwk" 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG [...] 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG "pod": "am-5b5bd8567b-8h2w4" 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 1712817024, 07:30:24 DEBUG "0" 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG } 07:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1712817024" 07:30:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1712817024" 07:30:24 INFO [http_cmd]: http status code OK 07:30:24 DEBUG --- status code --- 07:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:30:24 DEBUG --- http response --- 07:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:30:24 DEBUG ----- output ----- 07:30:24 DEBUG { 07:30:24 DEBUG "status": "success", 07:30:24 DEBUG "data": { 07:30:24 DEBUG "resultType": "vector", 07:30:24 DEBUG "result": [ 07:30:24 DEBUG { 07:30:24 DEBUG "metric": { 07:30:24 DEBUG "node": "gke-xlou-cdm-default-pool-1b0769c8-7vmj" 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG [...] 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG "node": "gke-xlou-cdm-ds-32e4dcb1-gxzj" 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 1712817024, 07:30:24 DEBUG "12930224128" 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG } 07:30:24 INFO [http_cmd]: http status code OK 07:30:24 DEBUG --- status code --- 07:30:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 07:30:24 DEBUG --- http response --- 07:30:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 07:30:24 DEBUG ----- output ----- 07:30:24 DEBUG { 07:30:24 DEBUG "status": "success", 07:30:24 DEBUG "data": { 07:30:24 DEBUG "resultType": "vector", 07:30:24 DEBUG "result": [ 07:30:24 DEBUG { 07:30:24 DEBUG "metric": { 07:30:24 DEBUG "instance": "10.142.0.107:9100" 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG [...] 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG 07:30:24 DEBUG "instance": "10.142.0.77:9100" 07:30:24 DEBUG }, 07:30:24 DEBUG "value": [ 07:30:24 DEBUG 1712817024, 07:30:24 DEBUG "0" 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG ] 07:30:24 DEBUG } 07:30:24 DEBUG } 07:30:33 INFO Finished: True 07:30:33 INFO Waiting for threads to register finish flag 07:31:24 INFO Done. Have a nice day! :) 127.0.0.1 - - [11/Apr/2024 07:31:24] "GET /monitoring/stop HTTP/1.1" 200 - 07:31:28 WARNING File /tmp/lodemon_data-20240411_022725/node_disk_read_bytes_total.json does not exist. Skipping graph generation... WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p6471', ...) 127.0.0.1 - - [11/Apr/2024 07:31:33] "GET /monitoring/process HTTP/1.1" 200 -